A single function that combines convolution and selection operations.

Abstract

A composite function specified by the instruction is executed. The composite function includes a number of operations that are executed as part of one invocation of the composite function. Executing the composite function includes performing a convolution with a first tensor and a second tensor, the second tensor including a scaled weight tensor created using a number of multipliers, to obtain one or more intermediate results. A value of the bias tensor is added to the one or more intermediate results to obtain one or more composite function results for the composite function.

Description

Technical Field

[0001] One or more aspects generally relate to facilitating processing within a computing environment, and particularly to improving such processing.

Background Art

[0002] To enhance processing in a computing environment where data or calculations or both are centralized, a coprocessor such as an artificial intelligence accelerator (also referred to as a neural network processor or a neural network accelerator) is utilized. Such an accelerator provides a large amount of computing power used when performing complex calculations, such as matrix or tensor calculations.

[0003] Tensor calculations are used, for example, in complex processing including deep learning, which is a subset of machine learning. Deep learning and machine learning, which are aspects of artificial intelligence, are used in various technologies including, but not limited to, engineering, manufacturing, medical technology, automotive technology, computer processing, and the like.

[0004] Deep learning uses a sequence of various operations that manipulate tensor data. In each sequence of operations, it is necessary to call an accelerator or other processors multiple times, consuming a huge amount of time and computing power. Therefore, improvements regarding the execution of such a sequence of operations are sought.

Summary of the Invention

[0005] The shortcomings of conventional technology are overcome, and further advantages are provided through the provision of computer program products to facilitate processing within the computing environment. These computer program products include executing compound functions specified by instructions. A compound function includes multiple operations performed as part of a single call to the compound function. Executing a compound function involves performing a convolution using a first tensor and a second tensor to obtain one or more intermediate results, the second tensor including a tuned weight tensor created using multiple multipliers. To obtain one or more results for the compound function, the values ​​of a bias tensor are added to one or more intermediate results.

[0006] By combining multiple operations into a single function, the number of times the processor is called to perform the operations is reduced. Furthermore, the need to store intermediate results in memory or another location accessible from one or more processors and then reload them is avoided. This improves processing speed, reduces system resource usage, and enhances overall performance.

[0007] In one example, executing a compound function further includes executing the selected activations on the results of one or more compound functions in order to provide one or more activation results of the selected activations. One or more activation results of the selected activations are, for example, at least part of an output tensor.

[0008] In one embodiment, the compound function replaces several separately invoked operations. For example, the several separately invoked operations include convolution of an input tensor and a weight tensor, followed by batch normalization, followed by scaling, and followed by activation.

[0009] Batch normalization takes multiple inputs, one example, including at least one convolution result of a convolution of an input tensor and a weight tensor, a selection multiplier, and a selection bias tensor, and uses the multiple inputs in batch normalization to provide at least one result. In one example, the at least one result is stored in a select location that is externally visible to one or more processors, and batch normalization is an operation called separately from the convolution.

[0010] In one example, scaling is an operation called separately from convolution and batch normalization, where at least one result and another selection multiplier are input. Scaling uses at least one result and another selection multiplier to reload at least one result stored in the selection location and provide at least one scaled result. The at least one scaled result is stored in the selection location.

[0011] As an example, at least one scaled result is reloaded from the selected position and used as input for activation. Activation is an operation that is called separately from, for example, convolution, batch normalization, and scaling.

[0012] In one example, a tuned weight tensor is created, and the creation involves multiplying the weight tensor by multiple multipliers in order to provide a tuned weight tensor.

[0013] In one example, one or more intermediate results are entered into the addition without one or more processors storing and reloading the one or more intermediate results in a location accessible from the outside.

[0014] For example, performing a convolution involves selecting a first input window from one or more windows of the input tensor, selecting a second input window from one or more windows of the adjusted weight tensor, multiplying the elements in the first input window by the corresponding elements in the second input window to obtain multiple products, and adding the multiple products to obtain a sum.

[0015] Furthermore, in one example, adding the values ​​of a bias tensor involves adding the values ​​of the corresponding elements of the bias tensor to the sum in order to provide another sum, which is, for example, at least part of the output tensor of a compound function.

[0016] In one embodiment, executing the compound function further includes executing the selected activations on another sum to provide one or more results of the selected activations. One or more results of the selected activations are, for example, at least part of the output tensor of the compound function.

[0017] As an example, performing a selected activation further includes determining whether other sums are in a pre-selection relationship with the selected value, and, as a result of one or more results, selecting the minimum value and clipping value of other sums based on other sums that have a pre-selection relationship with the selected value.

[0018] Computer implementation methods and systems relating to one or more embodiments are also described and asserted herein. Furthermore, services relating to one or more embodiments may also be described and asserted herein.

[0019] Further features and advantages are realized through the technologies described herein. Other embodiments and aspects are described in detail herein and are considered to be part of the claimed embodiments.

[0020] One or more aspects are particularly pointed out and distinctly claimed as examples in the claims at the end of this specification. The foregoing and the objects, features, and advantages of one or more aspects will be apparent from the following detailed description taken in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0021] [Figure 1A] An example of a computing environment for incorporating and using one or more aspects of the present invention is shown. [Figure 1B] A diagram showing further details of the processor of FIG. 1A according to one or more aspects of the present invention. [Figure 2A] A diagram showing an example of a process related to a neural network processing support instruction according to one or more aspects of the present invention. [Figure 2B] A diagram showing an example of incorporating a sequence of operations into one function of a neural network processing support instruction according to one or more aspects of the present invention. [Figure 2C] A diagram showing another example of incorporating a sequence of operations into one function of a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3A] A diagram showing an example of the format of a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3B] A diagram showing an example of a general-purpose register used by a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3C] A diagram showing an example of a function code supported by a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3D] A diagram showing an example of another general-purpose register used by a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3E]This figure shows an example of a parameter block used by a query function of a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3F] This figure shows an example of a parameter block used by one or more non-query functions of a neural network processing support instruction according to one or more aspects of the present invention. [Figure 3G] An example of a tensor descriptor used by neural network processing support instructions according to one or more aspects of the present invention is shown. [Figure 4] This figure shows an example of a format for a neural network processing (NNP) data type 1 data type according to one or more aspects of the present invention. [Figure 5] Figures (A) to (C) show examples of input data layouts used by neural network processing support instructions according to one or more aspects of the present invention. [Figure 6] Figures (A) to (C) show examples of outputs corresponding to the input data layouts in Figures 5(A) to (C) according to one or more embodiments of the present invention. [Figure 7A] This figure shows an example of facilitating processing within a computing environment according to one or more aspects of the present invention. [Figure 7B] This figure shows an example of facilitating processing within a computing environment according to one or more aspects of the present invention. [Figure 7C] This figure shows an example of facilitating processing within a computing environment according to one or more aspects of the present invention. [Figure 8A] This shows another example of a computing environment that incorporates and uses one or more aspects of the present invention. [Figure 8B] This figure shows an example of further details of the memory shown in Figure 8A according to one or more aspects of the present invention. [Figure 8C] This figure shows another example of further details of the memory shown in Figure 8A according to one or more aspects of the present invention. [Figure 9A]This shows yet another example of a computing environment that incorporates and uses one or more aspects of the present invention. [Figure 9B] This figure shows further details of the memory shown in Figure 9A according to one or more aspects of the present invention. [Figure 10] This document describes one embodiment of a cloud computing environment according to one or more aspects of the present invention. [Figure 11] This figure shows an example of an abstraction model layer according to one or more aspects of the present invention. [Modes for carrying out the invention]

[0022] One or more aspects of the present invention provide functions that facilitate processing within a computing environment. For example, instructions are provided configured to implement multiple functions, at least one of which is configured to combine a sequence of separately invoked operations into a single function and execute it as part of a single call to that function. Combining multiple operations into a single function reduces the number of times the processor is invoked to perform the operations. Furthermore, it avoids storing intermediate results in memory or another location accessible externally by one or more processors and then reloading them. This improves processing speed, reduces resource usage, and enhances performance.

[0023] One common sequence of operations used in deep learning networks is an extraction sequence, which extracts one or more features (e.g., parts of a particular image) from a given input. For example, an extraction sequence might involve performing several distinct operations, such as convolution, batch normalization, scaling, and activation (e.g., rectified linear units, gated recurrent units, tanh, sigmoid, etc.). Another common sequence of operations used in deep learning networks is a classification sequence, which might involve performing matrix multiplication followed by batch normalization in a fully connected network, and optionally performing a scaling operation.

[0024] In one or more aspects of the present invention, each of the above sequences of operations may be combined into a single function. For example, the extraction sequence may be performed by a convolution function, and the classification sequence may be performed by a matrix multiplication (matmul-op) function. These examples are described herein.

[0025] Executing a sequence of operations using a single function reduces the number of processor calls and the storage and reloading of intermediate values. This results in shorter execution times, reduced system resource usage, and improved processing speed.

[0026] For example, a function configured to perform a sequence of operations is initiated by an instruction. For example, an instruction is a neural network processing support instruction, which is a single instruction (e.g., a single architecturalized hardware machine instruction in a hardware / software interface) configured to perform multiple functions. Each function is configured as part of a single instruction (e.g., a single architecturalized instruction) to reduce system resource usage and complexity and improve system performance. Furthermore, one or more functions are configured to perform a sequence of operations as part of a single call to a function, as described herein.

[0027] The instruction may be part of a general-purpose processor instruction set architecture (ISA) dispatched by a program on a processor such as a general-purpose processor. The instruction may be executed by the general-purpose processor, or one or more functions of the instruction may be executed by a special-purpose processor, such as a coprocessor configured for a specific function, which is coupled to or part of the general-purpose processor, or both. Other variations are possible.

[0028] An embodiment of a computing environment incorporating and using one or more aspects of the present invention is described with reference to Figure 1A. As an example, the computing environment is based on the z / Architecture® instruction set architecture provided by International Business Machines Corporation in Armonk, New York. One embodiment of the z / Architecture instruction set architecture is described in the publication entitled "z / Architecture Principles of Operation," IBM publication number SA22-7832-12, 13th edition, September 2019, which is incorporated herein by reference in its entirety. However, the z / Architecture instruction set architecture is merely an example of an architecture, and other architectures or other types of computing environments of International Business Machines Corporation or other entities or both may include, or can be used with, or both of, one or more aspects of the present invention. z / Architecture and IBM are trademarks or registered trademarks of International Business Machines Corporation in at least one jurisdiction.

[0029] Referring to Figure 1A, the computing environment 100 includes, for example, a computer system 102 shown in the form of a general-purpose computing device. The computer system 102 may include, but is not limited to, one or more general-purpose processors or processing units 104 (e.g., central processing unit (CPU)), at least one special-purpose processor such as a neural network processor 105, memory 106 (e.g., system memory, main memory, primary memory, central memory, or storage), and one or more input / output (I / O) interfaces 108, all coupled to one or more buses or other connections, or both. For example, processors 104, 105 and memory 106 are coupled to the I / O interface 108 via one or more buses 110, and processors 104, 105 are coupled to one or more buses 111.

[0030] Bus 111 is, for example, a memory or cache coherence bus, and Bus 110 represents one or more of several types of bus structures, including, for example, a memory bus or memory controller, peripheral buses, accelerated graphics ports, and processor or local buses using any of various bus architectures. Examples of such architectures include, but are not limited to, Industry Standard Architecture (ISA), Microchannel Architecture (MCA), Extended ISA (EISA), VESA (Video Electronics Standards Association) local buses, and PCI (Peripheral Component Interconnect).

[0031] For example, one or more special-purpose processors (e.g., neural network processors) may be separate from one or more general-purpose processors, but may be coupled to one or more general-purpose processors, or incorporated into one or more general-purpose processors, or both. Many variations are possible.

[0032] Memory 106 may include a cache 112, such as a shared cache, which may be coupled, for example, to the local cache 114 of the processor 104 or the neural network processor 105 or both, via one or more buses 111. Furthermore, memory 106 may include one or more programs or application programs 116 and at least one operating system 118. An exemplary operating system is the z / OS® operating system provided by International Business Machines Corporation in Armonk, New York. z / OS is a trademark or registered trademark of International Business Machines Corporation in at least one jurisdiction. Other operating systems provided by International Business Machines Corporation or other entities or both may also be used. Memory 106 may also include one or more computer-readable program instructions 120, which may be configured to perform functions of embodiments of the present invention.

[0033] Furthermore, in one or more embodiments, the memory 106 includes processor firmware 122. The processor firmware includes, for example, the processor's microcode or millicode. The processor firmware includes, for example, hardware-level instructions or data structures or both used to implement higher-level machine code. In one embodiment, for example, the processor firmware includes trusted software, microcode or millicode specific to the underlying hardware, and proprietary code that is typically delivered as microcode or millicode controlling the operating system's access to the system hardware.

[0034] The computer system 102 can communicate with one or more external devices 130, such as a user terminal, tape drive, pointing device, display, and one or more data storage devices 134, for example, via an I / O interface 108. The data storage device 134 can store one or more programs 136, one or more computer-readable program instructions 138, or data or a combination thereof. The computer-readable program instructions 138 may be configured to perform functions of embodiments of the present invention.

[0035] The computer system 102 can also communicate with a network interface 132, for example, via an I / O interface 108, thereby enabling the computer system 102 to communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, and to provide communication with other computing devices or systems.

[0036] Computer system 102 may include, or be combined with, removable / non-removable, volatile / non-volatile computer system storage media, or both. For example, it may include, or be combined with, a non-removable non-volatile magnetic medium (typically called a “hard drive”), a magnetic disk drive for reading from and writing to removable non-volatile magnetic disks (e.g., “floppy disks”), or an optical disk drive for reading from and writing to removable non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media, or a combination thereof, or both. It should be understood that other hardware components or software components, or both, may be used with computer system 102. Examples include, but are not limited to, microcode or millicode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0037] Computer system 102 may operate in a number of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system 102 include, but are not limited to, personal computer (PC) systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable home appliances, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0038] In one example, a processor (e.g., processor 104 or processor 105 or both) includes several functional components (or a subset thereof) used to execute instructions. As depicted in Figure 1B, these functional components include, for example, an instruction fetch component 150 that fetches instructions to be executed, an instruction decode unit 152 that decodes the fetched instructions and retrieves the operands of the decoded instructions, one or more instruction execution components 154 that execute the decoded instructions, a memory access component 156 that accesses memory for instruction execution as needed, and a writeback component 158 ​​that provides the results of the executed instructions. One or more components may access, use, or both access one or more registers 160 in instruction processing. Furthermore, one or more components may include, or access, at least some of, one or more other components used when performing multiple operations as part of a single function call, or when performing neural network processing support operations of, for example, a neural network processing support instruction (or other processing that may use one or more aspects of the present invention), or both, according to one or more aspects of the present invention as described herein. One or more other components may include, for example, the neural network processing support component 172 (or one or more other components or both).

[0039] According to one or more aspects of the present invention, a neural network processing support instruction is initiated on a general-purpose processor (e.g., processor 104), and the function specified by the instruction is executed on either the general-purpose processor, a special-purpose processor (e.g., neural network processor 105), or both, depending on the function. The instruction then completes on the general-purpose processor. In other examples, the instruction is initiated, executed, and completed on one or more general-purpose processors or one or more special-purpose processors. Other variations are also possible.

[0040] Referring to Figure 2A, further details regarding the execution of neural network processing support instructions are described. Referring to Figure 2A, in one example, a neural network processing support instruction is received and decoded by a processor such as a general-purpose processor (e.g., processor 104) (200). The decoded instruction is issued on the general-purpose processor, for example (202). A decision is made about the function to be executed (204). In one example, this decision is made by checking the function code field of the instruction, an example of which is described below. The function is executed (210).

[0041] In one embodiment for executing a function, a decision is made whether to execute the function on a special-purpose processor (e.g., a neural network processor 105) (212). For example, in one example, query functions of neural network processing support instructions are executed on the general-purpose processor, and non-query functions are executed on the special-purpose processor. However, other variations are possible. If the function is not executed on the special-purpose processor (e.g., a query function, or in another example, one or more selected functions), in one example, it is executed on the general-purpose processor (214). However, if the function is executed on the special-purpose processor (e.g., it is a non-query function, or in another example, one or more selected functions), information for use when executing the function is provided to the special-purpose processor by the general-purpose processor, such as memory address information related to tensor data used in neural network computations (216). The special-purpose processor takes this information and executes the function (218). Once the execution of the function is complete, processing returns to the general-purpose processor (220), and the general-purpose processor completes the instruction (222). (In other examples, instructions can be started, executed, and completed on one or more general-purpose processors or one or more special-purpose processors. Other variations are also possible.)

[0042] Examples of functions to be performed are matrix multiplication functions and convolution functions, each of which is described herein. In one embodiment, these functions are performed by a special-purpose processor such as the neural network processor 105. However, in another embodiment, one or more functions may be performed by a general-purpose processor or other processors. Other variations are also possible.

[0043] Each of these functions performs a sequence of operations, as will be further explained with reference to Figures 2B-2C. Referring first to Figure 2B, in one example, the matrix multiplication function 230 (e.g., NNPA-MATMUL-OP, an example of which is described herein) takes an input tensor 232, a tuned weight tensor 234, and a bias tensor 236 as input. In a neural network, weights are, for example, learnable parameters, and biases are, for example, offsets. The input tensor 232 contains, for example, one or more features used for classification. Features describe what is being classified (e.g., an image). In one example, the tuned weight tensor is the result of multiplying the weight tensor 238 by the multiplication tensor m240. The multiplier is a value in a select range, such as -3 to +3, which normalizes the weight tensor to produce the result of a combined operation for a good artificial intelligence model. As a specific example, the value is, for example, 2.5. Other ranges or values ​​or both are possible. A function is executed that uses the input to generate at least a portion of the output tensor 242.

[0044] When the function is executed, for example, a matrix multiplication of the input tensor and the adjusted weight tensor is performed, providing one or more intermediate results with one or more bias values ​​added without restarting the processor. These operations (e.g., matrix multiplication and bias addition) are performed as part of the execution of a single function, reducing at least the number of times the neural network processor is called. Furthermore, the function is executed without storing the intermediate results (e.g., the result of multiplying the input tensor and the weight tensor) in memory or another location accessible from the outside to the processor, and then reloading those results for use in further processing. Instead, the intermediate results are temporarily stored in a scratchpad (e.g., internal registers) that is only visible to the neural network processor. This is in contrast to the previous implementation of the sequence of operations shown in 246, where each operation is performed independently, resulting in separate calls to the processor (e.g., the neural network processor 105), adding significant overhead. Furthermore, because separate operations are performed, the intermediate results of each operation are stored in memory or another externally accessible location and reloaded for the next operation, increasing overhead and system resource usage.

[0045] As an example, in implementation 246, matrix multiplication 248 is performed on an input tensor 250 and a weight tensor 252, which provide one or more intermediate results 254. Each intermediate result 254 is stored in memory and then reloaded as input to another operation, batch normalization operation 255, which also takes a bias tensor 256 and a multiplication tensor m258 as input. During batch normalization, the intermediate results are normalized to provide stabilization for the learning process. The result of batch normalization operation 255 is at least part of the output tensor 259. Here again, implementation 246 is overhead-intensive than using a single function 230 to perform multiple operations on the classification sequence. As an example, the matrix multiplication and batch normalization operations, which are called and performed independently, are replaced by a single call to a function that performs matrix multiplication using a weight tensor and bias addition. This improves system performance, reduces the use of system resources, or both.

[0046] Another function that combines multiple operations into a single function to improve overhead, system resource usage, and performance is the convolution function, as illustrated with reference to Figure 2C. Referring to Figure 2C, in one example, the convolution function 260 (e.g., NNPA-CONVOLUTION, an example of which is described herein) takes an input tensor 262, a tuned weight tensor 264, clipping values ​​265 (described herein), and a tensor bias 266 as input. In one example, the tuned weight tensor is the result of multiplying the weight tensor 268 by a first multiplication tensor m1270 and a second multiplication tensor m2272. In one example, m1270 has values ​​in the range of -3 to +3, e.g., 2.5, and m2272 has values ​​in the range of -3 to +3, e.g., 3.0. Other ranges or values ​​or both are possible for each multiplier, and the ranges or values ​​or both may be the same, different, or both for each multiplier. A function is executed that uses the input to generate at least a portion of the output tensor 274.

[0047] When the function is executed, for example, a convolution of the input tensor and the adjusted weight tensor is performed, providing one or more intermediate results with one or more bias values ​​added without restarting the processor. These operations are performed as part of the execution of a single function, reducing at least the number of times the neural network processor is called. Furthermore, this function is executed without storing the intermediate results (e.g., the result of the convolution using the input tensor and weight tensor) in memory or another location accessible from the outside of the processor, and without reloading those results. This is in contrast to the previous implementation of the sequence of operations shown in 280. In implementation 280, each operation is performed independently, resulting in separate calls to the processor (e.g., the neural network processor 105), adding significant overhead. Furthermore, because separate operations are performed, the intermediate results of each operation are stored in memory or another location accessible from the outside and reloaded as input to the next operation, increasing overhead and system resource usage.

[0048] As an example, in implementation 280, a convolution 282 is performed on the input tensor 284 and the weight tensor 286, producing one or more intermediate results 288. Each intermediate result 288 is stored in memory and then reloaded as input to another operation, a batch normalization operation 289. The batch normalization operation 289 also takes a bias tensor 290 and a multiplication tensor m1292 as input, producing one or more other intermediate results 294, each intermediate result 294 is stored in memory or elsewhere externally accessible. Each intermediate result 294 of the batch normalization operation 289, along with another multiplication tensor m2296, is input to another separately called operation, a scale operation 295. The scale operation is performed, and each intermediate result of the scale operation is stored, for example, in memory and then reloaded to be input to yet another operation, an activation operation 298. Activation operations (e.g., rectified linear unit, gated recurrent unit, tanh, sigmoid, etc.) are performed to generate at least a portion of the output tensor 299. In one embodiment, the activation operation includes a clipping operation 297, as described herein. Again, the implementation 280 is overhead-intensive than using a single function 260 to perform multiple operations on the extracted sequence. As an example, a convolution function performed using a weighted tensor and bias addition operation in a single call replaces convolution, batch normalization, scaling, and activation operations that are called and performed independently. This improves system performance, reduces the use of system resources, or both.

[0049] As shown, in one example, matrix multiplication and convolution functions are implemented as part of instructions such as Neural Network Processing Assistance (NNPA) instructions. Further details regarding Neural Network Processing Assistance instructions, including the NNPA-MATMUL-OP function and NNPA-CONVOLUTION function, are described with reference to Figures 3A-3G. Referring first to Figure 3A, in one example, Neural Network Processing Assistance instruction 300 has an RRE format indicating registers and register operations with extended operation codes (opcodes). In one example, Neural Network Processing Assistance instruction 300 includes an operation code (opcode) field 302 (e.g., bits 0-15) indicating a Neural Network Processing Assistance operation. In one example, bits 16-31 of the instruction are reserved and contain zeros. In this specification's description of instructions, functions of instructions, or operations, or both, specific locations, specific fields, or specific sizes of fields, or combinations thereof, are indicated (e.g., specific bytes or bits, or both). However, other locations, fields, or sizes, or combinations thereof, may be provided. Furthermore, it may be specified that a bit be set to a specific value, such as 1 or 0, but this is merely an example. Where a bit is set, in other examples it may be set to a different value, such as the opposite value or another value. Many modifications are possible.

[0050] In one example, an instruction uses several general-purpose registers implicitly specified by the instruction. For example, the neural network processing support instruction 300 uses the implicit registers general-purpose register 0 and general-purpose register 1, examples of which are illustrated with reference to Figures 3B and 3D, respectively.

[0051] Referring to Figure 3B, as an example, general-purpose register 0 includes a function code field and a status field that may be updated upon instruction completion. As an example, general-purpose register 0 includes a response code field 310 (e.g., bits 0-15), an exception flag field 312 (e.g., bits 24-31), and a function code field 314 (e.g., bits 56-63). Furthermore, as an example, bits 16-23 and 32-55 of general-purpose register 0 are reserved and contain zeros. One or more fields are used by specific functions executed by the instruction. As an example, not all fields are used by all functions. Each field is described below.

[0052] Response Code (RC) 310: This field (e.g., bit positions 0-15) contains the response code. When the execution of a neural network processing support instruction is completed with a condition code of, for example, 1, the response code is stored. If an invalid input condition is encountered, a non-zero value indicating the cause of the invalid input condition recognized during execution is stored in the response code field, and the selected condition code, for example, 1, is set. The codes stored in the response code field are defined as follows, for example:

[0053] Response code meaning

[0054] 0001 The parameter block format specified by the parameter block version number is not supported by the model.

[0055] 0002 The specified function is either not defined or not installed on the machine.

[0056] 0010 The specified tensor data layout format is not supported.

[0057] 0011 The specified tensor data type is not supported.

[0058] 0012 The single dimension of the specified tensor is greater than the maximum dimension index size.

[0059] 0013 The size of the specified tensor is greater than the maximum tensor size.

[0060] 0014 The specified tensor address is not located on a 4K byte boundary.

[0061] 0015 The function-specific storage area address is not located on a 4K byte boundary.

[0062] F000-FFFF are function-specific response codes. These response codes are defined for specific functions.

[0063] Exception Flags (EF) 312: This field (e.g., bit positions 24-31) contains exception flags. If an exception condition is detected during instruction execution, the corresponding exception flag control (e.g., bit) is set to 1, for example; otherwise, the control remains unchanged. The exception flags field is initialized to zero before the first call of the instruction. Reserved flags are not changed during instruction execution. The flags stored in the exception flags field are defined as follows, for example:

[0064] EF (bit) meaning

[0065] 0 Range violation. This flag is set when a non-numeric value is detected in the input tensor or stored in the output tensor. This flag is only valid, for example, when the instruction completes with a condition code, e.g., 0.

[0066] 1-7 are booked.

[0067] Function Code (FC) 314: This field (e.g., bit positions 56-63) contains the function code. Figure 3C shows an example of a function code assigned to a neural network processing support instruction. All other function codes are unassigned. If an unassigned or uninstalled function code is specified, a response code of, for example, 0002 in hexadecimal and a selection condition code of, for example, 1 are set. This field is not changed during execution.

[0068] As shown, in addition to general-purpose register 0, neural network processing support instructions also use general-purpose register 1, an example of which is depicted in Figure 3D. For example, bits 40-63 in 24-bit addressing mode, bits 33-63 in 31-bit addressing mode, or bits 0-63 in 64-bit addressing mode contain the address of parameter block 320. The contents of general-purpose register 1 specify, for example, the logical address of the leftmost byte of the parameter block in storage. The parameter block is specified on a double-word boundary; otherwise, a specification exception is recognized. For all functions, the contents of general-purpose register 1 remain unchanged.

[0069] In access register mode, access register 1 specifies an address space that includes, for example, a parameter block, an input tensor, an output tensor, and a function-specific storage area.

[0070] For example, a parameter block can have a different format depending on the function specified by the instruction being executed. For instance, the query function of an instruction might have a parameter block of one format, while other functions of the instruction might have parameter blocks of a different format. In another example, all functions might use the same parameter block format. Further modifications are possible.

[0071] For example, a parameter block, or the information within a parameter block, or both, can be stored in memory, hardware registers, or a combination of both. Other examples are possible.

[0072] Referring to Figure 3E, an example of a parameter block used by query functions such as the NNPA-Query Available Functions (QAF) operation is described. As shown, in one example, the NNPA-Query Available Functions parameter block 330 includes, for example, the following:

[0073] Installed Function Vector 332: This field of the parameter block (e.g., bytes 0-31) contains the installed function vector. In one example, bits 0-255 of the installed function vector correspond to function codes 0-255 of the neural network processing support instructions, respectively. If a bit is, for example, 1, the corresponding function is installed; otherwise, the function is not installed.

[0074] Installed parameter block format vector 334: This field of the parameter block (e.g., bytes 32-47) contains the installed parameter block format vector. In one example, bits 0-127 of the installed parameter block format vector correspond to parameter block formats 0-127 of the non-query functions of the neural network processing support instructions. If a bit is, for example, 1, the corresponding parameter block format is installed; otherwise, the parameter block format is not installed.

[0075] Installed Data Types 336: This field in the parameter block (e.g., bytes 48-49) contains the installed data type vector. In one example, bits 0-15 of the installed data type vector correspond to the data types to be installed. If a bit is, for example, 1, the corresponding data type is installed; otherwise, the data type is not installed. Examples of data types include the following (additional data types, fewer data types, other data types, or combinations thereof are also possible):

[0076] Bit data type

[0077] 0 NNP data type 1

[0078] 1-15 are reserved.

[0079] Installed Data Layout Format 338: This field in the parameter block (e.g., bytes 52-55) contains the installed data layout format vector. In one example, bits 0-31 of the installed data layout format vector correspond to the data layout format to be installed. If a bit is, for example, 1, the corresponding data layout format is installed; otherwise, the data layout format is not installed. Examples of data layout formats include (additional data types, fewer data types, other data types, or combinations thereof are also possible):

[0080] Bit Data Layout Format

[0081] 0 4D Feature Tensor

[0082] 1 4D kernel tensor

[0083] 2-31 Booked

[0084] Maximum Dimension Index Size 340: This field in the parameter block (e.g., bytes 60-63) contains a 32-bit unsigned binary integer that specifies, for example, the maximum number of elements within a specified dimension index size for any given tensor. In another example, Maximum Dimension Index Size specifies the maximum number of bytes within a specified dimension index size for a given tensor. Other examples are possible.

[0085] Maximum Tensor Size 342: This field in the parameter block (e.g., bytes 64-71) specifies the maximum number of bytes for a given tensor, including, for example, a 32-bit unsigned binary integer and pad bytes required by the tensor format. In another example, Maximum Tensor Size specifies the maximum total number of elements for any given tensor, including any padding required by the tensor format. Other examples are possible.

[0086] Installed NNP Data Type 1 Conversion Vector 344: This field in the parameter block (e.g., bytes 72-73) contains the installed NNP Data Type 1 conversion vector. In one example, bits 0-15 of the installed NNP Data Type 1 conversion vector correspond to the installed data type conversions to and from the NNP Data Type 1 format. When a bit is 1, the corresponding conversion is installed; otherwise, the conversion is not installed. Additional conversions, fewer conversions, other conversions, or a combination thereof can also be specified.

[0087] Bit Data Type

[0088] 0 reserved

[0089] 1 BFP Tiny Format

[0090] 2 BFP Short Format

[0091] 3-15 are booked.

[0092] While an example of a query function parameter block is illustrated with Figure 3E, other formats of query function parameter blocks, including the NNPA-Query Available Functions operator, may be used. The format may, for example, depend on the type of query function being executed. Furthermore, the parameter block, or each field of the parameter block, or both, may contain additional information, less information, other information, or a combination thereof.

[0093] In addition to parameter blocks for query functions, there are also parameter block formats for non-query functions, such as the non-query functions of neural network processing support instructions. An example of a parameter block used in non-query functions such as the MATMUL-OP function and CONVOLUTION function of neural network processing support instructions will be explained with reference to Figure 3F.

[0094] As shown, in one example, the parameter block 350 adopted by a non-query function of a neural network processing support instruction includes, for example, the following:

[0095] Parameter Block Version Number 352: This field of the parameter block (e.g., bytes 0-1) specifies the version and size of the parameter block. For example, bits 0-8 of the parameter block version number are reserved and contain zeros, while bits 9-15 contain an unsigned binary integer specifying the format of the parameter block. The query function provides a mechanism to indicate available parameter block formats. If the specified parameter block size or format is not supported by the model, a response code, for example 0001 in hexadecimal, is stored in general register 0, and the instruction completes with a condition code, for example condition code 1. The parameter block version number is specified by the program and does not change during instruction execution.

[0096] Model version number 354: This field in the parameter block (e.g., byte 2) is an unsigned binary integer that identifies the model that executed the instruction (e.g., a specific non-query function). If the continuation flag (described below) is 1, the model version number may be input to the operation for the purpose of resuming the operation by interpreting the contents of the continuation state buffer field (described below) in the parameter block.

[0097] Continuation Flag 356: This field in the parameter block (e.g., bit 63) indicates, for example, that the operation is partially complete, and the operation can be resumed using the contents of the continuation state buffer. The program should initialize the continuation flag to zero and not modify it when an instruction is re-executed for the purpose of resuming the operation. Otherwise, the result is unpredictable.

[0098] If the continuation flag is set at the start of the operation, and the contents of the parameter block have changed since the first call, the result is unpredictable.

[0099] Function-specific storage address 358: This field in the parameter block (e.g., bytes 56-63) contains the logical address of the function-specific storage area. In one example, the function-specific storage area address is located on a 4K byte boundary. Otherwise, for example, a response code of 0015 in hexadecimal is set to general-purpose register 0, and the instruction completes with a condition code of 1, for example. The address follows the current addressing mode. The size of the function-specific storage area depends on the function code.

[0100] If the entire function-specific storage area overlaps with the program event log (PER) storage area designation, then the PER storage change event will be recognized for the function-specific storage area. If only a portion of the function-specific storage area overlaps with the PER storage area designation, which of the following occurs depends on the model.

[0101] *PER storage change events are recognized for the entire function-specific storage area, where applicable.

[0102] *PER storage change events are recognized for the portion of the function-specific storage where they are stored, if applicable.

[0103] When the entire parameter block overlaps with the PER storage area specification, PER storage change events are recognized for the parameter block, if applicable. When only a portion of the parameter block overlaps with the PER storage area specification, which of the following occurs depends on the model.

[0104] *PER storage change events are recognized for the entire parameter block, if applicable.

[0105] *PER storage change events are recognized for the portion where the parameter block is stored, if applicable.

[0106] PER zero-address detection events are recognized for parameter blocks, if applicable. Zero-address detection does not apply to tensor addresses or function-specific storage area addresses, for example.

[0107] Output tensor descriptor (e.g., 1-2) 360 / Input tensor descriptor (e.g., 1-3) 365: An example of a tensor descriptor is explained with reference to Figure 3G. One example includes tensor descriptors 360 and 365.

[0108] Data layout format 382: This field of the tensor descriptor (e.g., byte 0) specifies the data layout format. Valid data layout formats include, for example, the following (additional data layout formats, fewer data layout formats, other data layout formats, or combinations thereof are also possible):

[0109] Format Description Alignment (bytes)

[0110] 0 4D Feature Tensor 4096

[0111] 1 4D kernel tensor 4096

[0112] 2-255 reserved --

[0113] If an unsupported or reserved data layout format is specified, for example, a response code of 0010 in hexadecimal is stored in general-purpose register 0, and the instruction is completed by setting a condition code of 1, for example.

[0114] Data type 384: This field (e.g., byte 1) specifies the data type of the tensor. Examples of supported data types are shown below (additional data types, fewer data types, other data types, or combinations thereof are also possible).

[0115] Value, Data Type, Data Size (bits)

[0116] 0 NNP data type 1 16

[0117] 1-255 Reserved --

[0118] If an unsupported or reserved data type is specified, for example, a response code of 0011 in hexadecimal is stored in general-purpose register 0, and the instruction is completed by setting a condition code of 1, for example.

[0119] Dimensions 1-4 Index Size 386: Collectively, dimension index sizes 1 to 4 specify the shape of the 4D tensor. Each dimension index size must be greater than zero and less than or equal to the maximum dimension index size (340 in Figure 3E). Otherwise, a response code, for example, 0012 in hexadecimal, is stored in general-purpose register 0, and the instruction is completed by setting the condition code, for example, to 1. The total tensor size must be less than or equal to the maximum tensor size (342 in Figure 3E). Otherwise, a response code, for example, 0013 in hexadecimal, is stored in general-purpose register 0, and the instruction is completed by setting the condition code, for example, to 1.

[0120] For example, to determine the number of bytes (i.e., total tensor size) of a 4D feature tensor with elements of NNPA data type 1, the following is used: Dimension Index 4 * Dimension Index 3 * ceil(Dimension Index 2 / 32) * 32 * ceil(Dimension Index 1 / 64) * 64 * 2.

[0121] Tensor address 388: This field of the tensor descriptor (e.g., bytes 24-31) contains the logical address of the leftmost byte of the tensor. The address follows the current addressing mode.

[0122] If the address is not aligned to the boundary of the associated data layout format, for example, a response code of 0014 in hexadecimal is stored in general-purpose register 0, and the instruction is completed by setting a condition code of 1, for example.

[0123] In access register mode, access register 1 specifies the address space that contains all active input and output tensors in storage.

[0124] Returning to Figure 3F, the parameter block 350 further includes, as an example, function-specific parameters 1 to 5 (370) that may be used by a particular function, as described herein.

[0125] Furthermore, the parameter block 350 includes, in one example, a continuation state buffer field 375 containing data (or the location of data) that would be used if the operation of this instruction were to be resumed.

[0126] As input to an operation, the reserved fields in a parameter block should contain zeros. Once the operation is complete, the reserved fields are either stored as zeros or remain unchanged.

[0127] An example of a parameter block for a non-query function is illustrated with reference to Figure 3F, but other formats of parameter blocks for non-query functions, including non-query functions for neural network processing support instructions, may be used. The format may, for example, depend on the type of function being executed. Furthermore, an example of a tensor descriptor is illustrated with reference to Figure 3G, but other formats may be used. Additionally, different formats may be used for the input and output tensors. Other variations are also possible.

[0128] Further details regarding the various functions supported by one embodiment of the neural network processing support instructions are described below.

[0129] Function code 0: NNPA-QAF (Query Available Functions)

[0130] Neural Network Processing Assisted (NNPA) query functions provide a mechanism to indicate selected information, such as the availability of installed functions, installed parameter block format, installed data types, installed data layout format, maximum dimension index size, and maximum tensor size. This information is retrieved and placed in selected locations, such as parameter blocks (e.g., parameter block 330). Once the operation is complete, reserved fields in the parameter block are either stored as zero or remain unchanged.

[0131] In the execution of one embodiment of the query function, a processor such as a general-purpose processor 104 obtains information related to a specific processor, such as a specific model of a neural network processor such as a neural network processor 105. A specific model of a processor or machine has specific capabilities. Another model of a processor or machine may have additional capabilities, fewer capabilities, different capabilities, or a combination thereof, or may be of a different generation (e.g., a current or future generation having additional capabilities, fewer capabilities, different capabilities, or a combination thereof). The obtained information is placed in a parameter block (e.g., parameter block 330), or in another structure that is accessible to or for use by one or more applications that may use this information in further processing, or both. In one example, the parameter block or the information in the parameter block, or both, is held in memory. In another embodiment, the parameter block or the information, or both, may be held in one or more hardware registers. In yet another example, the query function may be a privileged operation executed by an arithmetic system that makes available an application programming interface for making this information available to applications or unprivileged programs. In yet another example, the query function is executed by a special-purpose processor such as a neural network processor 105. Other transformations are also possible.

[0132] This information is obtained, for example, by the firmware of the processor executing the query function. The firmware has knowledge of the attributes of a particular model of a particular processor (e.g., a neural network processor). This information may be stored, for example, in a control block, registers, or memory, or a combination thereof, or it may be accessible to the processor executing the query function, or both.

[0133] The retrieved information includes model-dependent details about at least one or more data attributes of a specific processor, including, for example, one or more installed or supported data types, one or more installed or supported data layout formats, or one or more installed or supported data sizes, or combinations thereof, for a selected model of a particular processor. This information is model-dependent in that other models (e.g., previous models, future models, or both) may not support the same data attributes, such as the same data types, data sizes, or data layout formats, or combinations thereof. Condition code 0 is set as an example when the execution of a query function (e.g., the NNPA-QAF function) is complete. Condition codes 1, 2, and 3 are not applied to the query function as an example. Further information related to the retrieved information is described below.

[0134] As shown, in one example, the information obtained includes model-dependent information about one or more data attributes of a particular model of a neural network processor. An example of a data attribute is the installed data type of the neural network processor. For example, a particular model of a neural network processor (or other processor) may support one or more data types, such as NNP Data Type 1 (also called Neural Network Processing Data Type 1) or other data types, or both. NNP Data Type 1 is a 16-bit floating-point format that offers many advantages to deep learning training and inference computations, such as maintaining the precision of deep learning networks, eliminating subnormal formats that simplify rounding modes and corner case handling, automatically rounding to the nearest value in arithmetic operations, and combining the special entities of infinity and Not-a-Number (NaN) into a single value (NINF) that is accepted and handled in arithmetic operations. NINF provides better defaults for exponential overflows and invalid operations (such as division by zero). This allows many programs to continue running without hiding such errors or using special exception handlers. Other model-dependent data types are also possible.

[0135] An example of the format for NNP data type 1 is shown in Figure 4. As shown, in one example, data of NNP data type 1 may be represented in format 400, which includes, for example, a sign 402 (e.g., bit 0), an exponent +31 404 (e.g., bits 1-6), and a fraction 406 (e.g., bits 7-15).

[0136] The following are examples of properties in the NNP Data Type 1 format.

[0137] Property NNP Data Type 1

[0138] Format length (bits): 16 bits

[0139] Bias exponential length (bits): 6 bits

[0140] Fractional length (bits): 9 bits

[0141] Precision (p): 10 bits

[0142] Maximum exponent (Emax) in left-hand unit view: 32

[0143] Minimum left unit view exponent (Emin): -31

[0144] Left Unit View (LUV) Bias 31

[0145] Nmax (1-2 -9 )x2 33 ≈8.6x10 9

[0146] Nmin (1+2 -9 )x2 -31 ≈4.6x10 -10

[0147] Dmin --- Here, ≈ indicates that the value is an approximation, Nmax is the largest (largest) finite number that can be represented, and Nmin is the smallest (smallest) number that can be represented.

[0148] Further details regarding NNP data type 1 will be provided later.

[0149] Biased exponents: The bias used to allow exponents to be represented as unsigned numbers is shown above. Biased exponents are similar to the properties of binary floating-point formats, except that biased exponents of all 0s and all 1s are not given any special meaning, as will be discussed later with reference to the NNP Data Type 1 data type class.

[0150] Mantissa: For NNP data types, the binary point of a number is considered to be to the left of the leftmost fractional bit. To the left of the binary point is an implicit unit bit, which is considered 1 for normal numbers and 0 for zero. The fraction with the implicit unit bit on the left becomes the mantissa of the number.

[0151] The value of a standard NNP data type 1 is obtained by multiplying the mantissa by the base 2, and then multiplying that by an unbiased exponent.

[0152] Non-zero values: The values ​​of non-zero numbers are shown below.

[0153] Number class value

[0154] Normal number ±2 e-31 x(1.f) Here, e is a biased exponent expressed in decimal, and f is a fraction expressed in binary.

[0155] In one embodiment, the NNP data type 1 data has three classes, including numeric and associated non-numeric entities. Each data item includes a sign, exponent, and mantissa. The exponent is an unsigned number where all biased exponents are non-negative, and biased such that the smallest biased exponent is zero. The mantissa includes an explicit fraction and an implicit unit bit on the left side of the binary number. The sign bit is 0 for positive and 1 for negative.

[0156] All allowed non-zero finite numbers have a unique NNP data type 1 representation. There are no subnormal numbers, multiple representations are possible for the same value, and there are no subnormal arithmetic operations. Examples of these three classes include:

[0157] Data class, sign, bias exponent, unit bits*, fraction

[0158] Zero ± 0 0 0

[0159] Normal numbers ± 0 1 Not 0

[0160] A normal number is not ± 0, and not all numbers are 1.

[0161] Normal numbers ± all 1s - not all 1s

[0162] NINF ± All 1 - All 1 Here, - indicates that it does not apply, * indicates a unit bit, and NINF indicates that it is neither a number nor infinity.

[0163] Details of each class will be provided later.

[0164] Zero: Zero has biased exponents of zero and fractions of zero. The implicit unit bit is zero.

[0165] Normalized numbers: Normalized numbers can have biased exponents of any value. If the biased exponent is 0, the fraction is not 0. If all the biased exponents are 1, the fraction cannot be all 1. Other biased exponent values ​​can be any fractional values. The implicit unit bit is 1 for all normalized numbers.

[0166] NINF: NINF is represented by a biased exponent of all ones and a fraction of all ones. NINF represents a value that is not within the range of values ​​representable by the NNP data type 1 (i.e., a 16-bit floating-point number designed for deep learning with 6 exponent bits and 9 fraction bits). Typically, NINF is only propagated during computation and remains visible at the end.

[0167] In one example, one NNP data type is supported, but other model-dependent, special, or non-standard data types (including, but not limited to, IEEE 754 short-precision, 16-bit binary floating-point, IEEE half-precision floating-point, 8-bit floating-point, 4-bit integer format, or 8-bit integer format, or a combination thereof) may also be supported, as well as one or more standard data types. These data formats have different properties that make them suitable for neural network processing. For example, smaller data types (e.g., fewer bits) can be processed faster and use less cache / memory, while larger data types provide higher result accuracy in neural networks. Supported data types may have one or more allocated bits in the query parameter block (e.g., in the installed data type field 336 of parameter block 330). For example, special or non-standard data types supported by a particular processor are indicated in the installed data type field, while standard data types are not. In other embodiments, one or more standard data types are also indicated. Other variations are also possible.

[0168] In a particular example, bit 0 of the installed data type field 336 is reserved for the NNP data type 1 data type, and when it is set to, for example, 1, it indicates that the processor supports the NNP data type 1. As an example, the bit vector of installed data types is configured to represent up to 16 data types, with bits assigned to each data type. However, bit vectors in other embodiments may support more or fewer data types. Furthermore, the vector may be configured so that one or more bits are assigned to data types. Many examples are possible, and additional data types, fewer data types, or other data types, or a combination thereof, may be supported, indicated, or both in the vector.

[0169] In one example, a query function retrieves instructions for data types installed in a model-dependent processor and places those instructions in a parameter block, for example, by setting one or more bits in the installed data type field 336 of the parameter block 330. Furthermore, in one example, a query function retrieves instructions for an installed data layout format (another data attribute) and places that information in a parameter block, for example, by setting one or more bits in the installed data layout format field 338. Examples of data layout formats include, for example, a 4D feature tensor layout and a 4D kernel tensor layout. The 4D feature tensor layout is used, in one example, by a function shown herein, and in one example, a convolution function uses the 4D kernel tensor layout. These data layout formats arrange data in tensor storage in a way that improves processing efficiency when executing functions of neural network processing assistance instructions. For example, a neural network processing assistance instruction uses input tensors provided in a specific data layout format to operate efficiently. While exemplary layouts are provided, additional layouts, fewer layouts, other layouts, or combinations thereof may be provided for the functions described herein or other functions or both.

[0170] The use or availability of a layout for a particular processor model is provided by a vector of the installed data layout format (e.g., field 338 of parameter block 330). This vector is, for example, a bit vector of the installed data layout format, allowing the CPU to tell the application which layouts are supported. For example, bit 0 is reserved for the 4D feature tensor layout, and when it is set to, for example, 1, it indicates that the processor supports the 4D feature tensor layout; bit 1 is reserved for the 4D kernel tensor layout, and when it is set to, for example, 1, it indicates that the processor supports the 4D kernel tensor layout. In one example, the bit vector of the installed data layout format is configured to represent up to 16 data layouts, each with a bit assigned to it. However, in other embodiments, the bit vector may support more or fewer data layouts. Furthermore, a vector may be configured in which one or more bits are assigned to a data layout. Many examples are possible. Details of the 4D feature tensor layout and the 4D kernel tensor layout are described later. Again, other layouts may be used now or in the future to optimize performance.

[0171] As an example, neural network processing support instructions operate on 4D tensors, i.e., 4-dimensional tensors. These 4D tensors are obtained, for example, row-major from the common input tensors described herein. That is, when enumerating tensor elements in increasing order of memory addresses, the inner dimension called E1 is stepped up first with an E1 index size of 1, through a value of E1 index size starting from 0, before the index of the E2 dimension is incremented and the index of the E1 dimension is repeated. The index of the outer dimension called E4 is incremented last.

[0172] Tensors with lower dimensions (e.g., 3D tensors or 1D tensors) are represented as 4D tensors, where one or more dimensions of the 4D tensor are set to 1, exceeding the dimensions of the original tensor.

[0173] This document describes the conversion of row-major general-purpose 4D tensors with dimensions E4, E3, E2, and E1 to a 4D feature tensor layout (also referred to herein as NNPA data layout format 0 4D feature tensor).

[0174] The resulting tensor can be represented, for example, as a 4D tensor with 64 vector elements, or as a 5D tensor of a specific dimension.

[0175] TIFF0007872112000001.tif14168

[0176] The elements [e4][e3][e2][e1] of a typical tensor can be mapped to the following elements of the resulting 5D tensor.

[0177] TIFF0007872112000002.tif32169

[0178] The resulting tensor may be larger than a general tensor. Elements of the resulting tensor that do not correspond to elements in the general tensor are called PAD elements.

[0179] Consider the elements [fe4][fe1][fe3][fe2][fe0] of a 64-element vector in NNPA Data Layout Format 0, which is equivalent to a 4D feature tensor or a 5D tensor of elements. Whether these elements are PAD elements or corresponding elements of a general 4D tensor with dimensions E4, E3, E2, E1 can be determined by the following formula.

[0180] If fe2 ≥ E2, then this is an E2 (or page) PAD element.

[0181] If fe1*64+fe0≧E1, then this is an E1 (or row) PAD element.

[0182] The elements corresponding to the other 4D tensors are as follows:

[0183] [fe4][fe3][fe2][fe1*64+fe0]

[0184] In the case of convolutional neural network-based artificial intelligence models, the four-dimensional meaning of a feature tensor can generally be mapped as follows:

[0185] • E4: N-Mini Batch Size

[0186] • E3: H-3D tensor / image height

[0187] • E2:W-3D tensor / image width

[0188] • E1: Channel or class of C-3D tensor

[0189] In machine learning and recurrent neural network-based artificial intelligence models, the four-dimensional meaning of a 4D feature tensor is generally mapped as follows:

[0190] • E4: T - Number of time steps or models

[0191] • E3: Reserved, normally set to 1

[0192] ·E2:N mb- Mini badge size

[0193] ·E1:L-Features

[0194] The NNPA Data Layout Format 0 provides 2D data locality for the outer dimensions of the generated tensor using 4KB block data (pages) and 4KB block data alignment.

[0195] PAD element bytes are ignored in the input tensor and are unpredictable in the output tensor. Storage changes in PER for PAD bytes are unpredictable.

[0196] An example of the input data layout for a 4D feature tensor layout with dimensions E1, E2, E3, and E4 is shown in Figures 5(A) to (C), and an example of the output for a 4D feature tensor layout is shown in Figures 6(A) to (C). Referring to Figure 5(A), a 3D tensor 500 with dimensions E1, E2, and E3 is shown. In one example, each 3D tensor contains multiple 2D tensors 502. The numbers in each 2D tensor 502 represent the memory offset where each of its elements is located in memory. The input is used to lay out the data of the original tensor in memory (e.g., the original 4D tensor in Figures 5(A) to (C)), as shown in Figures 6(A) to (C), which correspond to Figures 5(A) to (C).

[0197] In Figure 6(A), as an example, a memory unit 600 (e.g., a memory page) contains a pre-selected number (e.g., 32) rows 602, each row identified by, for example, e2_page_idx, and each row has a pre-selected number (e.g., 64) elements 604, each element identified by, for example, e1_page_idx. If a row does not contain the pre-selected number of elements, padding 606 called row padding or E1 padding is performed, and if the memory unit does not have the pre-selected number of rows, padding 608 called page padding or E2 padding is performed. For example, row padding is, for example, zero or other values, and page padding is, for example, an existing value, zero, or other values.

[0198] In one example, the output elements of a row are provided in memory (e.g., within a page) based on the element position in the E1 direction of the corresponding input. For example, referring to Figure 5(A), the element positions 0, 1, and 2 of the three matrices shown (e.g., the element positions at the same positions in each matrix) are shown in Figure 6(A), such as row 0 on page 0. In this example, the 4D tensor is small, and all the elements of each 2D tensor representing the 4D tensor fit on one page. However, this is just one example. A 2D tensor may contain one or more pages. If a 2D tensor is created based on a reformatting of a 4D tensor, the number of pages in the 2D tensor is based on the size of the 4D tensor. In one example, one or more ceil functions are used to determine the number of rows in a 2D tensor and the number of elements in each row, which represents the number of pages used. Other modifications are also possible.

[0199] In addition to 4D feature tensor layouts, a neural network processor may, for example, support 4D kernel tensors, which rearrange the elements of a 4D tensor to reduce the number of memory access and data acquisition steps when performing certain artificial intelligence (e.g., neural network processing assistance) operations such as convolution. As an example, a row-first general-purpose 4D tensor with dimensions E4, E3, E2, and E1 is converted to an NNPA Data Layout Format 1 4D kernel tensor (4D-kernel tensor), as described herein.

[0200] The resulting tensor can be represented, for example, as a 4D tensor of 64 vectors, or as a 5D tensor with dimensions.

[0201] TIFF0007872112000003.tif13170

[0202] The elements [e4][e3][e2][e1] of a typical tensor can be mapped to the following elements of the resulting 5D tensor.

[0203] TIFF0007872112000004.tif26168

[0204] The resulting tensor may be larger than a general tensor. Elements of the resulting tensor that do not correspond to elements in the general tensor are called PAD elements.

[0205] Consider the elements [fe1][fe4][fe3][fe2][fe0] of a 64-element vector NNPA data layout format 1 4D feature tensor, or an equivalent representation of the elements as a 5D tensor. Whether these elements are PAD elements or corresponding elements of a general 4D tensor with dimensions E4, E3, E2, E1 can be determined by the following formula.

[0206] If fe2 ≥ E2, then this is an E2 (or page) PAD element.

[0207] If fe1*64+fe0≧E1, then this is an E1 (or row) PAD element.

[0208] The corresponding elements of other common 4D tensors are as follows:

[0209] [fe4][fe3][fe2][fe1*64+fe0]

[0210] In the case of convolutional neural network-based artificial intelligence models, the four-dimensional meaning of the kernel tensor can generally be mapped as follows:

[0211] • E4: H-3D tensor / image height

[0212] • E3:W-3D tensor / image width

[0213] • E2: Number of channels in a C-3D tensor

[0214] • E1: Number of K kernels

[0215] The NNPA Data Layout Format 1 provides, for example, two-dimensional kernel parallelism within a 4KB data block (page) and 4KB block data alignment for the outer dimensions of the generated tensor for efficient processing.

[0216] PAD bytes are ignored in the input tensor. Storage changes in PER for PAD bytes are unpredictable.

[0217] Here again, the exemplary data layout formats include the 4D feature tensor layout and the 4D kernel tensor layout, but other data layout formats may be supported by the processor (e.g., the neural network processor 105). An indication of a supported data layout is obtained, for example, by setting one or more bits in field 338 and placing them in the query parameter block.

[0218] The query parameter block also includes other data attribute information according to one or more aspects of the present invention, which includes, for example, supported data size information. A processor, such as a neural network processor, typically has limitations based on internal buffer size, processing unit, data bus structure, firmware limitations, etc., which can limit the maximum size of the tensor dimension, or the overall size of the tensor, or both. Therefore, the query function provides fields to communicate these limitations to the application. For example, the processor obtains various data sizes, such as the maximum dimension index size (e.g., 65,536 elements) and the maximum tensor size (e.g., 8GB), based on the execution of the query function, and includes this information in fields 340 and 342 of the parameter block (e.g., parameter block 330), respectively. Additional size information, less size information, or other size information, or a combination thereof, may also be supported by the processor (e.g., neural network processor 105), and is therefore obtained and placed in the parameter block, for example, fields 340, 342, or other fields or a combination thereof. In other embodiments, the limit may be smaller or larger, or the size may be in other units, such as bytes instead of elements, or elements instead of bytes, or both. Furthermore, in other embodiments, different maximum sizes may be allowed for each dimension, rather than the same maximum size for all dimensions. Many modifications are possible.

[0219] According to one or more aspects of the present invention, a query function is provided that conveys detailed information relating to a particular model of a selected processor (e.g., a neural network processor 105). This detailed information includes, for example, model-dependent information relating to the particular processor. (The processor may also support standard data attributes such as standard data types and standard data layouts, which are implied by the query function and not necessarily presented, but in other embodiments, the query function may indicate all data attributes or various selected subsets, etc.) Exemplary information is provided, but in other embodiments, other information may be provided. The obtained information, which may differ in different models of the processor or different processors or both, is used to perform artificial intelligence or other processing or both. The artificial intelligence or other processing or both may employ, for example, one or more non-query functions of the neural network processing support instruction. A particular non-query function to be employed in the processing is performed by executing the neural network processing support instruction one or more times and specifying the non-query specific function.

[0220] Further details of exemplary non-query functions supported by neural network processing support instructions are described below (in other embodiments, additional functions, fewer functions, other functions, or combinations thereof may be supported).

[0221] Function code 16: NNPA-ADD (Addition)

[0222] When the NNPA-ADD function is specified, each element of input tensor 1 described by tensor descriptor 1 is added to the corresponding element of input tensor 2 described by tensor descriptor 2, and the resulting sum is placed in the corresponding element of the output tensor described by the output tensor descriptor.

[0223] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0224] The shapes, data layouts, and data types of input tensor 1, input tensor 2, and output tensor are the same in one example, but exceptions to generic operand data are recognized.

[0225] In one example, the output tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0226] Function code 17: NNPA-SUB (Subtraction)

[0227] When the NNPA-SUB function is specified, each element of input tensor 2 described by tensor descriptor 2 is subtracted from the corresponding element of input tensor 1 described by tensor descriptor 1, and the resulting difference is placed in the corresponding element of the output tensor.

[0228] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), then a response code, for example 0010 or 0011 in hexadecimal, is set in general register 0, and the instruction is completed with a condition code, for example 1.

[0229] The shapes, data layouts, and data types of input tensor 1, input tensor 2, and output tensor are the same in one example, but exceptions to generic operand data are recognized.

[0230] In one example, the output tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0231] Function code 18: NNPA-MUL (Multiplication)

[0232] When the NNPA-MUL function is specified, the product of each element (multiplier) of input tensor 1 described by tensor descriptor 1 and the corresponding element (multiplier) of input tensor 2 described by tensor descriptor 2 is placed in the corresponding element of the output tensor.

[0233] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), then a response code, for example 0010 or 0011 in hexadecimal, is set in general register 0, and the instruction is completed with a condition code, for example 1.

[0234] The shapes, data layouts, and data types of input tensor 1, input tensor 2, and output tensor are the same in one example, but exceptions to generic operand data are recognized.

[0235] In one example, the output tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0236] Function code 19: NNPA-DIV (Division)

[0237] When the NNPA-DIV function is specified, each element (dividend) of input tensor 1, described by tensor descriptor 1, is divided by the corresponding element (divisor) of input tensor 2, described by tensor descriptor 2, and the quotient is placed in the corresponding element of the output tensor.

[0238] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), then a response code, for example 0010 or 0011 in hexadecimal, is set in general register 0, and the instruction is completed with a condition code, for example 1.

[0239] The shapes, data layouts, and data types of input tensor 1, input tensor 2, and output tensor are the same in one example, but exceptions to generic operand data are recognized.

[0240] In one example, the output tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0241] Function code 20: NNPA-MIN (Minimum)

[0242] When the NNPA-MIN function is specified, each element of input tensor 1 described by tensor descriptor 1 is compared with the corresponding element of input tensor 2 described by tensor descriptor 2. The smaller of the two values ​​is placed in the corresponding element of the output tensor descriptor. If both values ​​are equal, that value is placed in the corresponding element of the output tensor.

[0243] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0244] The shapes, data layouts, and data types of input tensor 1, input tensor 2, and output tensor are the same in one example, but exceptions to generic operand data are recognized.

[0245] In one example, the output tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0246] Function code 21: NNPA-MAX (Maximum)

[0247] When the NNPA-MAX function is specified, each element of input tensor 1, described by tensor descriptor 1, is compared to the corresponding element of input tensor 2, described by tensor descriptor 2. The larger of the two values ​​is placed in the corresponding element of the output tensor descriptor. If both values ​​are the same, that value is placed in the corresponding element of the output tensor.

[0248] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0249] The shapes, data layouts, and data types of input tensor 1, input tensor 2, and output tensor are the same in one example, but exceptions to generic operand data are recognized.

[0250] In one example, the output tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0251] Function code 32: NNPA-LOG (Natural Logarithm)

[0252] When the NNPA-LOG function is specified, for each element of the input tensor described by tensor descriptor 1, if that element is greater than 0, the corresponding element of the output tensor described by the output tensor descriptor will be the natural logarithm of that element. Otherwise, the corresponding element of the output tensor cannot be represented numerically, and a value related to negative infinity of the target data type will be stored.

[0253] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0254] In one example, the shape, data layout, and data type of input tensor 1 and output tensor are the same, but exceptions for generic operand data are recognized.

[0255] In one example, output tensor descriptor 2, input tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0256] Function code 33: NNPA-EXP (Exponential)

[0257] When the NNPA-EXP function is specified, for each element of the input tensor described by tensor descriptor 1, the corresponding element of the output tensor described by the output tensor descriptor becomes the exponent of that element.

[0258] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout = 0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type = 0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0259] In one example, the shape, data layout, and data type of input tensor 1 and output tensor are the same, but exceptions for generic operand data are recognized.

[0260] In one example, output tensor descriptor 2, input tensor descriptor 2, input tensor descriptor 3, function-specific parameters 1-5, and function-specific storage area address field are ignored.

[0261] Function code 49: NNPA-RELU (Rectified Linear Unit)

[0262] When the NNPA-RELU function is specified, for each element of the input tensor described by tensor descriptor 1, if that element is 0 or less, the corresponding element of the output tensor described by the output tensor descriptor is 0. Otherwise, the corresponding element of the output tensor is the minimum value between the elements of the input tensor and the clipping value specified by function-specific parameter 1.

[0263] For example, function-specific parameter 1 defines the clipping value for the RELU operation. For instance, the clipping value is bits 16-31 of function-specific parameter 1. The clipping value is specified, for example, in NNPA data type 1 format. A clipping value of zero indicates the use of the maximum positive value; in other words, no clipping is performed. If a negative value is specified, an exception for generic operand data is recognized.

[0264] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type specified in any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0265] In one example, the shape, data layout, and data type of input tensor 1 and output tensor are the same, but exceptions for generic operand data are recognized.

[0266] In one example, output tensor descriptor 2, input tensor descriptor 2, input tensor descriptor 3, and the function-specific storage area address field are ignored. In one example, function-specific parameters 2-5 include zero.

[0267] Function code 50: NNPA-TANH

[0268] When the NNPA-TANH function is specified, for each element of the input tensor described by tensor descriptor 1, the value of the corresponding element of the output tensor described by the output tensor descriptor becomes the hyperbolic tangent of that element.

[0269] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout = 0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type = 0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0270] In one example, the shape, data layout, and data type of input tensor 1 and output tensor are the same, but exceptions for generic operand data are recognized.

[0271] In one example, the output tensor descriptor 2, the input tensor descriptor 2, the input tensor descriptor 3, the function-specific parameters 1 to 5, and the function-specific storage area address field are ignored.

[0272] Function code 51: NNPA-SIGMOID

[0273] When the NNPA-SIGMOID function is specified, for each element of the input tensor described by the tensor descriptor 1, the corresponding element of the output tensor described by the output tensor descriptor becomes the sigmoid of that element.

[0274] In one example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout = 0), or if the data type of any of the specified tensor descriptors does not specify the NNP data type 1 (e.g., data type = 0), a response code, e.g., 0010 or 0011 in hexadecimal, is set in the general-purpose register 0, and the instruction completes with a condition code, e.g., 1.

[0275] In one example, the shapes, data layouts, and data types of the input tensor 1 and the output tensor are the same, but exceptions for general-purpose operand data are recognized.

[0276] In one example, the output tensor descriptor 2, the input tensor descriptor 2, the input tensor descriptor 3, the function-specific parameters 1 to 5, and the function-specific storage area address field are ignored.

[0277] Function code 52: NNPA-SOFTMAX

[0278] When the NNPA-SOFTMAX function is specified, for each vector in dimension 1 of the input tensor 1, the corresponding vector of the output tensor is calculated as described below.

[0279] * The maximum value of the vector is calculated.

[0280] *The sum of the exponential functions of the differences between each element of the dimension 1 vector and the maximum value calculated above is calculated. If both the element of the dimension 1 input vector and the maximum value calculated above are numerical, and their difference is not numerical, the result of the exponent for that element is forced to be zero.

[0281] *For each element in the vector, an intermediate quotient is formed by dividing the exponent of the difference between that element and the maximum value calculated above by the sum calculated above. An optional activation function is applied to this intermediate quotient to form the corresponding element of the output vector.

[0282] This process is repeated, for example, for all vectors with a dimensional 1 index size of dimensional 4 × dimensional 3 × dimensional 2.

[0283] For example, the function-specific parameter 1 of NNPA-SOFTMAX controls the activation function. For example, the ACT field of function-specific parameter 1 (e.g., bits 28-31) specifies the activation function. Examples of activation functions include the following:

[0284] ACT activation function

[0285] 0: Do not execute the activation function.

[0286] 1 LOG

[0287] 2-15 are booked.

[0288] If a reserved value is specified in the ACT field, for example, a response code of F001 in hexadecimal will be reported, and the calculation will be completed with a condition code of 1, for example.

[0289] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout = 0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type = 0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0290] For example, if the 3-dimensional index size of the input tensor is not equal to 1, a response code of F000 in hexadecimal might be stored, and the instruction might be completed with a condition code of 1.

[0291] In one example, the shape, data layout, and data type of input tensor 1 and output tensor are the same, but exceptions for generic operand data are recognized.

[0292] In one example, output tensor descriptor 2, input tensor descriptor 2, and input tensor descriptor 3 are ignored. In another example, function eigenparameters 2-5 include zero.

[0293] This function can use an 8KB function-specific storage area.

[0294] In one embodiment, when obtaining a 1-dimensional vector, the elements may not be contiguous in memory depending on the specified data layout format. If all elements of the 1-dimensional vector of input tensor 1 contain the largest negative number representable by the specified data type, the precision of the result may be reduced.

[0295] Function code 64: NNPA-BATCHNORM (Batch Normalization)

[0296] When the NNPA-BATCHNORM function is specified, for each vector in dimension 1 of the input 1 tensor, the corresponding vector in dimension 1 of the output tensor is calculated by multiplying each element of that vector by the corresponding element of the vector in dimension 1 that constitutes the input 2 tensor. Then, the full-precision product is added to the corresponding elements of the vector in dimension 1 that constitutes the input 3 tensor and rounded to the precision of the specified data type of the output tensor. This process is repeated, for example, for all vectors of dimension 1 of size dimension 4 index size x dimension 3 index size x dimension 2 index size.

[0297] In one example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout = 0), or if the data type in any of the specified tensor descriptors does not specify the NNP data type 1 (e.g., data type = 0), a response code, e.g., 0010 or 0011 in hexadecimal respectively, is set in general-purpose register 0 and the instruction completes with a condition code, e.g., 1.

[0298] In one example, the following conditions must be true, otherwise an exception for general-purpose operand data is recognized.

[0299] * The shapes and data layouts of the input tensor 1 and the output tensor are the same.

[0300] * The data types of the input tensor and the output tensor must be the same.

[0301] * The dimension 1 index sizes of the input tensors 1, 2, 3 and the output tensor must be the same.

[0302] * The index sizes of dimensions 2, 3, and 4 of the input tensors 2 and 3 are 1.

[0303] In one example, the output tensor descriptor 2 and the function-specific save area address field are ignored. In one example, the function-specific parameters 2 - 5 contain 0.

[0304] Function code 80: NNPA-MAXPOOL2D Function code 81: NNPA-AVGPOOL2D

[0305] When either the NNPA-MAXPOOL2D or NNPA-AVGPOOL2D function is specified, the input tensor 1, described by the input tensor 1 descriptor, is reduced by the specified operation, and the input window is summarized. The input window is selected by moving a 2D sliding window on dimension indices 2 and 3. The window summary is the elements of the output tensor. The dimensions of the sliding window are described, for example, by function-specific parameters 4 and 5. The amount the sliding window moves on the input tensor 1 when computing adjacent output tensor elements is called the stride. The stride of the sliding window is specified by function-specific parameters 2 and 3, etc. If the NNPA-MAXPOOL2D operation is specified, the Max operation defined below is performed on the window. If the NNPA-AVGPOOL2D operation is specified, the AVG operation defined below is performed on the window. If the specified padding type is valid, all elements in the window are added to the collection used to compute the resulting output elements. If the specified padding types are the same, depending on the window's position, only a subset of elements from the window may be added to the collection used to calculate the resulting output elements.

[0306] For example, the CollectElements operation adds an element to a collection of elements and increments the number of elements in the collection. The collection becomes empty each time the window's starting position moves. It's unpredictable whether elements not needed for the operation will be accessed.

[0307] Max operation: For example, the maximum value of a collection of elements in a window is calculated by comparing all elements in the collection with each other and returning the largest value.

[0308] AVG (Average) calculation: For example, the average value of a collection of elements within a window is calculated by dividing the sum of all elements in the collection by the number of elements in the collection.

[0309] In one example, the fields are assigned as follows:

[0310] *The pooling function-specific parameter 1 controls the padding type. For example, bits 29-31 of function-specific parameter 1 contain the PAD field, which specifies the padding type. Examples of types include the following:

[0311] PAD Padding type

[0312] 0 Valid

[0313] 1 Same

[0314] 2-7 Booked

[0315] If a reserved value is specified in the PAD field, for example, a response code of F000 in hexadecimal will be reported, and the calculation will be completed with a condition code, for example, 1.

[0316] In one example, bit positions 0-28 for function-specific parameter 1 are reserved and include zero.

[0317] *Function-specific parameter 2 includes, for example, a 32-bit unsigned binary integer that specifies the Dimension 2 stride (D2S), which specifies the number of moving elements in the sliding window in Dimension 2.

[0318] *Function-specific parameter 3 includes, for example, a 32-bit unsigned binary integer specifying the 3-dimensional stride (D3S), which specifies the number of elements the sliding window moves in 3 dimensions.

[0319] *Function-specific parameter 4 includes, for example, a 32-bit unsigned binary integer that specifies the dimension 2 window size (D2WS), which specifies the number of dimension 2 elements contained in the sliding window.

[0320] *Function-specific parameter 5 includes, for example, a 32-bit unsigned binary integer that specifies the Dimension 3 window size (D3WS), which specifies the number of Dimension 3 elements contained in the sliding window.

[0321] For example, the specified values ​​for function-specific parameters 2-5 are less than or equal to the maximum dimension index size, and the specified values ​​for function-specific parameters 4-5 are greater than zero. Otherwise, a response code, for example, 0012 in hexadecimal, is reported, and the operation is completed with a condition code, for example, 1.

[0322] If both the Dimension 2 stride and Dimension 3 stride are 0, and either the Dimension 2 window size or the Dimension 3 window size is greater than, for example, 1024, then a response code, for example, F001 in hexadecimal, is stored. If both the Dimension 2 stride and Dimension 3 stride are greater than, for example, 0, and either the Dimension 2 window size or the Dimension 3 window size is greater than, for example, 64, then a response code, for example, F002 in hexadecimal, is stored. If both the Dimension 2 stride and Dimension 3 stride are greater than, for example, 0, and either the Dimension 2 stride or the Dimension 3 stride is greater than, for example, 30, then a response code, for example, F003 in hexadecimal, is stored. If both the Dimension 2 stride and Dimension 3 stride are greater than, for example, zero, and either the Dimension 2 index size or the Dimension 3 index size of the input tensor is greater than, for example, 1024, then a response code, for example, F004 in hexadecimal, is stored. In all of the above conditions, the instruction completes with a condition code, for example, 1.

[0323] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0324] For example, the following condition must be true; otherwise, an exception for generic operand data will be recognized.

[0325] *The 4-dimensional index size and 1-dimensional index size of the input tensor and the output tensor must be the same.

[0326] *The data layout and data types of the input tensor and output tensor must be the same.

[0327] *If both the Dimension 2 stride and the Dimension 3 stride are zero, the following additional conditions may be met as an example.

[0328] *The dimensional 2 index size of the input tensor must be equal to the dimensional 2 window size.

[0329] *The 3D index size of the input tensor must be equal to the 3D window size.

[0330] *The 2-dimensional and 3-dimensional index sizes of the output tensor must be 1.

[0331] *The specified padding must be enabled.

[0332] *If either the Dimension 2 stride or the Dimension 3 stride is not 0, then both strides will be non-zero.

[0333] *If both the Dimension 2 stride and the Dimension 3 stride are greater than 0, the following additional condition may be met as an example.

[0334] *If specified padding is enabled, the 2D window size will be less than or equal to the 2D index size of the input tensor.

[0335] *If specified padding is enabled, the 3D window size will be less than or equal to the 3D index size of the input tensor.

[0336] *If the specified padding is the same, the 2D index size and 3D index size of the input and output tensors satisfy the following relationship (pooling the same padding):

[0337] TIFF0007872112000005.tif20140

[0338] TIFF0007872112000006.tif20140

[0339] Here,

[0340] IxDyIS: Dimension y of the input tensor x defined by the tensor descriptor x is the index size.

[0341] OxDyIS: Dimension y of the output tensor x defined by the tensor descriptor x is the index size.

[0342] D2S Dimension 2 Stride

[0343] D3S Dimensional 3-Stride

[0344] *If the specified padding is enabled, the 2-dimensional index sizes and 3-dimensional index sizes of the input and output tensors satisfy the following relationship (pooling of enabled padding):

[0345] TIFF0007872112000007.tif22160

[0346] TIFF0007872112000008.tif22160

[0347] Here, D2WS is the window size for dimension 2, and D3WS is the window size for dimension 3.

[0348] The output tensor descriptor 2, input tensor descriptors 2 and 3, and the function-specific storage area address field are ignored.

[0349] Function code 96: NNPA-LSTMACT (Long Short-Term Memory Activation)

[0350] When the NNPA-LSTMACT function is specified, the inputs to the LSTMACT operation are input tensor 1, described by input tensor 1 descriptor which divides the input tensor into four subtensors for each index value of dimension 4; input tensor 2, described by input tensor 2 descriptor which divides the input tensor into four subtensors for each index value of dimension 4; and input tensor 3, described by input tensor 3 descriptor. At the end of the LSTMACT operation, the results are written to output tensor 1, described by output tensor 1 descriptor, and output tensor 2, described by output tensor 2 descriptor.

[0351] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), then the response code 0010 or 0011 in hexadecimal is set to general-purpose register 0, and the instruction is completed with a condition code, e.g., 1.

[0352] In one embodiment, the following conditions must be true; otherwise, an exception for generic operand data is recognized.

[0353] *The input tensor is 3, and the output tensors are 1 and 2. The dimension is 4, and the index size is equal to, for example, 1.

[0354] *The 4-dimensional index size of input tensor 1 and input tensor 2 should be equal to, for example, 4.

[0355] *For example, the dimensional 3 index size of all input tensors and the two output tensors is equal to, for example, 1.

[0356] *For example, all input tensors and the two output tensors must have the same data layout and data type.

[0357] *For example, all input tensors and the two output tensors must have the same dimensional 1 index size.

[0358] *For example, all input tensors and the two output tensors must have the same dimensional 2 index size.

[0359] In one example, the function-specific storage area address field is ignored. In another example, function-specific parameters 1-5 include zeros.

[0360] Function code 97: NNPA-GRUACT (Gated Recurrent Unit Activation)

[0361] When the NNPA-GRUACT function is specified, the inputs to the GRUACT operation are input tensor 1, described by input tensor 1 descriptor which divides the input tensor into three subtensors for each dimensional 4 index value; input tensor 2, described by input tensor 2 descriptor which divides the input tensor into three subtensors for each dimensional 4 index value; and input tensor 3, described by input tensor 3 descriptor. At the end of the GRUACT operation, the output tensor described by the output tensor descriptor is stored.

[0362] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0363] In one embodiment, the following conditions must be true; otherwise, an exception for generic operand data will be recognized.

[0364] *The dimensional 4 index size of the output tensor and the input tensor 3 is, for example, equal to 1.

[0365] *The 4-dimensional index size of input tensor 1 and input tensor 2 should be equal to, for example, 3.

[0366] *For example, the dimensional 3 index size of all input and output tensors is equal to, for example, 1.

[0367] *For example, all input and output tensors must have the same dimensional 1 index size.

[0368] *For example, all input and output tensors must have the same dimensional 2 index size.

[0369] *For example, all input and output tensors must have the same data layout and data type.

[0370] In one example, the output tensor descriptor 2 and the function-specific storage area address field are ignored. In another example, function-specific parameters 2-5 include zero.

[0371] Function code 112: NNPA-CONVOLUTION

[0372] When the NNPA-CONVOLUTION function is specified, for each output element of the output tensor described by the output tensor 1 descriptor, a 3D input 1 window consisting of dimension indices 3, 2, and 1 is selected from input tensor 1 described by the input tensor 1 descriptor. A 3D input 2 window of the same size consisting of dimension indices 4, 3, and 2 is selected from tensor 2 described by the input tensor 2 descriptor. The elements of the input 1 window are multiplied with the corresponding elements of the input 2 window, and all products are added together to create an initial sum. This initial sum is added to the corresponding elements of input tensor 3 to calculate the intermediate sum. The elements of the output tensor are the result of applying the specified activation function to the intermediate sum. If no activation function is specified, the output elements are equal to the intermediate sum.

[0373] If the specified padding type is valid, all elements of the window are used to calculate the initial sum of the result. If the specified padding type is the same, depending on the window's position, some elements of the input 1 window may be treated as zero when calculating the resulting initial sum.

[0374] It is unpredictable whether elements that are not necessary for performing the operation will be accessed.

[0375] In one example, the fields of function-specific parameters used by a convolution function are assigned as follows:

[0376] *The NNPA-CONVOLUTION function-specific parameter 1 controls the padding type and activation function. As an example, bits 29-31 of function-specific parameter 1 include the PAD field, which specifies the padding type. Examples of types are shown below.

[0377] Padded type

[0378] 0 Valid

[0379] 1 Same

[0380] 2-7 Booked

[0381] If a reserved value is specified in the PAD field, for example, a response code of F000 in hexadecimal will be reported, and the calculation will be completed with a condition code of, for example, 1.

[0382] Furthermore, in one example, bits 24-27 of the NNPA-CONVOLUTION function-specific parameter 1 contain an activation field that specifies the activation function. An example of the function is shown below.

[0383] ACT activation function

[0384] 0: Do not execute the activation function.

[0385] 1 RELU

[0386] 2-15 are booked.

[0387] When the RELU activation function is specified, the resulting output element values ​​are determined as follows: if the intermediate sum is less than or equal to 0, the corresponding element of the output tensor is 0; otherwise, the corresponding element of the output tensor is the minimum value of the intermediate sum and the clipping value specified by the function-specific parameter 4.

[0388] If a reserved value is specified in the ACT field, for example, a response code of F001 in hexadecimal will be reported, and the calculation will be completed with a condition code of 1, for example.

[0389] *Function-specific parameter 2 includes, for example, a 32-bit unsigned binary integer and specifies the 2-dimensional (D2S) stride. The 2-dimensional (D2S) stride specifies, for example, the number of elements the sliding window moves in 2 dimensions.

[0390] *Function-specific parameter 3 includes, for example, a 32-bit unsigned binary integer that specifies the 3-dimensional (D3S) stride, which specifies the number of elements the sliding window moves across in 3 dimensions.

[0391] The values ​​specified by function-specific parameters 2 and 3 must be less than the maximum dimension index size. Otherwise, a response code such as 0012 in hexadecimal will be reported, and the operation will complete with a condition code such as 1.

[0392] *Function-specific parameter 4 defines the clipping value for the optional RELU operation. In one example, the clipping value is located in bits 16-31 of function-specific parameter 4.

[0393] For example, if the ACT field is zero, this field is ignored. If the ACT field specifies RELU, the clipping value is specified in NNP data type 1 format. A clipping value of zero indicates that the maximum positive value is used, in other words, no clipping is performed. If a non-zero value is specified, an exception for generic operand data is recognized.

[0394] In one example, if the specified data layout of a specified tensor descriptor other than input tensor 2 does not specify a 4D feature tensor (e.g., data layout=0), or if the specified data layout of input tensor 2 does not specify a 4D kernel tensor (e.g., data layout=1), then a response code, e.g., 0010 in hexadecimal, is set in general-purpose register 0, and the instruction completes with a condition code, e.g., 1. In another example, if the data type of a specified tensor descriptor does not specify an NNP data type 1 (e.g., data type=0), then a response code, e.g., 0011 in hexadecimal, is set in general-purpose register 0, and the instruction completes with a condition code, e.g., 1.

[0395] If both the Dimension 2 stride and the Dimension 3 stride are 0, and the Dimension 3 index size or Dimension 4 index size of input tensor 2 is greater than, for example, 448, then a response code, for example, F002 in hexadecimal, is stored. If both the Dimension 2 stride and the Dimension 3 stride are greater than 0, and either the Dimension 3 index size or Dimension 4 index size of input tensor 2 is greater than, for example, 64, then a response code, for example, F003 in hexadecimal, is stored, and the operation is completed with a condition code, for example, 1. If either the Dimension 2 stride or the Dimension 3 stride is greater than, for example, 13, then a response code, for example, F004 in hexadecimal, is stored, and the operation is completed with a condition code, for example, 1.

[0396] For example, the following condition must be true; otherwise, an exception for generic operand data will be recognized.

[0397] *The data layouts of input tensor 1, input tensor 3, and output tensor are assumed to be identical.

[0398] *All input and output tensors must have the same data type.

[0399] *The index sizes for dimensions 2, 3, and 4 of the input 3-tensor are 1.

[0400] * The 4-dimensional index size of the output tensor is equal to the 4-dimensional index size of the input 1 tensor.

[0401] * The dimensional 1 index size of the output tensor is equal to the dimensional 1 index size of the input 2 tensor and the dimensional 1 index size of the input 3 tensor.

[0402] *The dimensional 1 index size of the input tensor 1 must be equal to the dimensional 2 index size of the input tensor 2.

[0403] *If both the Dimension 2 stride and the Dimension 3 stride are zero, the following additional conditions may be met as an example.

[0404] *The dimensional 2 index size of the input 1 tensor must be equal to the dimensional 3 index size of the input 2 tensor.

[0405] *The input tensor's 1-dimensional, 3-index size is equal to the input 2-dimensional, 4-index size.

[0406] *The 2-dimensional and 3-dimensional index sizes of the output tensor are 1.

[0407] *The specified padding is valid.

[0408] *If either the Dimension 2 stride or the Dimension 3 stride is not 0, then both strides are considered non-zero.

[0409] *If both the Dimension 2 stride and the Dimension 3 stride are greater than 0, the following additional condition may be met as an example.

[0410] *If specified padding is enabled, the dimensional 2 index size of input tensor 1 must be greater than or equal to the dimensional 3 index size of input tensor 2.

[0411] *If specified padding is enabled, the dimensional 3 index size of the input tensor 1 must be greater than or equal to the dimensional 4 index size of the input tensor 2.

[0412] *If the specified padding is the same, the Dimension 2 index sizes and Dimension 3 index sizes of the input tensor and output tensor satisfy the following relationship as an example (convolution with the same padding):

[0413] TIFF0007872112000009.tif22152

[0414] TIFF0007872112000010.tif20152

[0415] Here,

[0416] O1D2IS Output tensor dimension 2 index size

[0417] O1D3IS Output Tensor Dimension 3 Index Size

[0418] I1D2IS Input 1 Tensor Dimension 2 Index Size

[0419] I1D3IS Input 1 Tensor Dimension 3 Index Size

[0420] D2S Dimension 2 Stride

[0421] D3S Dimensional 3-Stride

[0422] *When the specified padding is enabled, the relationship between the Dimension 2 index size and Dimension 3 index size of the input tensor 1, the Dimension 3 index size and Dimension 4 index size of the input tensor 2, and the output tensor is as follows (convolution with enabled padding):

[0423] TIFF0007872112000011.tif24139

[0424] TIFF0007872112000012.tif24139

[0425] Here,

[0426] O1D2IS Output tensor dimension 2 index size

[0427] O1D3IS Output Tensor Dimension 3 Index Size

[0428] I1D2IS Input 1 Tensor Dimension 2 Index Size

[0429] I1D3IS Input 1 Tensor Dimension 3 Index Size

[0430] I2D3IS Input 2 Tensor Dimension 3 Index Size

[0431] I2D4IS Input 2-tensor Dimension 4 Index Size

[0432] D2S Dimension 2 Stride

[0433] D3S Dimensional 3-Stride

[0434] In one example, the output tensor descriptor 2, function-specific storage area, and address area are ignored. In another example, function-specific parameter 5 includes zero.

[0435] Function code 113: NNPA-MATMUL-OP (Matrix Multiplication)

[0436] When the NNPA-MATMUL-OP function is specified, each element of the output tensor described by the output tensor descriptor is calculated as follows, for example:

[0437] *A 1-dimensional vector is selected from input tensor 1, which is described using input tensor 1 descriptors, using the get-dimension-1-vector operation described later.

[0438] *A 2-dimensional vector is selected from the input tensor 2 described by the input tensor 2 descriptor using the get-dimension-2-vector operation described later.

[0439] *The dot product of an intermediate dimension vector between a 1-dimensional vector and a 2-dimensional vector is calculated using the dot product operation described later.

[0440] The * operation is performed on the elements of input tensor 3 described by the input tensor 3 descriptor, using the same dimensional index 4 and dimensional index 1 values ​​as the output tensor elements, along with the intermediate dot product. The resulting elements are stored in the output tensor. The fusion operation is determined by function-specific parameter 1, which is described below.

[0441] Get-dimension-1-vector operation: For a given output element, a dimension 1 vector is selected from an input 1 tensor where input dimension 4 indices correspond to output dimension 4 indices, input dimension 3 indices correspond to output dimension 3 indices, and input dimension 2 indices correspond to output dimension 2 indices.

[0442] Get-dimension-2-vector operation: For a given output element, a dimension 2 vector is selected from two input tensors where input dimension 4 indices correspond to output dimension 4 indices, input dimension 3 indices correspond to output dimension 3 indices, and input dimension 1 indices correspond to output dimension 1 indices.

[0443] Dot product operation: The intermediate dot product of two vectors of the same size and data type is calculated as the sum of the products of each element of input vector 1 and the corresponding element of input vector 2.

[0444] Fusion Operation: Function-specific parameter 1 controls the operation performed on the intermediate dot product and the corresponding elements from the input tensor 3. For example, function-specific parameter 1 of NNPA-MATMUL-OP includes an operation field in bits 24-31, for instance. The operation field specifies the operation to be performed. An example operation is shown below.

[0445] Calculation Calculation type

[0446] 0 Add

[0447] 1. Compare when the dot product is high.

[0448] 2. Compare when the dot product is not low.

[0449] 3. Compare if the dot product and elements are equal.

[0450] 4. Dot product and comparison when elements are not equal.

[0451] 5. Compare when the dot product is not high.

[0452] 6. Compare when the dot product is low.

[0453] For example, in an addition operation, the three elements of the input tensor are added to the intermediate dot product. In a comparison operation, the three elements of the input tensor are compared to the intermediate dot product. If the comparison is true, the result is set to a value such as +1; otherwise, it is set to a value such as +0, using the data type specified for the output tensor.

[0454] In one example, all other values ​​in the OPERATION field are reserved. If a reserved value is specified in the OPERATION field, for example, a response code of F000 in hexadecimal will be reported, and the calculation will be completed with a condition code of 1, for example.

[0455] For example, if the data layout specified in any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type of any of the specified tensor descriptors does not specify an NNP data type 1 (e.g., data type=0), the response code, for example, 0010 or 0011 in hexadecimal, respectively, is set to general-purpose register 0, and the instruction completes with a condition code, for example, 1.

[0456] In one embodiment, the following conditions must be true; otherwise, an exception for generic operand data is recognized.

[0457] *All input and output tensors must have the same 4-dimensional index size.

[0458] *All input and output tensors have a 3-dimensional index size equal to 1.

[0459] *The size of the 2-dimensional index of the input tensor 3 is equal to 1.

[0460] *The input tensor 1 and the output tensor must have the same dimension 2 index size.

[0461] *The dimensional 1 index size of input tensor 1 and the dimensional 2 index size of input tensor 2 must be the same.

[0462] *The input tensor 2, input tensor 3, and output tensor must have the same dimensional 1 index size.

[0463] *All input and output tensors must have the same data layout and data type.

[0464] In one embodiment, the output tensor descriptor 2 and the function-specific storage area address field are ignored. Function-specific parameters 2-5 include zero as an example.

[0465] Function code 114: NNPA-MATMUL-OP-BCAST23 (Matrix Multiplication Operation - Broadcast23)

[0466] When the NNPA-MATMUL-OP-BCAST23 function is specified, each element of the output tensor described by the output tensor descriptor is computed as shown below as an example.

[0467] *A 1-dimensional vector is selected from input tensor 1, which is described by the input tensor 1 descriptor, using the get-dimension-1-vector operation described later.

[0468] *A 2-dimensional vector is selected from the input tensor 2 described by the input tensor 2 descriptor using the get-dimension-2-vector operation described later.

[0469] *The dot product of a 1-dimensional vector and a 2-dimensional vector is calculated using the dot product operation described later.

[0470] *Elements of input tensor 3 that have the same dimension index 1 value as the elements of the output tensor, as described by the descriptors of input tensor 3, are added to the previously calculated dot product and stored in the output tensor.

[0471] Get-dimension-1-vector operation: For a given output element, a dimension 1 vector is selected from an input 1 tensor where input dimension 4 indices correspond to output dimension 4 indices, input dimension 3 indices correspond to output dimension 3 indices, and input dimension 2 indices correspond to output dimension 2 indices.

[0472] Get-dimension-2-vector operation: For a given output element, a dimension 2 vector is selected from an input 2 tensor where the input dimension 4 index is 1, the input dimension 3 index is the output dimension 3 index, and the input dimension 1 index is the output dimension 1 index.

[0473] Dot product: The intermediate product of two vectors of the same size and data type is calculated as the sum of the products of each element of input vector 1 and the corresponding elements of input vector 2.

[0474] For example, if the data layout specified for any of the specified tensor descriptors does not specify a 4D feature tensor (e.g., data layout=0), or if the data type of the specified tensor descriptor does not specify an NNP data type 1 (e.g., data type=0), the response code, for example 0010 or 0011 in hexadecimal, is set to general-purpose register 0, and the instruction completes with a condition code, for example 1.

[0475] In one embodiment, the following conditions must be true; otherwise, an exception for generic operand data is recognized.

[0476] *The input tensor 1 and the output tensor must have the same dimension 4 index size.

[0477] *The size of the 4-dimensional index of input tensor 2 and input tensor 3 is equal to 1.

[0478] *All input and output tensors have a 3-dimensional index size equal to 1.

[0479] *The size of the 2-dimensional index of the input tensor 3 is equal to 1.

[0480] *The input tensor 1 and the output tensor must have the same dimension 2 index size.

[0481] *The dimensional 1 index size of input tensor 1 is the same as the dimensional 2 index size of input tensor 2.

[0482] *The input tensor 2, input tensor 3, and output tensor must have the same dimensional 1 index size.

[0483] *All input and output tensors must have the same data layout and data type.

[0484] In one embodiment, the output tensor descriptor 2 and the function-specific storage area address field are ignored. In one example, function-specific parameters 1 to 5 include zero.

[0485] In one embodiment of the neural network processing support instructions, if the output tensor overlaps with the input tensor or parameter block, the result is unpredictable.

[0486] A specification exception is recognized when attempting to execute a neural network processing support instruction, for example, if the parameter block is not specified by a double-word boundary.

[0487] Exceptions to general-purpose operand data are recognized when attempting to execute neural network processing support instructions, for example, if there is a tensor descriptor mismatch.

[0488] Condition codes obtained as a result of neural network processing support instructions include, for example, the following: 0 - Successful completion; 1 - Response code is set; 2 - --; 3 - Amount of processing data determined by the CPU.

[0489] In one embodiment, the execution priority of neural network processing support instructions includes, for example, the following:

[0490] 1.-7. Exceptions with the same priority as the general program interruption conditions.

[0491] 8.A Condition code 1 due to the specified function code being unassigned or not installed.

[0492] 8.B Specification exception due to parameter block not being specified with a double word boundary.

[0493] 9. Access exception for accessing a parameter block.

[0494] 10. Condition code 1, due to the specified format of the parameter block not being supported by the model.

[0495] 11.A Condition code 1 due to the specified tensor data layout not being supported.

[0496] 11.B Exceptions to generic operand data due to differences in data layout between tensor descriptors.

[0497] 12.A Condition code 1 based on conditions other than those included in 8.A, 10, 11.A above and 12.B.1 below.

[0498] 12.B.1 Condition code 1 due to invalid output tensor data type in NNPA-RELU and NNPA-CONVOLUTION.

[0499] 12.B.2 Exceptions for generic operand data when the values ​​of NNPA-RELU function-specific parameter 1 and NNPA-CONVOLUTION function-specific parameter 4 are invalid.

[0500] 13.A Access exception for accessing the output tensor.

[0501] 13.B Access exceptions for accessing input tensors.

[0502] 13. Access exceptions for accessing function-specific storage areas.

[0503] 14. Condition code 0.

[0504] As described herein, a single instruction (e.g., a neural network processing support instruction) is configured to execute multiple functions, including a query function and multiple non-query functions. Selected non-query functions, such as the NNPA-MATMUL-OP function and the NNPA-CONVOLUTION function, can execute a sequence of operations as part of a single function call, reducing the overhead associated with calling a processor, such as the neural network processor 105, for each operation in the sequence of operations, and improving performance by eliminating the need to store intermediate results of each operation outside the processor and reload those results as input to the next operation.

[0505] For example, a fully connected layer + batch norm / scale can be mapped to a matmul + biasadd compound function. The scaling and multiplication of the batch norm are performed with matrix multiplication weights, and the addition portion of the batch norm is performed via biasadd. This eliminates the need to save / reload intermediate data, as the addition portion of the batch norm is an element operation that can be performed directly on the result of matmul, for example. The execution time of the last two steps is eliminated, which speeds up accelerators that would otherwise need to save / reload data between each of these operations, for example.

[0506] Furthermore, in one example, Conv+BatchNorm+Scale+Activation can be mapped to a composite function such as Convolution+Biasadd+Activation, where the scaling and multiplication of Batchnorm are performed on the Convolution weights, and the addition part of Batchnorm is performed via Biasadd. This eliminates the need to save / reload intermediate data, as the addition part of Batchnorm is an elemental operation that can be performed directly on the convolution result before applying an activation function (e.g., Relu). This eliminates the execution time of the last three steps and speeds up accelerators that would otherwise need to save / reload data between each of these operations.

[0507] One or more aspects of the present invention are closely related to computer technology and facilitate processing within computers, thereby improving their performance. The use of a single architect machine instruction configured to perform various functions improves performance in computing environments by reducing complexity, reducing resource usage, and increasing processing speed. The use of a single function to implement a sequence of operations reduces overhead and resource usage, improving system performance. Instructions, functions, or operations, or combinations thereof, can be used in many technical fields such as computer processing, medical processing, engineering, automotive technology, and manufacturing. By providing optimizations, these technical fields are improved, for example, by reducing overhead, execution time, or both.

[0508] Further details of one embodiment that facilitates processing within a computing environment, relating to one or more aspects of the present invention, are described with reference to Figures 7A-7B.

[0509] Referring to Figure 7A, the compound function specified by the instruction is executed (700). The compound function includes multiple operations that are performed as part of a single call to the compound function (702). In one example, executing the compound function includes performing a convolution using a first tensor and a second tensor to obtain one or more intermediate results, the second tensor including, in one example, a tuned weight tensor created using multiple multipliers (704). The values ​​of the bias tensor are added to one or more intermediate results to obtain one or more results of the compound function (706).

[0510] By combining multiple operations into a single function, the number of times the processor is called to perform the operations is reduced. Furthermore, the need to store intermediate results in memory or another location accessible from one or more processors and then reload them is avoided. This improves processing speed, reduces system resource usage, and enhances overall performance.

[0511] In one example, executing a compound function further includes executing the selected activation on the result of one or more compound functions in order to provide one or more activation results of the selected activation (708). The one or more activation results of the selected activation are, for example, at least part of the output tensor (710).

[0512] In one embodiment, the compound function replaces several separately invoked operations (712). For example, the several separately invoked operations include convolution of an input tensor and a weight tensor, followed by batch normalization, followed by scaling, and followed by activation (714).

[0513] Batch normalization, in one embodiment, takes multiple inputs, including at least one convolution result of a convolution of an input tensor and a weight tensor, a selection multiplier, and a selection bias tensor, and uses the multiple inputs in batch normalization to provide at least one result (716). In one embodiment, the at least one result is stored in a select location visible to one or more processors externally (718), and batch normalization is an operation called separately from the convolution (720).

[0514] In one embodiment, referring to Figure 7B, at least one result and another selection multiplier are input to scaling, and scaling is an operation called separately from convolution and batch normalization (730). Scaling uses at least one result and another selection multiplier to reload at least one result stored in the selection location and provide at least one scaled result (732). At least one scaled result is stored in the selection location (734).

[0515] As an example, at least one scaled result is reloaded from the selected position and used as input for activation (736). Activation is an operation called separately from convolution, batch normalization, and scaling (738).

[0516] In one example, a tuned weight tensor is created (740), and the creation involves multiplying the weight tensor by several multipliers in order to provide a tuned weight tensor (742).

[0517] In one embodiment, one or more intermediate results are entered into the addition without one or more processors storing and reloading the one or more intermediate results in a location accessible from the outside (744).

[0518] As an example, referring to Figure 7C, performing a convolution involves selecting a first input window from one or more windows of the input tensor and a second input window from one or more windows of the adjusted weight tensor (750), multiplying the elements in the first input window by the corresponding elements in the second input window to obtain a multiplicative product (752), and adding the multiplicative products to obtain a sum (754).

[0519] Furthermore, in one example, adding values ​​of a bias tensor involves adding the values ​​of the corresponding elements of the bias tensor to the sum in order to provide another sum (756). This other sum is, for example, at least part of the output tensor of a compound function (758).

[0520] In one embodiment, performing the compound function further includes performing the selected activations on another sum to provide one or more results of the selected activations (760). The one or more results of the selected activations are at least part of the output tensor of the compound function (762).

[0521] As an example, performing a selected activation further includes determining whether other sums are in a pre-selection relationship with the selected value (770), and as a result of one or more outcomes, selecting the minimum value and clipping value of other sums based on other sums that have a pre-selection relationship with the selected value (772).

[0522] Other variations and embodiments are also possible.

[0523] Aspects of the present invention can be used in many types of computing environments. Another example of a computing environment incorporating one or more aspects of the present invention is described with reference to Figure 8A. As an example, the computing environment in Figure 8A is based on the z / Architecture® instruction set architecture provided by International Business Machines Corporation in Armonk, New York. However, the z / Architecture instruction set architecture is only one example of an architecture. Here again, the computing environment may be based on other architectures, including but not limited to the Intel® x86 architecture, other architectures of International Business Machines Corporation, or architectures of other companies, or a combination thereof. Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries.

[0524] In one example, the computing environment 10 includes a central electronic equipment complex (CEC) 11. The central electronic equipment complex 11 includes multiple components such as memory 12 (also known as system memory, main memory, primary memory, central memory, storage) coupled to one or more processors, such as one or more general-purpose processors (also known as central processing units (CPUs) 13) and one or more special-purpose processors (e.g., neural network processors 31), and an input / output (I / O) subsystem 14.

[0525] For example, one or more special-purpose processors may be separate from one or more general-purpose processors, or at least one special-purpose processor may be incorporated into at least one general-purpose processor, or both. Other variations are also possible.

[0526] The I / O subsystem 14 may be part of the central electronics complex or it may be separate. The I / O subsystem 14 directs the flow of information between the main memory 12 and the input / output (I / O) devices 16 coupled to the input / output control unit 15 and the central electronics complex.

[0527] Many types of I / O devices can be used. One particular type is a data storage device 17. The data storage device 17 can store one or more programs 18, one or more computer-readable program instructions 19, or data, or a combination thereof. The computer-readable program instructions can be configured to perform functions of embodiments of the present invention.

[0528] The central electronics complex 11 may include, or be combined with, a removable / non-removable, volatile / non-volatile storage medium for a computer system, or both. For example, it may include, or be combined with, a magnetic disk drive for reading from and writing to a non-removable non-volatile magnetic medium (typically called a “hard drive”), a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a “floppy disk”), or an optical disk drive for reading from and writing to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical medium, or a combination thereof, or both. It should be understood that other hardware components or software components, or both, may be used with the central electronics complex 11. Examples include, but are not limited to, microcode or millicode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0529] Furthermore, the central electronics complex 11 can operate in a number of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations or combinations thereof that are suitable for use with the central electronics complex 11 include, but are not limited to, personal computer (PC) systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0530] The central electronics complex 11 provides, in one or more embodiments, logical partitioning, virtualization support, or both. In one embodiment, as shown in Figure 8B, the memory 12 includes, for example, one or more logical partitions 20, a hypervisor 21 that manages the logical partitions, and processor firmware 22. An example of the hypervisor 21 is Processor Resource / System Manager (PR / SM®), provided by International Business Machines Corporation of Armonk, New York. PR / SM is a trademark or registered trademark of International Business Machines Corporation in at least one jurisdiction.

[0531] Each logical partition 20 can function as a separate system. That is, each logical partition can be independently reset and run a guest operating system 23, such as the z / OS® operating system provided by International Business Machines Corporation in Armonk, New York, or other control codes 24, such as Coupling Facility Control Code (CFCC), and run on different programs 25. An operating system or application program running on a logical partition may appear to have access to a complete and perfect system, but in reality, only a portion of it is available. While the z / OS operating system is provided as an example, other operating systems provided by International Business Machines Corporation, other companies, or both may be used according to one or more aspects of the present invention.

[0532] Memory 12 is coupled to, for example, a CPU 13 (Figure 8A), which is a physical processor resource that can be allocated to a logical partition. For example, a logical partition 20 may contain one or more logical processors, each logical processor representing all or a share of the physical processor resource 13 that can be dynamically allocated to the logical partition.

[0533] In yet another embodiment, the central electronics complex provides virtual machine support (with or without logical partitioning support). As shown in Figure 8C, the memory 12 of the central electronics complex 11 includes, for example, one or more virtual machines 26, a virtual machine manager such as a hypervisor 27 that manages the virtual machines, and processor firmware 28. An example of a hypervisor 27 is the z / VM® hypervisor provided by International Business Machines Corporation in Armonk, New York. A hypervisor is sometimes referred to as a host. z / VM is a trademark or registered trademark of International Business Machines Corporation in at least one jurisdiction.

[0534] The virtual machine support of the central electronics complex provides the ability to operate a large number of virtual machines 26, each virtual machine 26 running on a different program 29 and capable of running a guest operating system 30, such as the Linux® operating system. Each virtual machine 26 can function as an independent system; that is, each virtual machine can be independently reset, run a guest operating system, and run on a different program. The operating system or application program running within a virtual machine appears to have access to a complete and perfect system, but in reality, only a part of it is available. While z / VM and Linux are provided as examples, other virtual machine managers and / or operating systems may also be used according to one or more aspects of the present invention. The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, an exclusive licensee of Linus Torvalds, the worldwide trademark holder.

[0535] Another embodiment of a computing environment using one or more aspects of the present invention will be described with reference to Figure 9A. In this example, the computing environment 36 includes, for example, a native central processing unit (CPU) 37, memory 38, and one or more input / output devices or interfaces 39 or both, coupled to each other via, for example, one or more buses 40 or other connections or both. For example, the computing environment 36 may include an HP Superdome with a PowerPC® processor provided by International Business Machines Corporation in Armonk, New York, an Intel® Itanium® II processor provided by Hewlett Packard Co. in Palo Alto, California, or other machines or combinations based on architectures provided by International Business Machines Corporation, Hewlett Packard, Intel Corporation, Oracle, or others or a combination thereof. PowerPC is a trademark or registered trademark of International Business Machines Corporation in at least one jurisdiction. Itanium is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries.

[0536] The native central processing unit 37 includes one or more native registers 41, such as one or more general-purpose registers, one or more special-purpose registers, or both, used during processing within the environment. These registers contain information representing the state of the environment at any given point in time.

[0537] Furthermore, the native central processing unit 37 executes instructions and code stored in memory 38. In a particular example, the central processing unit executes emulator code 42 stored in memory 38. This code enables a computing environment configured on one architecture to emulate another architecture. For example, the emulator code 42 enables machines based on architectures other than the z / Architecture instruction set architecture, such as PowerPC processors and HP Superdome servers, to emulate the z / Architecture instruction set architecture and execute software and instructions developed based on the z / Architecture instruction set architecture.

[0538] Further details regarding the emulator code 42 are described with reference to Figure 9B. The guest instructions 43 stored in memory 38 include software instructions (e.g., those correlated with machine instructions) developed to run on architectures other than the native CPU 37 architecture. For example, a guest instruction 43 might be designed to run on a processor based on the z / Architecture instruction set architecture, but instead is emulated on the native CPU 37, which might be, for example, an Intel Itanium II processor. In one example, the emulator code 42 includes an instruction fetch routine 44 that retrieves one or more guest instructions 43 from memory 38 and optionally provides local buffering for the retrieved instructions. It also includes an instruction translation routine 45 that determines the type of retrieved guest instruction and translates the guest instruction into one or more corresponding native instructions 46. This translation includes, for example, identifying the function performed by the guest instruction and selecting a native instruction to perform that function.

[0539] Furthermore, the emulator code 42 includes an emulation control routine 47 for executing native instructions. The emulation control routine 47 can cause the native CPU 37 to execute a native instruction routine that emulates one or more previously obtained guest instructions, and upon completion of such execution, can return control to an instruction fetch routine that emulates obtaining the next guest instruction or set of guest instructions. Execution of native instructions 46 may include loading data from memory 38 into registers, storing data back from registers into memory, or performing some type of arithmetic or logical operation determined by the translation routine.

[0540] Each routine is implemented, for example, in software, stored in memory, and executed by the native central processing unit 37. In other examples, one or more routines or operations are implemented in firmware, hardware, software, or a combination thereof. Registers of the emulated processor can be emulated using the registers 41 of the native CPU or by using locations in memory 38. In embodiments, the guest instruction 43, native instruction 46, and emulator code 42 may reside in the same memory or may be distributed across different memory devices.

[0541] Instructions that can be emulated include neural network-assisted processing instructions described herein in accordance with one or more aspects of the present invention. Furthermore, other instructions, functions, operations, or one or more aspects, or combinations thereof, of neural network processing may be emulated in accordance with one or more aspects of the present invention.

[0542] The computing environments described above are merely examples of available computing environments. Other environments may be used, including but not limited to non-partitioned environments, partitioned environments, cloud environments, or emulated environments, or combinations thereof, and embodiments are not limited to any one of these environments. Although various examples of computing environments are described herein, one or more embodiments of the present invention may be used in many types of environments. The computing environments provided herein are for illustrative purposes only.

[0543] Each computing environment can be configured to include one or more aspects of the present invention.

[0544] One or more aspects relate to cloud computing.

[0545] This disclosure includes a detailed description of cloud computing, but the implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in any other type of computer environment that is currently known or may be developed in the future.

[0546] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0547] The characteristics are as follows:

[0548] On-demand self-service: Cloud consumers can unilaterally prepare computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0549] Broad network access: Computing power is available over the network and accessible through standard mechanisms. This facilitates use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, personal digital assistants (PDAs)).

[0550] Resource pooling: A provider's computing resources are pooled and delivered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated as needed. Generally, consumers have a sense of location independence because they do not manage or know the exact location of the resources provided. However, consumers may be able to identify the location at a higher level of abstraction (e.g., country, state, data center).

[0551] Rapid Elasticity: Computing power can be prepared quickly and flexibly, allowing it to scale out automatically and immediately, and to be quickly released and scale in immediately. To consumers, the computing power available for preparation often appears unlimited and can be purchased in any quantity at any time.

[0552] Measured Services: Cloud systems leverage metric capabilities at a certain level of abstraction, appropriate for the type of service (e.g., storage, processing, bandwidth, active user accounts), to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0553] The service model is as follows:

[0554] Software as a Service (SaaS): The functionality offered to consumers is the ability to use the provider's applications running on a cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., webmail). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for configuring a limited number of user-specific applications.

[0555] Platform as a Service (PaaS): The functionality offered to consumers is the ability to deploy applications they have created or acquired to cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, and storage, but they can control the deployed applications and, in some cases, the configuration of their hosting environment.

[0556] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provision of processors, storage, networking, and other basic computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, and deployed applications, and in some cases, partially control certain network components (e.g., host firewalls).

[0557] The deployment model is as follows:

[0558] Private Cloud: This cloud infrastructure is operated exclusively for a specific organization. This cloud infrastructure can be managed by that organization or a third party and can reside on-premises or off-premises.

[0559] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by the organization or a third party and can reside on-premises or off-premises.

[0560] Public Cloud: This cloud infrastructure is provided to a large number of people or large industry groups and is owned by organizations that sell cloud services.

[0561] Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community, or public). While maintaining the unique entities of each model, they are bound together by standards or individual technologies to achieve data and application portability (e.g., cloud bursting for load balancing across clouds).

[0562] Cloud computing environments are service-oriented environments that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0563] Referring to Figure 10, an exemplary cloud computing environment 50 is shown. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 52. Local computer devices used by cloud consumers (e.g., personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof) can communicate with these nodes. The nodes 52 can communicate with each other. The nodes 52 can be grouped physically or virtually (not shown) in one or more networks, such as the private, community, public, or hybrid clouds or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, without requiring cloud consumers to maintain resources on their local computer devices. Note that the types of computer devices 54A-N shown in Figure 10 are illustrative only, and it should be understood that the computing nodes 52 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network addressable connection (e.g., using a web browser) or both.

[0564] Referring to Figure 11, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 10) is shown. It should be understood that the components, layers, and functions shown in Figure 11 are illustrative only, and embodiments of the present invention are not limited to these. As illustrated, the following layers and corresponding functions are provided.

[0565] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include a mainframe 61, a reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage 65, and a network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0566] The virtualization layer 70 provides an abstraction layer. From this layer, for example, the following virtual entities can be provided: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0567] As an example, the management layer 80 can provide the following functions: Resource preparation 81 enables the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. As an example, these resources may include licenses for application software. Security enables not only protection of data and other resources but also identification and verification of cloud consumers and tasks. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables the allocation and management of cloud computing resources to ensure that requested service levels are met. Service Level Agreement (SLA) planning and execution 85 enables the pre-arrangement and procurement of cloud computing resources that are expected to be needed in the future in accordance with the SLA.

[0568] Workload layer 90 provides examples of the capabilities available in a cloud computing environment. Examples of workloads and capabilities available from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and neural network processing support processing 96.

[0569] The present invention may be a system, method, or computer program product or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium storing computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0570] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium may, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROM, DVD, memory stick, floppy disk, punch cards, or grooved raised structures, and mechanically encoded devices on which instructions are recorded, and suitable combinations thereof. The computer-readable storage medium as used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0571] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network consists of copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on the computer-readable storage medium within each computing / processing device.

[0572] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++ and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions are executable as a standalone software package, either entirely on the user's computer or partially on the user's computer. Alternatively, they may be executable partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by personalizing them using state information of computer-readable program instructions in order to perform aspects of the present invention.

[0573] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0574] These computer-readable program instructions can be provided to a computer processor or other programmable data processing device to generate a machine, such that instructions executed via the processor of the computer or other programmable data processing device generate means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram or both. These computer-readable program instructions can also be stored in a computer-readable storage medium that can be connected to a computer, a programmable data processing device, or other device or combination of devices that function in a particular way, such that the computer-readable storage medium on which the instructions are stored constitutes one of the outputs containing instructions that implement the modes of functions / operations specified in one or more blocks of a flowchart or block diagram or both.

[0575] Computer-readable program instructions, like instructions that perform a function / action specified in one or more blocks of a flowchart or block diagram or both on a computer, other programmable device, or other device, can also be loaded into a computer, other programmable data processing device, or other device and perform a series of operational steps on the computer, other programmable device, or other device to produce a computer-implemented process.

[0576] The flowcharts and block diagrams in the figures illustrate the configuration, function, and operation of executable implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction, which constitutes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions shown in the blocks may differ from the order shown in the figures. For example, two blocks shown consecutively may actually be achieved as a single step, executed simultaneously, substantially simultaneously, partially or entirely in overlapping time, or the blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart diagram, or both, and any combination of blocks in a block diagram or flowchart diagram, or both, can be implemented by a special-purpose hardware-based system that performs a specified function or operation, or a combination of special-purpose hardware and computer instructions.

[0577] In addition to the above, one or more aspects may be provided, offered, deployed, managed, serviced, etc. by a service provider that provides management of the customer environment. For example, a service provider may create, maintain, and support computer code or computer infrastructure, or both, that runs one or more aspects for one or more customers. In return, the service provider may receive payments from customers, for example, on the basis of a subscription or fee agreement or both. Furthermore, or alternatively, the service provider may receive payments from the sale of advertising content to one or more third parties.

[0578] In one embodiment, the application may be deployed to perform one or more embodiments. For example, the deployment of the application includes providing a computer infrastructure capable of operating to perform one or more embodiments.

[0579] In a further embodiment, the computing infrastructure may be deployed by integrating computer-readable code into a computing system, which, in combination with the computing system, can perform one or more embodiments.

[0580] In yet another embodiment, a process for integrating a computing infrastructure may be provided, which includes integrating computer-readable code into a computer system. The computer system includes a computer-readable medium, the computer medium includes one or more embodiments. The code combined with the computer system can perform one or more embodiments.

[0581] The various embodiments described above are merely illustrative. For example, one or more embodiments can be incorporated, used, or both, using computing environments of other architectures. Furthermore, different instructions, functions, or operations, or combinations thereof, may be used. Furthermore, different types of registers, different registers, or both may be used. Furthermore, other data formats, data layouts, or data sizes, or combinations thereof, may be supported. In one or more embodiments, one or more general-purpose processors, one or more special-purpose processors, or a combination of general-purpose and special-purpose processors may be used. Many modifications are possible.

[0582] This specification describes various embodiments. Furthermore, many modifications are possible without departing from the spirit of the embodiments of the invention. It should be noted that, unless in particular contradiction, each embodiment or feature described herein, and its variations thereof, can be combined with any other embodiment or feature.

[0583] Furthermore, other types of computing environments can also benefit and be used. For example, a data processing system suitable for storing or executing program code, or both, can be used, which includes at least two processors directly or indirectly coupled to a memory element via a system bus. The memory element may include, for example, local memory used during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage for at least some of the program code to reduce the number of times the code is retrieved from bulk storage during execution.

[0584] Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, DASDs, tapes, CDs, DVDs, thumb drives, and other memory media) can be coupled to the system directly or via an intermediary I / O controller. Network adapters can also be coupled to the system to enable data processing systems to connect with other data processing systems or remote printers or storage devices via an intermediary private or public network. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters available.

[0585] The terms used herein are for the sole purpose of describing specific embodiments and are not intended to be limiting. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context makes it clear otherwise. Where used herein, the terms “comprises,” “comprising,” or both, specify the presence of a described feature, integer, step, action, element, or component or combination thereof, but should be further understood that they do not preclude the presence or addition of one or more other features, integers, steps, actions, elements, components, or groups or combinations thereof.

[0586] All means or step-plus-function corresponding structures, materials, actions, and equivalents in the following claims are intended to include any structures, materials, or actions for performing a function in combination with elements of other claims specifically claimed. The descriptions of one or more embodiments are presented for illustrative and explanatory purposes, but are not intended to be exhaustive or to limit oneself to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments have been selected and described to best illustrate various aspects and applications, and to enable those skilled in the art to understand various embodiments with various modifications to suit a particular intended use.

Claims

1. A computer program for facilitating processing within a computing environment, wherein the computer program includes program instructions, and the instructions are: The execution of a compound function specified by an instruction, wherein the compound function includes a plurality of operations performed as part of a single call to the compound function, and the execution of the compound function is Performing a convolution using a first tensor and a second tensor to obtain one or more intermediate results, wherein the second tensor includes a tuned weight tensor created using multiple multipliers, To obtain the result of one or more composite functions, the value of a bias tensor is added to the one or more intermediate results, wherein the composite function replaces a plurality of separately invoked operations, the plurality of separately invoked operations including convolution of the input tensor and weight tensor, followed by batch normalization, followed by scaling, followed by activation, and so on. A computer program that includes [this].

2. The computer program according to claim 1, wherein executing the compound function further comprises performing the selected activations on the results of the one or more compound functions in order to provide one or more activation results of the selected activations, the one or more activation results of the selected activations being at least a portion of an output tensor.

3. The computer program according to claim 2, wherein the batch normalization takes a plurality of inputs, including at least one convolution result of the convolution of the input tensor and the weight tensor, a selection multiplier, and a selection bias tensor, and uses the plurality of inputs in the batch normalization to provide at least one result, the at least one result being stored in a select location visible to one or more processors, and the batch normalization is an operation called separately from the convolution.

4. The computer program according to claim 3, wherein at least one result and another selection multiplier are input to scaling, the scaling being an operation called separately from the convolution and the batch normalization, the scaling using the at least one result and the other selection multiplier to reload the at least one result stored in the selection location and provide at least one scaled result, the at least one scaled result being stored in the selection location.

5. The computer program according to claim 4, wherein the at least one scaled result is reloaded from the selected location and used as input to the activation, the activation being an operation called separately from the convolution, the batch normalization, and the scaling.

6. The computer program according to claim 1, wherein the method further comprises creating the adjusted weight tensor, the creation comprising multiplying the weight tensor by the plurality of multipliers in order to provide the adjusted weight tensor.

7. The computer program according to claim 1, wherein the one or more intermediate results are input to an addition without storing and reloading the one or more intermediate results in a location accessible from the outside by one or more processors.

8. Performing the aforementioned convolution means Selecting a first input window from one or more windows of the input tensor, and selecting a second input window from one or more windows of the adjusted weight tensor, To obtain multiple products, the elements in the first input window are multiplied by the corresponding elements in the second input window, In order to obtain the sum, the above multiple products are added together, The computer program according to claim 1, including the computer program described in claim 1.

9. The computer program according to claim 8, wherein adding the values ​​of the bias tensor comprises adding the values ​​of the corresponding elements of the bias tensor to the sum in order to provide another sum, the other sum being at least a portion of the output tensor of the composite function.

10. The computer program according to claim 9, wherein executing the compound function further comprises executing the selected activations on the other sum to provide one or more results of the selected activations, the one or more results of the selected activations being at least a portion of the output tensor of the compound function.

11. Performing the selected activation means To determine whether the selected activation is specified for the other sum, As a result of the one or more of the above results, the minimum value of the other sum and the clipping value are selected based on the other sum in which the selected activation is specified. The computer program according to claim 10, further comprising:

12. A computer system for facilitating processing within a computing environment, wherein the computer system is Memory and The computer system is configured to perform a method, and the method includes at least one processor that communicates with the memory, The execution of a compound function specified by an instruction, wherein the compound function includes a plurality of operations performed as part of one call to the compound function, and the execution of the compound function is Performing a convolution using a first tensor and a second tensor to obtain one or more intermediate results, wherein the second tensor includes a tuned weight tensor created using multiple multipliers, To obtain the result of one or more composite functions, the value of a bias tensor is added to the one or more intermediate results, wherein the composite function replaces a plurality of separately invoked operations, the plurality of separately invoked operations including convolution of the input tensor and weight tensor, followed by batch normalization, followed by scaling, followed by activation, and so on. A computer system, including a computer system.

13. The computer system according to claim 12, wherein the execution of the compound function further comprises performing the selected activations on the results of the one or more compound functions in order to provide one or more activation results of the selected activations, the one or more activation results of the selected activations being at least a portion of an output tensor.

14. The computer system according to claim 12 or 13, wherein the one or more intermediate results are input to an addition without storing and reloading the one or more intermediate results in a location accessible from the outside by one or more processors.

15. A computer implementation method for facilitating processing within a computing environment, wherein the computer implementation method is The execution of a compound function specified by an instruction, wherein the compound function includes a plurality of operations performed as part of one call to the compound function, and the execution of the compound function is Performing a convolution using a first tensor and a second tensor to obtain one or more intermediate results, wherein the second tensor includes a tuned weight tensor created using multiple multipliers, To obtain the result of one or more composite functions, the value of a bias tensor is added to the one or more intermediate results, wherein the composite function replaces a plurality of separately invoked operations, the plurality of separately invoked operations including convolution of the input tensor and weight tensor, followed by batch normalization, followed by scaling, followed by activation, and so on. Computer implementation methods, including those mentioned above.

16. The computer implementation method according to claim 15, wherein the execution of the compound function further comprises performing the selected activations on the results of the one or more compound functions in order to provide one or more activation results of the selected activations, the one or more activation results of the selected activations being at least a part of an output tensor.

17. The computer implementation method according to claim 15 or 16, wherein the one or more intermediate results are input to the addition without storing and reloading the one or more intermediate results in a location accessible from the outside by one or more processors.

Images