Hardware auto-loader

Abstract

A new apparatus and method for securely distributing an application to the processors of a processing unit. The processing unit is formed as part of an integrated circuit and includes a plurality of processors (referred to as tiles), each having their own execution unit and storage for storing application data and additional executable instructions. The integrated circuit includes a hardware module (referred to herein as an autoloader) configured to distribute a set of bootloader instructions (referred to herein as a secondary bootloader) to each of at least some of the tiles. Each of the tiles then executes the instructions of the received secondary bootloader, which causes each tile to issue a read request to read a set of executable application instructions from a memory external to the integrated circuit. Each tile then executes operations using the received set of executable application instructions in order to execute the application using the processing unit.

Description

Technical Field

[0001] This disclosure relates to providing applications to multiple processors of integrated circuits. Background Technology

[0002] When performing massively parallel operations, processing units comprising multiple processors can be provided on a single integrated circuit (i.e., a chip). Each processor is configured to perform the application's operations by executing a set of executable application instructions using a set of application data (e.g., the application's input variables).

[0003] An example of the use of processing units involving multiple processors can be found in the context of machine learning algorithms, such as deep neural networks. As is familiar to those skilled in the art of machine intelligence, machine intelligence algorithms are based on a “knowledge model,” which can be represented by a graph of multiple interconnected nodes. Each node represents a function of its input. Some nodes receive the input to the graph, and some nodes receive input from one or more other nodes, while the outputs of some nodes form the inputs of other nodes, and the outputs of some nodes provide the output of the graph (and in some cases a given node can even have all of these: the input to the graph, the output of the graph, and connections to other nodes). Furthermore, the function at each node is parameterized by one or more corresponding parameters (i.e., weights and biases).

[0004] Typically, at least some of the processing at each node can be executed independently of some or all other nodes in the graph, thus large graphs offer significant opportunities for concurrency and / or parallelism. Each processor in a computer can be used to execute processing associated with different nodes. In this way, multiple processors can be used collaboratively to train or operate a single neural network.

[0005] The processing associated with machine learning models can be divided into an operation phase and a training phase. During the training phase, the set of input data is processed using data that defines the state of the machine learning model to produce output values ​​for the model. The data defining the state of the machine learning model includes information indicating which nodes of the model are connected, as well as model parameters (including, for example, weights and biases) that additionally change during training. The set of output values ​​obtained during training is compared to the set of labels, and the model parameters are updated to tune the model to more accurately reproduce the labels from the set of input values. Once the machine learning model is trained, the set of input data is processed during the operation phase to produce output values ​​using the tuned parameters obtained during training.

[0006] The various sets of data used in the operation and training phases of a machine learning model (i.e., input data, information defining connections between nodes, model parameters, and labels) constitute application data, which is processed by one or more processors to perform training and / or operations on the machine learning model. A set of executable instructions must be loaded into a multiprocessor system before this application data can be used to perform operations. While loading the set of executable instructions into a multiprocessor system is a requirement in the machine learning context, it is not limited to this context and can occur in other types of application contexts. Summary of the Invention

[0007] When loading a set of executable instructions into a multiprocessor integrated circuit, a challenge is preventing untrusted instructions and data from being loaded into each processor. If a malicious third party wants access to each processor's memory, they could potentially install software into the processor, allowing them to access real application data provided to the processor by the tenant. In many cases, real application instructions and data are confidential and should be kept secret from malicious third parties. Therefore, it is necessary to ensure that only trusted application instructions are moved from external memory into each processor's memory.

[0008] According to a first aspect, an integrated circuit including a plurality of processors is provided, each of the plurality of processors including: at least one memory for storing application data and a set of executable application instructions; and at least one execution unit, wherein the integrated circuit includes a hardware module including a memory containing a set of executable boot instructions, wherein the hardware module includes processing circuitry configured to dispatch the set of executable boot instructions to at least some of the plurality of processors via interconnects of the integrated circuit, wherein for each of the at least some of the plurality of processors, the corresponding at least one execution unit is configured to: execute the received set of executable boot instructions to cause a read request to be issued to at least one memory outside the integrated circuit to obtain the set of executable application instructions; and execute the set of executable application instructions to perform operations using the application data.

[0009] Multiprocessor integrated circuits include a hardware module configured by trusted software to provide a bootloader to each processor in the processor set. In doing so, each processor is securely equipped with a trusted bootloader. This trusted bootloader ensures that each processor issues read requests to external memory locations storing the correct application instructions. This prevents processors from reading incorrect instructions, which could be from malicious third-party software. Therefore, the security of the integrated circuit is improved. Furthermore, this technology offers the advantage of enabling rapid application booting.

[0010] In some embodiments, for each of at least some of the plurality of processors: executing the received set of executable boot instructions includes calculating the address of external memory based on the identifier of the corresponding processor in the integrated circuit, such that a read request is issued includes issuing a read request to retrieve a set of executable application instructions from the calculated address in the external memory.

[0011] In some embodiments, the hardware module includes processing circuitry configured to dispatch one or more write requests to each of at least some of the plurality of processors, such that memory space not occupied by a set of executable boot instructions is cleared.

[0012] In some embodiments, for each of at least some of the plurality of processors: at least one corresponding execution unit is arranged such that checkpoint data generated during the execution of the corresponding set of executable instructions is dispatched to a storage device outside the integrated circuit in a write request.

[0013] In some embodiments, the processing circuitry of the hardware module is configured to, after the generated checkpoint data is dispatched, cause a set of executable boot instructions to be dispatched again via the interconnect of the integrated circuit to at least some of the plurality of processors, wherein for each of the plurality of processors, a corresponding at least one execution unit is configured to subsequently: execute the received set of executable boot instructions to cause a read request to be issued to a memory outside the integrated circuit to obtain a set of executable application instructions and another set of application data including the checkpoint data; and execute the set of executable application instructions to perform operations using the value of the checkpoint data.

[0014] In some embodiments, another set of application data includes a set of immutable data, wherein the immutable data is a portion of the application data acquired before the dispatch of checkpoint data, and wherein the other set of application data includes checkpoint data instead of variable data, wherein the variable data is a portion of the application data acquired before the dispatch of checkpoint data.

[0015] In some embodiments, for each of at least some of the plurality of processors, a corresponding execution unit is configured to execute a corresponding set of executable application instructions to load at least a portion of application data from at least one memory outside the integrated circuit.

[0016] In some embodiments, for each of at least some of the plurality of processors, a corresponding execution unit is configured to execute a set of received executable boot instructions such that a read request is issued to at least one memory outside the integrated circuit to obtain at least a portion of the application data.

[0017] In some embodiments, the hardware module includes volatile memory configured to store a set of executable boot instructions, wherein the processing circuitry of the hardware module is configured to, after a reset of the integrated circuit: receive the set of executable boot instructions from a device external to the integrated circuit; and store the received set of executable boot instructions in the volatile memory.

[0018] In some embodiments, receiving a set of executable boot instructions from a device external to the integrated circuit includes receiving a set of executable boot instructions via a JTAG interface.

[0019] In some embodiments, the hardware module includes non-volatile memory configured to store a set of executable boot instructions.

[0020] In some embodiments, the memory of the hardware module is configured to store a plurality of sets of executable boot instructions, wherein the processing circuitry is configured such that each set of the plurality of sets of executable boot instructions is assigned to a subset of the processor of the integrated circuit.

[0021] According to a second aspect, a data processing system is provided, comprising: an integrated circuit according to the first aspect; and a data providing system including a memory external to the integrated circuit.

[0022] In some embodiments, the data providing system includes at least one processor configured to load application data for a set of processors into a memory outside the integrated circuit in response to receiving a synchronization request from the integrated circuit at the data providing system.

[0023] In some embodiments, at least one processor of the data providing system is configured to arrange application data in a memory outside the integrated circuit based on an identifier of the processor group received from the integrated circuit.

[0024] According to a third aspect, a method is provided implemented in an integrated circuit including multiple processors, wherein a set of executable boot instructions is stored in a hardware module of the integrated circuit, such that the set of executable boot instructions is dispatched to at least some of the multiple processors via interconnects of the integrated circuit; on each of the at least some of the multiple processors, the received set of executable boot instructions is executed to cause a read request to be issued to a memory outside the integrated circuit to obtain a set of executable application instructions; and the set of executable application instructions is executed to perform operations using application data.

[0025] According to a fourth aspect, a computer program including computer-executable instructions is provided, which, when executed by processing circuitry of a hardware module comprising an integrated circuit including a plurality of processors, cause to perform a method comprising: storing a set of executable boot instructions in the hardware module of the integrated circuit, such that the set of executable boot instructions is dispatched via interconnects of the integrated circuit to at least some of the plurality of processors; on each of the at least some of the plurality of processors, executing the received set of executable boot instructions to cause a read request to be issued to a memory outside the integrated circuit to retrieve a set of executable application instructions; and executing the set of executable application instructions to perform an operation using application data.

[0026] According to a fifth aspect, a non-transitory computer-readable medium is provided for storing a computer program according to a fourth aspect. Attached Figure Description

[0027] To better understand the invention and illustrate how to implement it, reference will now be made to the accompanying drawings by way of example, wherein:

[0028] Figure 1 This is a schematic diagram of a multi-chip processor;

[0029] Figure 2 This is a schematic diagram illustrating the computation and switching stages within a multi-chip processor.

[0030] Figure 3 The diagram illustrates the data exchange in a massive synchronous parallel system.

[0031] Figure 4 The diagram illustrates the arrangement used to send data packets from the die to the outside of the chip.

[0032] Figure 5A The illustration depicts a die for distributing secondary bootloader code to a chip according to an embodiment.

[0033] Figure 5B The illustration shows a die for distributing secondary bootloader code to a chip according to yet another embodiment.

[0034] Figure 6 The diagram illustrates an example of the layout of data in the processor's memory.

[0035] Figure 7 The diagram illustrates the propagation of synchronization requests and acknowledgments.

[0036] Figure 8 The diagram illustrates the process of a chip reading application data from external memory.

[0037] Figure 9 The diagram illustrates the checkpoint settings for application data by the chip, and

[0038] Figure 10 This is a schematic block diagram of an example processor. Detailed Implementation

[0039] Embodiments of this application relate to a novel apparatus and method for securely distributing application instructions to a processor of a processing unit. The processing unit is formed as part of an integrated circuit and includes multiple processors (referred to as tiles), each processor having its own execution unit and memory for storing application data and executable application instructions. The integrated circuit includes a hardware module (referred to herein as an autoloader) configured to distribute a set of bootloader instructions (referred to herein as a secondary bootloader) to each of at least some of the tiles. Each tile then executes the received instructions from the secondary bootloader, causing each tile to issue a read request to read application instructions from memory outside the integrated circuit. Each tile then uses the received application instructions to perform operations on the application data to execute the application. The application data includes variables that can be loaded by executing the secondary bootloader or by executing the application instructions themselves.

[0040] The secondary bootloader instructions and the software used to configure the autoloader to deliver the secondary bootloader instructions to the chip are both trusted. This means they can be relied upon to enforce security policies, in this case, as a safeguard against loading malicious code into the chip. The system from which application instructions are loaded (the host in the described embodiment) is untrusted and cannot be relied upon to provide the correct application instructions to the chip without using a trusted secondary bootloader.

[0041] Embodiments of this application may be implemented using intelligent processing units (IPUs) described in our earlier U.S. Application No. 15 / 886315, the contents of which are incorporated herein by reference. Each of these IPUs is formed on a single integrated circuit. However, the invention is not limited to IPUs and may be implemented in other types of processing units.

[0042] Figure 1The example processing unit 2 is further illustrated. Figure 1 An example of a multi-die processing unit 2 is illustrated. The processing unit 2 includes an array 6 of multiple processor dies 4 and interconnects 34 connecting the dies 4. The processing unit 2 can be implemented individually as one of multiple dies packaged in the same IC package. The interconnects 34 may also be referred to herein as a “switching architecture” 34, as it enables the dies 4 to exchange data with each other. Each die 4 includes a corresponding instance of an execution unit and memory. For example, as an illustration, the processing unit 2 could comprise orders of hundreds or even more than a thousand dies 4. For completeness, it should also be noted that the term “array” as used herein does not necessarily imply any particular number of dimensions or physical layout of the dies 4.

[0043] In an embodiment, each processing unit 2 further includes one or more external links 8, enabling the processing unit 2 to connect to one or more other processing units (e.g., one or more other instances of the same processing unit 2). These external links 8 may include any one or more of the following: one or more processor-to-host links for connecting the processing unit 2 to a host system, and / or one or more processor-to-processor links for connecting one or more instances of the processing unit 2 together on the same IC package or card or on different cards. Multiple instances of the processing unit 2 can be connected together to the card via processing unit-to-processor links. The processing unit 2 receives work from external memory connected to it in the form of application data to be processed by the processing unit 2.

[0044] Interconnect 34 is configured to enable different chips 4 in array 6 to communicate with each other. However, in addition to dependencies that may exist between threads on the same chip 4, dependencies may also exist between parts of a program running on different chips 4 in array 6. Therefore, a technique is needed to prevent a piece of code on one chip 4 from running before the data it depends on becomes available through another piece of code on another chip 4.

[0045] Communication between chips 4 on processing unit 2 occurs in a time-deterministic manner. However, other forms of inter-chip exchange are also possible. Dependencies can exist between parts of a program running on different chips 4 in array 6. That is, processing data on one chip 4 may depend on the results from another chip 4, for example, the results that the other chip depends on may be provided. Therefore, a technique is needed to prevent a piece of code on one chip 4 from running before the data it depends on becomes available through another piece of code on another chip 4.

[0046] Parallel programming models used in AI and data science typically follow a three-phase iterative execution model: computation, barriers, and exchanges. This means that data transfers to and from processors often rely on barriers to provide data consistency between processors and between each processor and external storage devices. Commonly used data consistency models are massive synchronous parallelism (BSP), legacy synchronous parallelism (SSP), and asynchronous parallelism. The embodiments described in this paper use the BSP model, but it is clear that other synchronous models can be used as alternatives.

[0047] refer to Figure 2 and Figure 3 These illustrations depict implementations of a BSP switching scheme, where each chip 4 performs a computation phase 33 and a switching phase 32 in alternating cycles, separated from each other by inter-chip synchronization 30. Figure 2 and 3 In the illustrated scenario, a barrier synchronization is placed between each computation phase 33 and the subsequent exchange phase 32. During computation phase 33, each chip 4 performs one or more computational tasks locally on the chip, but does not communicate any results of these computations with any other chips in the chip 4. In exchange phase 32, each chip 4 is allowed to exchange one or more results of computations from previous computation phases to and / or from one or more other chip 4s, but does not perform any new computations until it receives any data from other chip 4s that its tasks(s) have dependencies on. It also does not send any data to any other chip 4 other than the data computed in the previous computation phase. Other operations, such as internal control-related operations, may be performed in exchange phase 32. Communication outside the chip group may optionally use a BSP mechanism, but alternatively, a BSP may not be used and it may use some other synchronization mechanism of its own.

[0048] According to the BSP principle, barrier synchronization 30 is placed at the junction of the transition from computation phase 33 to exchange phase 32, or at the junction of the transition from exchange phase 32 to computation phase 33, or both. That is, either: (a) all chips 4 are required to complete their respective computation phase 33 before any chip in the group is allowed to proceed to the next exchange phase 32, or (b) all chips 4 in the group are required to complete their respective exchange phase 32 before any chip in the group is allowed to proceed to the next computation phase 33, or (c) both conditions are enforced. In all three variations, individual chips alternate between phases, and the entire assembly is synchronized. The sequence of exchange and computation phases can then be repeated multiple times. In BSP terminology, each repetition of the exchange and computation phases is sometimes referred to as a “superstep” (but note that this term is not always used consistently in the literature: sometimes each individual exchange and computation phase is referred to as a superstep individually, while elsewhere, as in the terminology adopted herein, the exchange and computation phases are referred to together as a superstep).

[0049] It should also be noted that it is possible that multiple distinct independent groups of chips 4 on the same processing unit 2 or different processing units can each form separate corresponding BSP groups that operate asynchronously to each other, wherein the BSP cycles for computation, synchronization, and switching are imposed only within each given group, but each group does so independently of other groups. That is, the multi-chip array 6 may include multiple internal synchronization groups, each operating independently and asynchronously with other such groups (discussed in more detail later). In some embodiments, there is a hierarchical grouping of synchronization and switching, as will be discussed in more detail later.

[0050] Figure 3 The diagram illustrates the BSP principle implemented in groups 4i, 4ii, and 4iii of some or all chips in array 6 under the condition of imposing: (a) barrier synchronization from computation phase 33 to exchange phase 32 (see above). Note that in this arrangement, some chips 4 are allowed to begin computation 33 while others are still exchanging.

[0051] Communication between the chips 4 of processing unit 2 occurs in a time-deterministic manner, wherein data packets are transmitted without headers. This is explained in our earlier U.S. Patent Application No. 15 / 886315.

[0052] Figure 4An exemplary mechanism for sending data packets from the chip to a destination outside processing unit 2 is illustrated. This mechanism is non-deterministic in time. The mechanism is implemented in dedicated hardware logic within external interconnect 72. Data is sent via external interconnect 72 in the form of packets. Unlike packets sent via internal interconnect 34, these packets have headers: because the order of transmission can change, they require the destination address to be presented in the packet header. External interconnect 72 includes routing tables for statically routing data packets between different processors based on the packet headers.

[0053] At the physical layer, the interconnect mechanism is lossy, but at the service layer, due to the link layer architecture, the mechanism is not lossy: if a packet is not acknowledged, it will be automatically retransmitted by the hardware in the interconnect 72. However, the possibility of loss and retransmission at the data link layer means that the delivery of data packets through the external interconnect 72 is not time-deterministic. Furthermore, all packets in a given exchange can arrive together or separately in any time order, thus requiring flow control and queuing for external interconnects. Additionally, the interconnect can use Clock Data Recovery (CDR) techniques to infer the clock from the received data stream with sufficient data signal transition to maintain bit lock. The phase relationship between this inferred clock and the transmit clock is unknown, thus representing an additional source of nondeterminism.

[0054] As shown in the figure, the external interconnect 72 includes an external switching block (XB) 78. The compiler specifies that one of the chips 4 sends an external switching request (XREQ) to the switching block 78 (operation S1). The XREQ is a message including one or more control packets indicating which of the chips 4 have the data packets (content) to be sent. This is in... Figure 4The diagram is illustrated using checkmarks and crosses: as an example scenario, those marked with checkmarks have data packets to send externally, while those marked with crosses do not. In operation S2, switching block 78 sends a switching open (XON) control packet to the first of the chips 4 that has data to send externally. This causes the first chip 4 to begin sending its packets to the relevant destination via external interconnect 72 (operation S3). The routing table in external interconnect 72 statically routes the data packets received from the first chip at the external interconnect to the destination. If at any time XB 78 is unable to continue sending packets to the interconnect (e.g., due to previous packet loss and retransmissions in the interconnect, or due to oversubscription of the external interconnect by many other XBs and chips), then XB 78 will send a switching close (XOFF) to that chip 4 before the XB 78 queue overflows. Once the congestion is cleared and XB 78 has enough space in its queue again, it will send an XON to chip 4, allowing it to continue transmitting its content. Once chip 4 has sent its last data packet, in operation S4, switching block 78 sends a switching off (XOFF) control packet to chip 4, and then in operation S5 sends another XON and the data packet to be sent to the next chip 4, and so on. The XON and XOFF signaling is implemented as a hardware mechanism in dedicated hardware logic in the form of external switching block 78.

[0055] Although Figure 4 Only a single switching block 78 is shown, but multiple different switching blocks can exist, each serving a subset of the chips 4 in the processing unit 2. In one embodiment, for example, the integrated circuit forming the processing unit 2 includes eight switching blocks, each performing operations on one-eighth of the chips 4 in the processing unit. Figure 4 The functions shown are as follows.

[0056] Each processor chip 4 includes processing circuitry and memory. In some example embodiments, the processing circuitry is a multi-threaded processor 10. Figure 10An example of a processor chip 4 according to an embodiment of the present disclosure is illustrated. The processor chip 4 includes a multi-threaded processor 10 in the form of a bucket-thread processor 10 and local memory 11 (i.e., on the same chip in the case of a multi-chip array, or on the same chip in the case of a single-processor chip). The bucket-thread processor 10 is a type of multi-threaded processor 10 in which the execution time of the pipeline is divided into a repetitive sequence of interleaved time slots, each time slot potentially owned by a given thread. This will be discussed in more detail later. The memory 11 includes an instruction memory 12 and a data memory 22 (which may be implemented in different addressable memory units or in different regions of the same addressable memory unit). The instruction memory 12 stores machine code to be executed by the processing unit 10, while the data memory 22 stores data to be manipulated by the executed code and data output by the executed code (e.g., as a result of such operations).

[0057] Memory 12 stores various threads of a program, each thread comprising a corresponding sequence of instructions for performing one or more specific tasks. Note that, as mentioned herein, instructions refer to machine code instructions, that is, instances of one of the basic instructions in the processor's instruction set, consisting of a single opcode and zero or more operands.

[0058] Within processor 10, multiple different threads from instruction memory 12 can be interleaved via a single execution pipeline 13 (although typically only a subset of all threads stored in instruction memory can be interleaved at any given point in the program). Multithreaded processor 10 includes: multiple context register files 26, each arranged to represent the state (context) of a different corresponding thread among the threads to be executed concurrently; a shared execution pipeline 13, which is shared by the concurrently executing threads; and a scheduler 24 for scheduling concurrent threads for execution in an interleaved manner, preferably in a round-robin manner, via the shared pipeline. Processor 10 is connected to a shared instruction memory 12 shared by multiple threads, and a shared data memory 22 also shared by multiple threads.

[0059] Execution pipeline 13 includes a fetch stage 14, a decode stage 16, and an execution stage 18. Execution stage 18 includes execution units capable of performing arithmetic and logical operations, address calculations, load and store operations, and other operations defined by the instruction set architecture. Each of the context register files 26 includes a corresponding set of registers used to represent the program state of the corresponding thread.

[0060] refer to Figure 5AThe diagram illustrates integrated circuit 51, in which secondary bootloader code is distributed to multiple chips 4 of integrated circuit 51. Integrated circuit 51 includes the processing unit 2 discussed above. Integrated circuit 51 also includes hardware module 52, which is referred to herein as autoloader 52. Autoloader 52 is defined as the master bootloader for integrated circuit 51. Autoloader 52 performs the function of providing secondary bootloaders to the multiple chips 4 and erasing the remaining areas of the chip memory.

[0061] The autoloader 52 includes a memory 53 storing one or more sets of instructions that can be executed by the chip 4 to fetch application instructions from external storage. Each boot instruction in the one or more sets of boot instructions is referred to as a secondary bootloader or secondary bootloader image. The same secondary bootloader is loaded into multiple different chips 4. In some cases, the same secondary bootloader may be loaded into all chips 4 in the processing unit 2. In other cases, a primary bootloader may be loaded into a first set of chips 4, while one or more additional bootloaders are loaded into other chips 4 in the processing unit 2. In some cases, some chips 4 do not receive any secondary bootloaders. Chips 4 that do not receive any secondary bootloaders will not fetch application instructions from external memory.

[0062] To deliver a secondary bootloader to one of the chips 4, the processing circuitry 54 of the autoloader 52 loads the secondary bootloader from memory 53 and processes it to generate one or more data packets. Each of the one or more data packets contains secondary bootloader code in its payload. Each data packet contains an identifier of the relevant chip 4 to which the secondary bootloader will be assigned in its header. The header also includes an address in block memory indicating the location in the memory of the identified chip to which the secondary bootloader will be written. The processing circuitry 54 dispatches the relevant one or more data packets to a switching block 78 via the interconnects of integrated circuit 51. The switching block 78 converts the packets into an appropriate format for transmission to the relevant chip 4 via the interconnects of processing unit 2. The switching block 78 dispatches the one or more data packets to the chip 4 indicated in the address in the header. Upon receiving the one or more data packets, the processing circuitry of chip 4 processes the data packets to extract the secondary bootloader code and stores it in memory.

[0063] The autoloader 52 can be configured to write secondary bootloader code to different subsets of chips 4 in integrated circuit 51. For example, chips 4 can be divided into 32 different subsets, and the autoloader 52 is programmed in a configuration register that provides the secondary bootloader to the subset. When the secondary bootloader is assigned to a subset of chips, the autoloader 52 can issue a write-to-memory instruction to each chip 4 in the first subset of the secondary bootloader to the location in memory of each chip 4 in that subset. The autoloader 52 then loops through the subset of chips again, this time by writing incremental chip addresses. The autoloader 52 writes the second portion of the secondary bootloader to the next free location in memory for each chip 4 in the subset. The autoloader 52 continues in this manner until the bootloader has been written to all chips 4 in the subset.

[0064] refer to Figure 6 The illustration shows an example of the layout of data and instructions retrieved into memory 11 within the chip memory 11. Memory 11 does not need to be physically contiguous memory space. The first portion of memory 11 is a reserved area. The secondary bootloader occupies a portion of the reserved area, and the remainder of the reserved area does not contain any data or code, i.e., it is blank. In this embodiment, the reserved area is 16KB, and the secondary bootloader occupies approximately 1KB of the reserved area. In addition to the reserved area, memory 11 also includes memory space that can be used to store application instructions and data retrieved from external memory.

[0065] After the secondary bootloader is written to chip 4, the autoloader 52 is then configured to clear the remaining chip memory 11. By resetting the chip memory 11 in this way, any data belonging to the previous tenant of processing unit 2 that has been persisted after a reset or power cycle will be removed. The autoloader 52 performs this clearing by dispatching write packets to write to all chip memory space except for all block memory space occupied by the secondary bootloader code. The dispatched packets are of the same type as the packets dispatched by the autoloader 52 to write the secondary bootloader to the chip memory, but instead of including the code of the secondary bootloader to be written to the chip memory 11, the packets include a series of zeros at the locations in the chip memory 11 indicated in the packet header to be written to. Thus, for each of the plurality of chips 4 to which the secondary bootloader code is written, the autoloader 52 dispatches a series of packets to write zeros to other locations in the chip memory 11 that do not include the secondary bootloader.

[0066] As noted, the secondary bootloader may not be written to each chip 4 in processing unit 2. Therefore, for those chips 4 where the secondary bootloader is not written, the autoloader 52 will write zeros to all block memories 11. These chips 4, whose entire memory 11 is blank, will not be used during application processing.

[0067] The clearing of the chip memory 11 performed by the autoloader 52 causes each chip in the chip 4 that received the secondary bootloader to begin executing the secondary bootloader.

[0068] exist Figure 5A In the illustrated embodiment, autoloader 52 receives one or more secondary bootloaders from device 55 external to integrated circuit 51. Device 55 is referred to herein as a Smart Cryptographic Unit (ICU). ICU 55 includes or is associated with non-volatile memory (such as flash memory). On the other hand, autoloader memory 53 is volatile memory, such as SRAM. Thus, the non-volatile memory associated with ICU 55 stores one or more secondary bootloaders, making the secondary bootloader(s) available to chip 4 after integrated circuit 51 is powered off and restarted. After integrated circuit 51 restarts, ICU 55 provides one or more secondary bootloaders to autoloader 52 of integrated circuit 51. ICU 55 performs sequential writes via an interface to provide one or more secondary bootloaders to integrated circuit 51. ICU 55 provides one or more secondary bootloaders to autoloader 52 via JTAG interface of integrated circuit 51. The one or more secondary bootloaders are then stored in volatile memory 53 of autoloader 52 before being provided to chip 4.

[0069] Compared to the on-chip interconnect of the memory of chip 4, which allows autoloader 52 to write to the memory of chip 4, the interface provided by ICU 55 for writing the second bootloader code to memory 53 has high latency. Therefore, it would be slow for ICU 55 to write the bootloader code individually to each chip 4. By having ICU 55 write the bootloader code to autoloader 52, and then having the autoloader provide that bootloader code to multiple different chips 4 via a high-speed on-chip interconnect, a significant efficiency gain is achieved.

[0070] refer to Figure 5BThe illustration shows an alternative example of integrated circuit 51a. In this case, integrated circuit 51a includes CCU 56. CCU 56 acts as an autoloader and includes memory 53a and processing circuitry 54a. Memory 53a performs the same functions discussed above for memory 53, i.e., memory 53a stores secondary bootloaders to be distributed to multiple chips 4. Processing circuitry 54a performs the same functions discussed above for processing circuitry 54, including executing instructions to distribute secondary bootloaders to multiple chips 4 and clearing chip memory. Processing circuitry 54a is configured by trusted software to issue write requests to write secondary bootloaders to the memory of chip 4. Memory 53a includes non-volatile memory 53a configured to store a second bootloader such that the secondary bootloader is not lost from memory 53a when integrated circuit 51a is powered off. Therefore, in this embodiment, external devices such as ICU 55 are not required to store secondary bootloader code and provide it to integrated circuit 51a.

[0071] Figure 5B The other elements shown are in conjunction with Figure 5A The corresponding components shown operate in the same way.

[0072] refer to Figure 7 and 8 The illustration shows how application instructions are retrieved from external memory 70 once the secondary bootloader has been written to multiple chips 4.

[0073] In this example, external memory 70 is shown as part of host system 71. However, in other examples, system 71 may be a gateway that interfaces integrated circuit 51 with the host system. System 71 may also be another type of system that includes external memory 70.

[0074] System 71 includes at least one processor 711 for loading application instructions from memory 70. The at least one processor 711 may be a processor configured to execute computationally readable instructions.

[0075] Chip 4, including the secondary bootloader, is divided into different synchronization groups, each of which retrieves its application instructions during a separate exchange phase. For all chips 4 in integrated circuit 51, there can be four different synchronization groups. Different ways exist for chip 4 to identify its synchronization group. In some cases, different secondary bootloader code can be assigned to chips 4 belonging to different synchronization groups. In other cases, the same secondary bootloader code can be assigned to chips belonging to different synchronization groups, but when executed on each chip 4, the secondary bootloader code uses the chip ID of chip 4 to determine which synchronization group chip 4 belongs to.

[0076] Each of the first synchronization group's chips 4 sends a synchronization request 73 to the switching block 78. Each synchronization request 73 includes an indication of the synchronization group to which the requesting chip 4 belongs, or is preceded by that indication. The switching block 78 stores an indication of the number of synchronization requests 73 expected for that synchronization group. The switching block 78 receives the synchronization requests 73 and determines when it has received that number of expected synchronization requests 73 for that synchronization group. Once it has received that number of expected synchronization requests 73, the switching block aggregates these synchronization requests into a synchronization request 74 to be sent via interface 75. Interface 75 converts the received synchronization requests (in packet form for transmission via the interconnect of integrated circuit 51) into PCIe packets for transmission to system 71.

[0077] Upon receiving synchronization request 74, system 71 loads application instructions into memory 70 for delivery to chip 4 belonging to the synchronization group that issued synchronization request 73. Synchronization request 74 contains an indication of the associated synchronization group. System 71 loads the application instructions required by chip 4 of that synchronization group into memory 70 according to the indication of the synchronization group. Instructions are loaded from storage device 710 into memory 70. Storage device 710 may be the data providing system 71 or may be separate from system 71. Therefore, memory 70 can be understood as a data transfer memory, in which instructions are preloaded before being retrieved by the relevant chip 4.

[0078] The memory 70 is arranged into multiple different address spaces or buffers from which the chip reads data. Each of these address spaces corresponds to a different data stream read by the chip. The address spaces do not need to be contiguous memory spaces, but can be virtual buffers.

[0079] Once the application instructions for transmission to chip 4 have been loaded into memory 70, system 71 issues a synchronization acknowledgment 76, which is returned to switching block 78. In response to receiving synchronization acknowledgment 76, switching block 78 dispatches synchronization acknowledgments 79 to all chips 4 in the synchronization group.

[0080] refer to Figure 8 The diagram illustrates the steps that occur after the synchronization acknowledgment 79 returns to chip 4 in processing unit 2. Upon receiving the synchronization acknowledgment, each chip 4 waits to receive control from switching block 78 in order to send a read request packet. This is based on the above regarding... Figure 4 The given description occurs. Each chip in the synchronization group dispatches a read request 81 to the 4-way exchange block 78. For simplicity of explanation, in Figure 8The diagram shows only one read request 81 originating from a single chip 4. A switching block 78 passes the read request 81 to an interface 75. The read request 81 is translated into a PCIe read request by the interface 75. The interface 75 then passes the read request 81 to the system 71. Each read request 81 includes an indication of the address in memory 70 it targets. In response to the read request 81, the processor of the system 71 generates one or more data packets 82, which include data read from memory 70 at the location indicated in the read request 81. These one or more generated data packets 82 are referred to as read completions. The system 71 returns one or more read completions 82 to the integrated circuit 51. Upon receiving one or more read completions 82 at the switching block 78, the switching block 78 provides the read completions to the chip 4 from which the read request 81 originated. Control is then passed to the next chip 4 in the synchronization group, which issues its read request to read from a specific location in memory 70 and then receives one or more completions in response.

[0081] After each chip 4 in a synchronization group reads its application instructions from memory 70, chip 4 in another synchronization group will issue a synchronization request. These synchronization requests are aggregated by the switching block and provided to system 71. In response to receiving this next synchronization request, system 71 loads the relevant application instructions into memory 70 for delivery to the chip 4 in the corresponding synchronization group. System 71 then returns a synchronization confirmation to integrated circuit 51, which causes the chip 4 in the synchronization group to issue a read request to read from memory 70.

[0082] The process of synchronous request / acknowledgment and reading from memory 70 continues until all chips 4 with secondary bootloaders have loaded their application instructions from memory 70.

[0083] Each chip in chip 4 that loads application instructions from memory 70 uses a chip identifier (chip ID) of chip 4 stored in a storage device (different from memory 11) within chip 4 to determine the address to be read from it. Since multiple chips 4 receive the same secondary bootloader, the secondary bootloader is configured to load instructions from memory 70 at locations dependent on the chip ID. The secondary bootloader calculates the address from which to read from memory 70 based on the ID of the chip on which it operates.

[0084] As described above, application instructions are executed by the chip to perform operations using application data (e.g., variables). This application data can be loaded into chip 4 simultaneously with and in the same manner as the application instructions, i.e., by executing secondary bootloader instructions to issue read requests to load data from the host 71's memory 70. Additionally or alternatively, application data can be loaded by chip 4, which executes the application instructions themselves to issue read requests to load application data from the host 71's memory 70. In particular, secondary bootloader instructions can be used to issue read requests to load a portion of invariant application data (e.g., hyperparameters for a machine learning model), while application instructions can be used to issue read requests to load variable portions of the data (e.g., training data for a machine learning model).

[0085] During application runtime, if the application running on processing unit 2 fails and needs to be restarted from a certain point, it may be desirable to set a checkpoint for some data that can be read back into chip 4. The secondary bootloader can be used to load the checkpoint data.

[0086] When the application runs on processing unit 2, some of the application data stored in chip memory 11 is modified. The application data stored in chip memory 11 is divided into variable data and invariant data. Variable data includes variables that are modified during application runtime, such as the weights of a neural network. Invariant data includes data that does not change during application runtime, such as data defining which nodes are connected in a neural network.

[0087] In order to perform application checkpoints, the chip group, before issuing a write command to write data to system 51, uses a similar mechanism. Figure 7 The synchronization request is issued in the manner shown. Chip 4 of processing unit 2 issues a write request containing variable data generated during execution.

[0088] refer to Figure 9 The illustration shows chip 4 of integrated circuit 51 writing checkpoint data to memory 70 accessible by system 71. The write 91 includes one or more data packets 91 issued by chip 4. The one or more data packets 91 contain checkpoint data to be written to memory 710. The write is provided to system 71. At least one processor 711 of system 71 causes the checkpoint data to be written to memory 710. The checkpoint data is identified as associated with the chip 4 from which the write is received. Other chips 4 of integrated circuit 51 will similarly provide writes of their variable data, which are written to memory 710.

[0089] Later, when chip 4 of integrated circuit 51 requests to load checkpoint data (e.g., due to application failure), the secondary bootloader is loaded by the autoloader onto those chips 4, as described above. Figure 5AAs discussed in / 5B, the remaining area of ​​the chip memory is erased. Chip 4 then issues synchronization requests, which are aggregated and provided to system 71. In response to the receipt of the synchronization requests, system 71 loads application data from storage device 710 into memory 70. Unlike when the application is initially started, loading application data from storage device 710 in this case includes loading checkpoint data written by chip 4. The application data loaded into memory 70 includes checkpoint data, which is variable data written to storage device 710 by chip 4, and additionally includes previously loaded immutable data. System 71 arranges the application data in memory 70 in the same layout as the initial application data used to load into memory 70. Therefore, the same secondary bootloader used by chip 4 to load the initial application data can also be used to load the application data, including the checkpoint data. In some cases, if desired, the secondary bootloader used to load the application data with the checkpoint set can be different from the secondary bootloader used to initially load the application instructions.

[0090] Once system 71 has loaded the application data, including checkpoint data, into memory 70, system 71 sends an acknowledgment to chip 4, which issued the synchronization request. Chip 4 then sends a read request to memory 70 to read the application data, including checkpoint data, from memory 70. This reading of the application data, including checkpoint data, is related to the above... Figure 8 The application instructions discussed are executed using the same initial read mechanism. Once chip 4 has read in the data, including checkpoint data, they again begin executing the application from the point where the checkpoint was obtained.

[0091] It should be recognized that the above embodiments have been described by way of example only.

Claims

1. An integrated circuit comprising a plurality of processors, each of the plurality of processors comprising: At least one memory for storing a set of application data and executable application instructions; as well as At least one execution unit, The integrated circuit includes a hardware module comprising a memory containing a set of executable boot instructions, and processing circuitry configured by trusted software during operation such that the set of executable boot instructions is dispatched to at least some of the plurality of processors via the interconnects of the integrated circuit. For each of at least some of the plurality of processors, at least one corresponding execution unit is configured as follows: The received set of executable boot instructions is executed to cause a read request to be issued to at least one memory outside the integrated circuit to obtain a set of executable application instructions; as well as A set of executable application instructions is executed to perform operations using the application data. For each of at least some of the plurality of processors: Executing the received set of executable boot instructions includes calculating the address of external memory based on the identifier of the corresponding processor in the integrated circuit. The process of issuing a read request includes a set of actions that cause a read request to be issued to retrieve executable application instructions from a computed address in external memory.

2. The integrated circuit of claim 1, wherein the hardware module includes processing circuitry configured such that one or more write requests are dispatched to each of at least some of the plurality of processors to clear memory space not occupied by a set of executable boot instructions.

3. The integrated circuit of claim 1, wherein for each of at least some of the plurality of processors: The corresponding at least one execution unit is arranged such that checkpoint data generated during the execution of the corresponding set of executable application instructions is dispatched to a storage device outside the integrated circuit in a write request.

4. The integrated circuit of claim 3, wherein the processing circuitry of the hardware module is configured to, after the generated checkpoint data has been dispatched, cause a set of executable boot instructions to be dispatched again via the interconnect of the integrated circuit to at least some of the plurality of processors. Wherein, for each of at least some of the plurality of processors, the corresponding at least one execution unit is configured to subsequently: The received set of executable boot instructions is executed to cause a read request to be issued to a memory external to the integrated circuit to obtain a set of executable application instructions and another set of application data including the checkpoint data; and A set of executable application instructions is executed to perform operations using the values ​​of the checkpoint data.

5. The integrated circuit of claim 4, wherein the other set of application data includes a set of invariant data, wherein the invariant data is a portion of the application data acquired before the checkpoint data is dispatched. The other set of application data includes checkpoint data instead of variable data, wherein the variable data is a portion of the application data acquired before the checkpoint data is dispatched.

6. The integrated circuit of claim 1, wherein for each of at least some of the plurality of processors, the corresponding at least one execution unit is configured to execute a corresponding set of executable application instructions to load at least a portion of the application data from the at least one memory outside the integrated circuit.

7. The integrated circuit of claim 1, wherein for each of at least some of the plurality of processors, the corresponding at least one execution unit is configured to execute a set of received executable boot instructions such that a read request is issued to at least one memory outside the integrated circuit to obtain at least a portion of the application data.

8. The integrated circuit of claim 1, wherein the hardware module includes volatile memory configured to store a set of executable boot instructions, wherein the processing circuitry of the hardware module is configured to, after a reset of the integrated circuit: Receive a set of executable boot instructions from a device outside the integrated circuit; and The set of received executable boot instructions is stored in the volatile memory.

9. The integrated circuit of claim 7, wherein receiving the set of executable boot instructions from a device external to the integrated circuit includes receiving the set of executable boot instructions via a JTAG interface.

10. The integrated circuit of any one of claims 1 to 6, wherein the hardware module includes a non-volatile memory configured to store a set of executable boot instructions.

11. The integrated circuit of claim 1, wherein the memory of the hardware module is configured to store a plurality of sets of executable boot instructions, wherein the processing circuitry is configured such that each set of the plurality of sets of executable boot instructions is assigned to a subset of the processor of the integrated circuit.

12. A data processing system, comprising: The integrated circuit as described in claim 1; as well as A data providing system, which includes a memory external to the integrated circuit.

13. The data processing system of claim 12, wherein the data providing system includes at least one processor configured to, in response to receiving a synchronization request from the integrated circuit at the data providing system, load application data for a set of processors into a memory external to the integrated circuit.

14. The data processing system of claim 13, wherein the at least one processor of the data providing system is configured to arrange the application data in a memory outside the integrated circuit according to the identifiers of the group of processors received from the integrated circuit.

15. A method implemented in an integrated circuit including multiple processors, the method comprising: The hardware module of the integrated circuit stores a set of executable boot instructions. During runtime, trusted software configures the processing circuitry of the hardware module such that a set of executable boot instructions is distributed to at least some of the plurality of processors via the interconnection of the integrated circuits; On each of at least some of the plurality of processors, a set of received executable boot instructions is executed such that a read request is issued to a memory outside the integrated circuit to obtain a set of executable application instructions; as well as Execute a set of executable application instructions to perform operations using application data. In each of at least some of the plurality of processors: Executing the received set of executable boot instructions includes calculating the address of external memory based on the identifier of the corresponding processor in the integrated circuit. The process of issuing a read request includes a set of actions that cause a read request to be issued to retrieve executable application instructions from a computed address in external memory.

16. A computer program comprising computer-executable instructions, which, when executed by processing circuitry of a hardware module comprising an integrated circuit of a plurality of processors, cause the processing circuitry to perform: The hardware module of the integrated circuit stores a set of executable boot instructions, and At runtime, trusted software configures the processing circuitry of the hardware module such that a set of executable boot instructions is distributed to at least some of the plurality of processors via the interconnects of the integrated circuits. The executable boot instructions are configured such that, when executed on at least some of the plurality of processors, a read request is issued to a memory external to the integrated circuit to retrieve a set of executable application instructions for execution on at least some of the plurality of processors to perform operations using application data. In each of at least some of the plurality of processors: Executing the received set of executable boot instructions includes calculating the address of external memory based on the identifier of the corresponding processor in the integrated circuit. The process of issuing a read request includes a set of actions that cause a read request to be issued to retrieve executable application instructions from a computed address in external memory.

Images