Device for controlling computing devices of a neural network inference accelerator

Abstract

The invention relates to the field of computing means, and more particularly to microprocessors. A device for controlling the computing devices of a neural network inference accelerator contains the following elements disposed on a chip: a parameter configuration unit connected to an external control device; a segmented buffer memory connected to an external memory; at least two units for processing instruction streams, each of which is configured to be capable of buffering a stream of instructions received from the segmented buffer memory and decoding and synchronizing the instruction stream; a descriptor pointer unit configured to be capable of receiving descriptor pointers from the external control device; a unit for managing requests to read descriptors according to pointers from the descriptor pointer unit; a unit for processing descriptors of an instruction set; an instruction stream scheduler configured to be capable of loading an instruction set and monitoring the fullness of segments of the buffer memory. The invention makes it possible to enhance accelerator performance by reducing the idle time of computing cores.

Description

CONTROL DEVICE FOR COMPUTING DEVICES OF A NEURAL NETWORK INFERENCE ACCELERATOR AREA OF TECHNOLOGY

[0001] The claimed technical solution generally relates to the field of microprocessors, and in particular to a device for controlling the computing cores of a neural network inference accelerator. LEVEL OF TECHNOLOGY

[0002] With the development and implementation of machine learning and artificial intelligence algorithms in all areas of life, specialized devices designed for training and inference of neural networks, such as neural network inference accelerators, neural processors, etc., have recently undergone significant development.

[0003] There are several classes of accelerators, which determine the organization of control of the computing cores of the said accelerators. Thus, there are accelerators with a GPU-like architecture, containing, as a rule, a 2D array of specialized processor cores with a vector ALU, and accelerators of a non-GPU-like (heterogeneous architecture), containing a set of computing cores optimized and designed for various operations, for example, such accelerators may contain computing cores for matrix multiplications (a computing unit for matrix multiplications), a vector processor, devices for permutations and shifts of data bits in microprocessors (RU Patent No. RU 2488161 C1), calculators of special mathematical operations, etc. Accelerators of neural network inference with a heterogeneous architecture are preferable for such tasks as the inference and training of neural networks, image processing, etc., since they are built to perform parallel processing of large volumes of data.

[0004] In such devices (heterogeneous neural network inference accelerators), as a rule, instruction sets for computing cores (hardware computing devices included in the accelerator) are stored in external memory, relative to the control device, the reading latency of which can range from tens to hundreds of cycles depending on the organization of the memory itself and PC17RU2024 / 000312 access to it. The load on each of the computing cores (computing units) of the devices—the number of executed instructions per inference cycle (the load profile on the command interface)—is typically uneven and will vary depending on the specific neural network. This unevenness must typically be compensated for in the control unit, which loads instructions from external memory and sends them to the computing cores, which can be combined into computing clusters.

[0005] Thus, to compensate for delays in reading external memory, instruction buffers (queues) are added to the control device circuits of accelerators, the arbitration of which is also performed by the control device, for example, as disclosed in the prior art, in US Patent No. US 11113101 B2 (C AVIUM LLC [US] et al.), published 09 / 07 / 2021.

[0006] The disadvantages of this approach include significant limitations in the organization of instruction storage and arbitration for the accelerator's computing units, due to the fact that the memory used to organize the queues has a fixed size and may be insufficient to compensate for delays in the case of non-deterministic access times to external instruction memory and uneven loads on the computing units (uneven instruction flow across the computing cores). Furthermore, using equal-sized queues in such conditions requires large memory capacity with low utilization due to uneven loads, which consequently increases the area of ​​such control units on the chip.

[0007] Accordingly, the goal of this technical solution is to create a control device for neural network inference accelerator computing devices that overcomes the aforementioned shortcomings and improves accelerator performance by reducing the idle time of the accelerator's computing cores. Furthermore, by implementing double buffering, the claimed device enables flexible operation according to the workload profile under non-deterministic memory access conditions with limited device resources, which also reduces the total memory capacity and the device's on-chip area. DISCLOSURE OF THE INVENTION

[0008] The claimed technical solution proposes a new approach to the architecture of the control device for the computing devices of the neural network inference accelerator, which ensures a reduction in the area of ​​the control device, as well as an increase in the performance of the neural network inference accelerator, by reducing the idle time of the computing cores of the said accelerator, as well as by reducing the total memory volume in the control device.

[0009] The technical result achieved by solving this problem is a reduction in the area of ​​the control device of the neural network inference accelerator on a chip while maintaining the performance of the control device.

[0010] The specified technical result is achieved thanks to a device for controlling the computing devices of a neural network inference accelerator, which contains the following elements located on a crystal: • a parameter configuration unit associated with an external control device, configured to receive data containing configuration parameters for the control device; • a segmented buffer memory connected to an external memory, configured to receive and store instruction sets, wherein each segment has a dynamic memory volume set by the received configuration parameters, and is configured to store a set of instructions for at least one computing device of the neural network inference accelerator; • at least two instruction flow processing units connected to at least two computing devices of the neural network inference accelerator, each of which is configured to: o buffer the instruction flow received from the segmented buffer memory; o decode the received instruction flow; o synchronize the instruction flow; PC17RU2024 / 000312 about sending a decoded instruction stream to at least one computing device of a neural network inference accelerator; • a block of descriptor pointers configured to receive descriptor pointers from an external control device, wherein the descriptor pointers contain the address of the location in the external memory and the size of the descriptor of the instruction set for at least one computing device; • a block for managing requests for reading descriptors using pointers from a block of descriptor pointers, configured to obtain from external memory descriptors of a set of instructions for at least one computing device based on a descriptor pointer; • a unit for processing instruction set descriptors, configured to allocate the address of the location and size of the instruction set of the computing device, as well as the parameters of the control device for the executable output of the neural network; • an instruction flow scheduler configured to load a set of instructions, based on a processed descriptor, from external memory into at least two segments of segmented buffer memory and to monitor the fullness of the segments of said memory. [OOP] In one particular embodiment, the computing device of the neural network inference accelerator is a computing core of the neural network inference accelerator or a cluster of computing cores of the neural network inference accelerator.

[0012] In another particular embodiment, the configuration parameters represent at least one of: a configuration of the segmented buffer memory; a configuration of the descriptor processing request control unit; a configuration of the descriptor processing unit; a configuration of each of the instruction flow processing units.

[0013] In another particular embodiment, the instruction flow scheduler is configured to load a set of instructions in portions.

[0014] In another particular embodiment, the portioned loading of a set of instructions is a sequential loading of individual parts of a set of instructions into a segment of buffer memory, the size of which does not exceed the free volume of the memory segment in the segmented buffer memory.

[0015] In another particular implementation, each part of the instruction set is marked with a tag corresponding to the buffer memory segment number.

[0016] In another particular implementation, each buffer memory segment is a circular buffer with a configurable memory size.

[0017] In another particular implementation, the descriptor pointer block is a circular buffer (FIFO).

[0018] In another particular embodiment, the descriptor pointer contains at least the address of the location of the descriptor of the instruction set for the computing device of the neural network inference accelerator in the external memory and the size of the specified descriptor.

[0019] In another particular implementation, the instruction set contains an identifier of the instruction recipient in the neural network inference accelerator.

[0020] In another particular embodiment, buffering of instructions in at least two instruction flow processing units is carried out in an internal circular buffer with a data bus width equivalent to the data bus of a segmented buffer memory.

[0021] In another particular implementation, the decoded instruction stream is sent as a data packet.

[0022] In another particular embodiment, when sending a decoded instruction stream, the instruction stream processing unit is additionally configured to count the number of instructions sent and detect the last instruction.

[0023] In another particular embodiment, the instruction flow processing unit further comprises a synchronization unit configured to pause and resume sending instructions upon receipt of a corresponding pause or resume command. BRIEF DESCRIPTION OF DRAWINGS

[0024] The features of the claimed technical solution and a detailed description are given below in the form of attached drawings.

[0025] Fig. 1 illustrates the structural diagram of the control device for the computing devices of the neural network inference accelerator.

[0026] Fig. 2 illustrates the structural diagram of a segmented buffer memory.

[0027] Fig. 3 illustrates an example of the distribution of instruction sets in a segmented buffer memory.

[0028] Fig. 4 illustrates the structural diagram of the instruction flow processing unit. IMPLEMENTATION OF THE INVENTION

[0029] Below we will describe the concepts and terms necessary for understanding this technical solution.

[0030] A neural network inference accelerator is a specialized class of hardware accelerators designed to accelerate algorithms for artificial neural networks, computer vision, voice recognition, machine learning, and other artificial intelligence methods. Neural network inference accelerators belong to a class of heterogeneous (non-GPU-like) architectures that include matrix multiplication cores and vector computation units. In addition to these cores, the system may also include data permutation units, such as bit permutation and shift units in microprocessors, memory read / write units, special mathematical operations units, and others. The accelerator is controlled by a control unit located within the accelerator.

[0031] The term "inference" of a neural network in this solution should be understood as a continuous process of operation of a trained neural network to obtain a logical conclusion.

[0032] The proposed technical solution offers a new approach to creating a control device for neural network inference accelerator computing units, featuring high-speed control of the accelerator's computing cores and a small chip footprint. The architecture of the proposed technical solution enables a reduction in memory capacity. Control units, and, as a result, a reduction in the device's on-chip area due to double buffering of instruction queues and the use of segmented buffer memory. These features also allow for additional adaptation to the workload profile under conditions of non-deterministic memory access with limited device resources.

[0033] The term "instructions" as used in this application may generally refer to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, performing calculations in the computing cores of an accelerator, buffering data in the memory of a control device, etc. The instructions may be implemented in a variety of ways, including, for example, streaming configuration of machine instructions, object-oriented methods, etc. The instructions that implement the processes described in this solution may be transmitted via wired channels from external devices (hosts) to the control device, for example, the control device of the inference and training accelerator of neural networks, as well as from the control device to the computing cores of the accelerator.

[0034] Fig. 1 shows a block diagram of a control device for computing devices of a neural network inference accelerator 100. Said device 100 includes a parameter configuration unit 110, a segmented buffer memory 120, at least two instruction flow processing units 130, a descriptor pointer unit 140, a descriptor reading request control unit 150, an instruction set descriptor processing unit 160, an instruction flow scheduler 170, a module of counters of executed instructions by computing devices 180, an instruction set execution control unit 190, and an instruction request arbiter 200.

[0035] Device 100 is primarily part of neural network inference accelerators. Thus, device 100 is configured to provide an external (with respect to the accelerator and device 100) control device with a register interface for controlling the accelerator and reading status registers; processing descriptors representing an indication of the address and size of instruction sets for the accelerator's computing devices or clusters of computing devices; reading instruction sets of computing devices from external memory according to the description in the descriptors, and buffering them, processing and sending commands to computing devices via command interfaces; processing commands to control execution dependencies (stopping the supply of command packets to computing devices and waiting for events); ensuring the operation of command-by-command debugging modes, unloading debug information, performance and error counters, and interrupt handling.

[0036] It is worth noting that in this solution, the term neural network inference accelerator computing devices is equivalent to the term neural network inference accelerator computing cores and can be used interchangeably.

[0037] Also, in another particular embodiment, the device 100 is configured to send a set of instructions to a group of computing devices of the neural network inference accelerator, united into a single cluster.

[0038] The elements of device 100 are located on an integrated circuit (IC) chip. In one particular embodiment, the IC chip may be part of another IC chip, such as a neural network accelerator, which also contains other components associated with device 100 (a vector processor, a matrix computing unit, a data permutation unit, etc.).

[0039] Parameter configuration unit 110 is a hardware unit consisting of simple computing elements. Said unit 110 is connected to an external control device and is configured to receive data containing configuration parameters for the control device. Said unit 110 is designed to receive and transmit configuration parameters of device 100 to the corresponding elements of device 100. That is, said unit 110 is designed to receive an array of control and status registers for setting the operating modes of device 100 elements. Thus, configuration parameters may be received from an external control device, for example, in the form of a data file, a set of instructions, etc.The configuration parameters are at least one of: the configuration parameters of the segmented buffer memory 130; the configuration parameters of the descriptor processing request control unit 150; the configuration parameters of the descriptor processing unit 160; the configuration parameters of each of the instruction flow processing units 130. Thus, for example, the configuration parameters of the memory 130 may be the allocated sizes of the memory segments before the start of operation of the device 100, for example, by specifying them in the configuration parameters. The addresses of the allocated memory segment for the corresponding computing device. The parameters of block 150 may represent the priority of the sequence of reading descriptors from external memory, etc.

[0040] It should be noted that the load profile of each accelerator computing unit may vary depending on the type of neural network being executed. Accordingly, the configuration parameters are intended to pre-configure device 100 for operation in a specific mode, for example, in a mode corresponding to the type of neural network. The frequency of retrieving the configuration parameters may depend on the executed actions of the accelerator computing units. For example, the configuration parameters may be retrieved once before the accelerator executes a neural network inference operation. In another particular embodiment, the configuration parameters may be modified in accordance with changes in the operations executed by the computing units. For example, when a new type of neural network is executed by a matrix computing unit or a vector processor, device 100 will receive new configuration parameters corresponding to the specified neural network.Obtaining configuration parameters is performed via a communication interface with an external control device, such as a host.

[0041] Segmented buffer memory 120 is a fixed-size SRAM memory with one read port and one write port. This memory 120 is shown in more detail in Fig. 2. Accordingly, all segments are implemented on this memory 120. The size of each segment is specified by configuration parameters and depends on the workload profile of the accelerator's computing unit to which the instruction stream is destined. The workload profile can be understood as the frequency and volume of instructions addressed to a specific accelerator's computing unit. Thus, depending on the neural network being executed, certain computing units are utilized differently. The workload profile on device 100 will be proportional to the workload of the computing unit.

[0042] The segments of the buffer memory 120 represent a dynamically allocated physical memory volume from the total memory volume 120. The use of one physical memory 120 with dynamically configurable segments compared to the use of several physical memories for each segment is due to the fact that the large memory volume has a denser layout (bit / um 2 ), which significantly reduces the required area compared to several smaller memories (with the same total volume).

[0043] Each segment has the following programmable parameters: the segment start address in shared segmented memory and the segment end address in shared segmented memory. These addresses are used to construct a FIFO-type circular buffer. Each instruction packet arriving from the external control device to segmented memory 120 is tagged with a segment identifier for the packet's intended segment. This tag is used to increment the write pointers pointing to the SRAM addresses to which the write occurs. A request to unload instructions into the instruction stream processing units is also tagged with a segment identifier, which is used to increment the read address pointers from shared SRAM.

[0044] Memory 120 is intended for receiving and storing instruction sets for the accelerator's computing units. An example of loading instruction sets into memory 120 is shown in Fig. 3. Each segment of memory 120 is intended to store a set of instructions for one accelerator's computing unit. That is, the number of memory segments 120 is equal to the number of controlled units in the accelerator, or the number of clusters of computing units if they are combined into a cluster, for example, 4 segments, 8 segments, 4 clusters, 8 clusters, etc. Thus, for example, the neural network inference accelerator may contain the following elements: two matrix computing units, a data permutation unit, and a vector computer. Accordingly, to control each computing unit or group of computing units (matrix computing units), device 100 will have a corresponding number of segments in memory 120.As mentioned above, in one particular embodiment, a segment may correspond to several computing devices combined into a group (cluster). That is, the devices may be connected either individually or as a group (cluster). In both cases, each individual device and the group of devices are assigned a single block 130, respectively, with a single command interface. Each block 130 is assigned a single segment of segmented memory 120. Memory 120, in turn, is connected to external memory via a data bus.

[0045] Loading of instruction sets into memory 120 is performed after the descriptor is processed by descriptor processing unit 160. After receiving the requested io descriptor block 160, said block 160 allocates the address of the location and size of the instruction set of the computing device or group of computing devices, and service information, for example, the parameters of the device 100 applicable to a specific inference.

[0046] From the received addresses and instruction set sizes, instruction flow scheduler 170 generates requests to read the instruction set itself for each computing device or group of computing devices combined into a group and controlled by a single instruction set. The instruction set, in turn, contains the size and identifier of the instruction recipient if the instruction is intended for a cluster of computing cores.

[0047] Requests for reading instruction sets by scheduler 170 are generated based on the free capacity of memory 120 and software-defined maximum request sizes for each of the segments (one segment of memory 120 stores a set of instructions for one computing core or a group of computing cores). In the case of simultaneous requests from several segments, scheduler 170 arbitrates the requests. Since the entire set of instructions may not fit in memory 120, the specified set can be loaded into memory 120 through a plurality of separate requests, the size of which does not exceed the free capacity of memory 120 in the corresponding segment. Each read request is marked with a tag corresponding to the id of the segment of memory 120. The requested set of instructions is sent to the input demultiplexer and from it to memory 120, where the tag returned with the set of instructions is used to address one or another segment of memory 120. As can be seen from Fig.2, each of the segments is a circular buffer with a dynamic (software-defined) memory size (e.g., via the start_addr[x] / stop_addr[x] commands) and a block of read pointers, such as rd_ptr[x] and write pointers, such as wr_ptr[x]. Thus, the Start_addr[x] / stop_addr[x] commands can be commands for assigning the address of each segment, i.e., these commands essentially determine the size of each segment. The pointers are controlled by the pointer control block (PtrCtrl), which receives requests to read instructions from memory 120 and controls the addresses of instructions written to memory 120 and read from memory 120.

[0048] Each of the at least two blocks 130 may be a hardware block consisting of simple computing elements. The structural diagram of the block 130 is shown in Fig. 4. Thus, each of at least li two blocks 130 consists of the following elements: buffer memory 131, decoding block 132, synchronization block 133.

[0049] Memory 131 is a buffer memory, such as a FIFO with a data bus width equivalent to the data bus of memory 120. This feature increases the throughput of device 100. The depth of buffer memory 131 can be selected based on the number of command interfaces connected to device 100 and compensation for the delay in processing requests from other units 130.

[0050] Block 132 is designed to decode instructions. Decoding can be accomplished by receiving a packed instruction at the input, generating an unpacked instruction, and sending it to the computing devices as a data packet. Each packed instruction has a preamble—in this particular case, 32 bits—that contains the instruction length in bytes, the identifier (address) of the destination device, as well as the last instruction flag (LAST) and the synchronization command flag (WAIT). Generally, using the instruction length information, decoding block 132 generates an instruction packet without a preamble and sends it to the address specified in the identifier. The width of the output bus (which connects the computing device to block 130) can be arbitrary and is selected based on the tracing requirements and the bandwidth of the command interface.

[0051] Synchronization block 133 additionally pauses the sending of instructions via the command interface upon receiving a WAIT command. The condition for continuing instruction sending can be specified in the command body and may include either a conditional or unconditional delay in system clock cycles, or one or more events from module 180, which counts the number of executed instructions on the computing cores. When the continue condition is met, block 133 asserts the continue signal to the decoding module.

[0052] Each of at least two blocks 130 is connected to a computing device of the accelerator or a group of computing devices of the accelerator and is configured to: buffer a stream of instructions received from a segmented buffer memory; decode the received stream of instructions; synchronize the stream of instructions; send the decoded stream of instructions to at least one computing device of the neural network inference accelerator.

[0053] In one particular embodiment, to avoid blocking the read channel (when memory 120 is waiting for receiving block 130 to be ready), the size of the read request should not exceed the free memory capacity in block 130, the size of the minimum request and maximum request can also be configured depending on the load profile.

[0054] To send a stream of instructions to the computing devices, arbiter 200 receives requests from units 130, performs arbitration using the round-robin algorithm, and sends them to memory 120, indicating the number of requested instructions and the identifier of the specific unit 130 that issued the request. Instructions read from memory 120 pass through a demultiplexer and are sent to the corresponding unit 130, which performs additional instruction buffering, instruction decoding, instruction synchronization based on a programmed event, and sending the decoded instructions to the computing cores or clusters of computing cores. When sending instructions, unit 130 increments the counters of sent instructions and detects the last instruction, which is marked with the LAST flag.

[0055] It's worth noting that the presence of memory 131 in block 130 reduces latency when reading instruction sets from memory 120. Since each accelerator computing unit is controlled by its own instruction set, to eliminate latency when the computing units simultaneously access memory 120, the instruction sets are buffered in the corresponding blocks 130. Accordingly, implementing memory 120 as a single physical memory with dynamic segments, in turn, compensates for the latency when reading from external memory. This approach reduces the overall memory footprint while maintaining high instruction read speeds for the command interface.

[0056] Descriptor pointer block 140 is designed to receive descriptor pointers from an external control device. The descriptor pointers contain the address of the external memory location and the size of the instruction set descriptor for at least one computing device.

[0057] The configuration parameters of block 140 are set by means of configuration streams received from block 110. The parameters of block 140 are set by means of the configuration received from block 110. Thus, the parameters of block 140 can PC17RU2024 / 000312 represent pointers to a circular buffer of descriptor addresses, etc. In one particular embodiment, block 140 may be a circular buffer (FIFO) implemented in SRAM memory, with one read port and one write port. Each entry in block 140 contains the address and size of a descriptor. It is worth noting that the descriptor itself is stored in external memory, as its size can reach several kilobytes, significantly exceeding the memory size of device 100. This feature also reduces the required on-chip memory. After the external control device (host) writes descriptors to external memory, the host writes pointers (the address and size of the descriptors) to block 140.

[0058] The block for managing requests for reading descriptors 150 based on pointers from the block of descriptor pointers 140 is configured to obtain from the external memory descriptors of a set of instructions for at least one computing device based on the descriptor pointer.

[0059] Block 150 is designed to monitor the fullness of block 140 and send descriptor read requests from block 140. The read request is multiplexed with instruction set read requests in scheduler 170 and sent to the memory access controller. The requested descriptor is sent to the input demultiplexer and from there to descriptor processing block 160, which allocates the addresses and sizes of instruction sets of the computing cores and service information—device 100 settings applicable to a specific inference.

[0060] The instruction set descriptor processing unit 160 is designed to allocate the address of the location and size of the instruction set of the computing device, as well as the parameters of the control device for the executable output of the neural network.

[0061] The specified block 160 receives the descriptor, allocates addresses and sizes of program streams, and also sends the specified data to block 170.

[0062] The instruction flow scheduler 170 is configured to load a set of instructions, based on the processed descriptor, from external memory into at least two segments of the segmented buffer memory and to monitor the fullness of the segments of said memory.

[0063] Thus, based on the data received from block 160 (from the received addresses and instruction set sizes), scheduler 170 generates requests to read the instruction stream for each of the computing cores or group of computing cores. The instruction set is a packed set of instructions, each of which contains the instruction size and recipient identifier if the instruction is intended for a cluster of computing cores.

[0064] Thus, in one particular embodiment, scheduler 170 is configured to load a portion of a set of instructions into memory 120 incrementally, based on the available memory capacity of each segment. Thus, after receiving data from block 160, the scheduler, based on the known memory capacity of the segment and the instruction stream size, can perform a portioned load (i.e., load not the entire instruction set, but a portion of it) into memory 120. Accordingly, after sending such a portion of instructions to memory 120, the scheduler issues a command to load the next portion of instructions. In another particular embodiment, a request to load a portion of instructions can be sent when a memory segment is empty to a certain threshold, for example, to 80% of the segment's capacity, etc.

[0065] Module of counters of instructions executed by computing devices 180.

[0066] Said module 180 is designed to count the number of executed instructions of the computing cores. Module 180 is configured to receive a signal from each computing device or group of computing devices regarding the execution of sent instructions and to count the number of executed instructions.

[0067] Each execution unit, whether individual or clustered, is assigned its own identifier. When executing an instruction, the units emit a strobe signal (done) lasting one system clock cycle, which is counted in block 180. Thus, block 180 contains information on the number of instructions executed by each execution unit. After the entire inference is executed, the executed instruction counters in block 180 are reset.

[0068] The instruction set execution control unit 190 is designed to control the execution of current instructions.

[0069] The specified block is intended to determine the correctness of the execution of a set of instructions on each computing device.

[0070] Thus, the counter values ​​for sent and executed instructions (module 180) and the flags of the last instructions of each block 130 are sent to block 190. This block 190 monitors the execution of the current set of instructions and, after the instructions are executed, generates a request to read a new descriptor pointer. This implementation also enables additional descriptor buffering to compensate for the latency of reading from external memory.

[0071] The instruction request arbiter 200 is designed to receive requests from blocks 130, perform arbitration, for example, according to the round-robin algorithm, and send requests to memory 120 indicating the number of instructions requested and the identifier of block 130 that issued the request.

[0072] Thus, the submitted application materials described various options for implementing control of computing devices of the neural network inference accelerator.

[0073] The specific selection of elements of device 100 for implementing various software, hardware and / or architectural solutions may vary while maintaining the required functionality provided.

[0074] The submitted application materials disclose preferred embodiments of the technical solution and should not be construed as limiting other, specific embodiments within the scope of the requested legal protection that are obvious to specialists in the relevant technical field. Therefore, the scope of this technical solution is limited only by the scope of the attached claims.

Claims

FORMULA 1. A control device for computing devices of a neural network inference accelerator, containing the following elements placed on a crystal: • a parameter configuration unit associated with an external control device, configured to receive data containing configuration parameters for the control device; • a segmented buffer memory connected to an external memory, configured to receive and store instruction sets, wherein each segment has a dynamic memory capacity established by the received configuration parameters, and is configured to store a set of instructions for at least one computing device of the neural network inference accelerator; • at least two instruction flow processing units connected to at least two computing devices of the neural network inference accelerator, each of which is configured to: buffer the instruction flow received from the segmented buffer memory; decode the received instruction flow; synchronize the instruction flow; send the decoded instruction flow to at least one computing device of the neural network inference accelerator; • a block of descriptor pointers configured to receive descriptor pointers from an external control device, wherein the descriptor pointers contain the address of the location in the external memory and the size of the descriptor of the instruction set for at least one computing device; • a block for managing requests for reading descriptors using pointers from a block of descriptor pointers, configured to obtain from external memory descriptors of a set of instructions for at least one computing device based on a descriptor pointer; • a unit for processing instruction set descriptors, configured to allocate the address of the location and size of the instruction set of the computing device, as well as the parameters of the control device for the executable output of the neural network; • an instruction flow scheduler configured to load a set of instructions, based on a processed descriptor, from external memory into at least two segments of segmented buffer memory and to monitor the fullness of the segments of said memory.

2. The device according to paragraph 1, characterized in that the computing device of the neural network inference accelerator is a computing core of the neural network inference accelerator or a cluster of computing cores of the neural network inference accelerator.

3. The device according to claim 1, characterized in that the configuration parameters represent at least one of: a configuration of the segmented buffer memory; a configuration of the descriptor processing request control unit; a configuration of the descriptor processing unit; a configuration of each of the instruction flow processing units.

4. The device according to claim 1, characterized in that the instruction flow scheduler is designed with the ability to load a set of instructions in portions.

5. The device according to claim 4, characterized in that the portioned loading of the instruction set is a sequential loading of individual parts of the instruction set into a segment of the buffer memory, the size of which does not exceed the free volume of the memory segment in the segmented buffer memory.

6. The device according to paragraphs 4-5, characterized in that each part of the instruction set is marked with a tag corresponding to the number of the buffer memory segment.

7. The device according to claim 1, characterized in that each segment of the buffer memory is a circular buffer with a configurable memory capacity.

8. The device according to claim 1, characterized in that the block of descriptor pointers is a circular buffer (FIFO).

9. The device according to claim 1, characterized in that the descriptor pointer contains at least the address of the location of the descriptor of the instruction set for the computing device of the neural network inference accelerator in external memory and the size of the specified descriptor.

10. The device according to claim 1, characterized in that the set of instructions contains an identifier of the instruction recipient in the neural network inference accelerator.

11. The device according to claim 1, characterized in that the buffering of instructions in at least two instruction flow processing units is carried out in an internal circular buffer with a data bus width equivalent to the data bus of a segmented buffer memory.

12. The device according to claim 1, characterized in that the sending of the decoded instruction stream is carried out in the form of a data packet.

13. The device according to claim 1, characterized in that when sending a decoded instruction stream, the instruction stream processing unit is additionally configured to count the number of instructions sent and detect the last instruction.

14. The device according to claim 13, characterized in that the instruction flow processing unit additionally contains a synchronization unit configured to pause and resume sending instructions upon receipt of a corresponding pause or resume command.

Images