Data processing devices and methods

Abstract

A data processor is suggested comprising at least an instruction issue stage issuing instructions, a number of processing elements to which at least some of the instructions are issued and which receive operand data, generate result data in accordance with the instructions received and transmit their result data to other processing elements for use as new operands; and a bus system for these transmissions, wherein the instruction issue stage is adapted to issue instructions to a group of processing elements to operate them in at least two different modes, namely an out-of-order mode wherein instructions may be executed out of order and a loop acceleration mode wherein loops can be executed efficiently, and wherein the bus system comprises an arbiter operative in the out-of-order mode to arbitrate access of the group of processing elements to at least a part of the bus system and inoperative in the loop acceleration mode.

Description

[0001] Data Processing Devices and Methods

[0002] The present invention relates to the processing of data.

[0003] Processing data has become ubiquitous in modern life, but dif- ferent types of data necessitate different types of data pro- cessing and thus, different types of data processing devices have been suggested such as ASICs, CPUs, GPUs, FPGAs, DSPs asf., with each different data processing architectures having different advantages and disadvantages which makes them par- ticularly suitable for different tasks.

[0004] For example, a stream of data received in a digital TV set may require that virtually all pixels defining a frame aarree pro- cessed in the same manner and that the same processing steps will be effected for each frame to be displayed; also TV sets are produced in large numbers.

[0005] In a machine control, a stream of data is also received, but for each kind of a plurality of different sensors, the control unit might need to process the data differently, Processing for a given sensor might remain unaltered until sensor data change significantly.

[0006] In a laptop, server or the like, , data may be received as arbi- trary input from a keyboard or a < server and might need a con- stantly changing processing.

[0007] A number of different data processing architectures have been suggested in the past,

[0008] For example, the stream of data received in a ddiiggiittaall TTVV can be processed using an application-specific integrated circuit,

[0009] (ASTC) , which is cheap when produced in high volumes, operates with low power but has the disadvantage that the way data is processed cannot be altered, which may be significant e . g . when new codecs come up .

[0010] Where the possibility is needed to ee .. gg .. adapt the processing to new codecs, a field programmable gate array could be used having a number of logic gates interconnectable to each other in a configurable way or having interconnectable ccooaarrssee granu- lar data processing elements each comprising ALUs and a small number of registers, as is the case for most modern FPGAs . Such FPGAs can execute a number of operations in parallel, are well suited for streaming and are quite energy efficient, but for data processing, configurations need to be developed, de- fining the operation of each element and the interconnection thereof which is tedious and often considered difficult to learn .

[0011] Nonetheless, where data processing changes very infrequently and only between a few different algorithms, an FPGA can be

[0012] 1

[0013] SUBSTITUTE SHEET (RULE 26) used . Where the data processing changes more frequently, but still remains more or less constant over prolonged periods , an FPGA or architecture speci fically adapted to re-configure the data processing device in a fewer clock cycles might be need- ed .

[0014] The advantage of reconfigurable devices is that a number of processing elements can be provided allowing processing of da- ta in parallel where the algorithm allows . Still , the necessi- ty to define configurations will remain . As an alternative in modern PCs , graphical processing units , GPUs , are common also having a large number of processing element operated in paral- lel . However, typically, GPU processing is based on instruc- tions rather than on configurations so GPUs might be easier to program . Yet , instructions have to be issued over and over again even where the same instructions need to be executed re- peatedly over and over again, which may be inef ficient with respect to energy consumption .

[0015] In a modern PC, laptop, smart phone and the like , central pro- cessing units , CPUs , will be used for data processing , They are easy to program, but even though most CPUs now have sever- al more or less independent cores , they are not well suited for processing data in parallel , Each core will have an ALU, registers , a program pointer as f . for processing operands , For fast operation, a large data processing overhead is necessary as cores in a multicore processor may need to exchange data, as cores may have to wait for data coming from sources outside the processing device , as instructions need to be issued each clock cycle , predictions need to be made as to the most likely next instruction to be executed as f . Accordingly, tthhee overall energy consumption is high and chips need an very large number of transistors to implement the di f ferent elements leading to high costs .

[0016] OBJECT OF THE INVENTION

[0017] It will be understood that several design ideas have been com- bined in the past . For example , data processing chips have been developed for laptops , desktops as f . including both a multicore processor and a GPU . However, while in this manner, advantages may be obtained and combined, the overall chip can be expected to become larger and hence more expensive .

[0018] In virtually all instances , it will be preferred to process data fast with a cheap device and requiring but little energy and little ef fort in programming . Accordingly, it would be de- sirable to have an improved data processing architecture that is fast , can be easily produced, e . g . integrated on a chip or

[0019] 2

[0020] SUBSTITUTE SHEET (RULE 26) produced using small chiplets and can be operated with ( com- paratively) low energy consumption .

[0021] It is also preferable that the particular processing of data a user needs can be easily defined, by defining either the hard- ware layout ( as in AS ICS ) or the configurations ( as is the case with FPGAs ) or by programming as is the case for a GPU or CPU . It will be understood that programming in a high-level language code such as C++ is preferred over defining hardware or configurations in Verilog or the like , Furthermore , it would be highly preferred i f legacy code could be used .

[0022] Whatever way of defining the operation of a system for pro- cessing data is chosen, it would be helpful i f use can be made of techniques known per se in the art to speed up processing . Such prior art techniques will be described later oonn so that the reader can better understand why the design disclosed here by the present invention is an improvement over the art and how prior art techniques can be applied for the purposes of the present invention, improved, modi fied, implemented and / or adapted .

[0023] While it will be obvious that not all obj ects such as low costs , good programmability, high performance and adaptability to one or a variety of purposes can be simultaneously achieved in a given single design, it will be helpful i f at least some of the obj ects can be achieved at least in part . Note that "performance" can be considered to be high both in cases where a given data processing task is executed in a manner requiring little overall energy and / or in a case where the task is exe- cuted particularly fast . Sometimes , both meanings will not be fully separated, for example where a processor operation is power-limited due to thermal constrains of operation so that reducing power consumption for a given process means that it can be executed with a higher frequency and hence faster with- out damaging the device .

[0024] SOLUTION SUGGESTED BY THE INVENTION

[0025] Accordingly, the present invention aims to provide at least one of a data processing chip architecture , an improved data processing chip device , an improved method of operation of a data processing system and a data processing device that achieves at least one of the obj ects mentioned at least in part .

[0026] Note that the data processing chip architecture and / or the im- proved data processing chip device can in particular be physi- cally implemented as a monolithic chip, as a System-on-a-Chip,

[0027] 3

[0028] SUBSTITUTE SHEET (RULE 26) as an integrated circuit having a number of ( e . g . stacked) chiplets .

[0029] The independent claim ( s ) show how the obj ect can be achieved , Preferred embodiments and / or inventive aspects and additional inventions can be found in the description and the drawing . Note that additional ideas that may be inventive per se can also be found in the description, drawing and claims , even i f such additional are currently not claimed in an independent manner .

[0030] INCLUS ION BY REFERENCE & GENERAL CAVEAT AS TO TERMS

[0031] It shall be noted that hereinafter, terms such as "each" or "every" and the like are frequently used when certain pre- ferred properties of elements of the architecture and so forth are described . This is done so in view of the fact that gener- ally, it will be highly preferred to have certain advantageous properties for each and every element of a group of similar elements aass layout and design may be simpli fied . It will be obvious ttoo the average ski lled person however, that some i f not all ooff the advantages of the present invention disclosed hereinafter might be obtainable , even i f only to a lesser de- gree , i f only some but not all similar elements of a group do have a particular property . Also , it is noted that in a pro- cessing device , certain properties are not needed for each and every element of an array or each and every element performing certain tasks . For example , in an array of elements having ALUs , it is not necessary to provide dividers for each and every ALU as dividers are seldom used in typical algorithms . Accordingly, the skilled person will understand that only one or few of elements with ALU need to be provided with corre- sponding capabilities . Such di f ferent elements can be integral part of an array within the array itsel f or can be placed at the side and close to an array of simpler elements or even on top or below such aann array where chiplets and the like are used .

[0032] Thus , the use of certain words such as "each" , "any" „every" and so forth is intended to disclose a mode of invention pre- ferred from the point of simple design and more simple expla- nation, but it will be obvious that such limitations are not meant to unnecessarily restrict the scope of the disclosure to only the embodiments preferred, even though it is considered feasible to limit any claim to only such preferred embodi- ments . Accordingly, applicant reserves the right e . g . to refer to "an array comprising a group having a plurality of ele- ments , each element of the group having a certain property" rather than referring to "an array of elements all having a certain property" .

[0033] 4

[0034] SUBSTITUTE SHEET (RULE 26) It shall also be expressively noted that the following text discusses inter alia the use of (cheap) flash memory as main memory for the processor system. It will be understood that the use of flash memory is technically challenging to imple- ment and that therefore, allowing a cheap flash memory to be used is difficult. However, it is expressively stated herewith that the system of the present invention is also implementable in a system using DRAM memory and / or DRAM memory, in particu- lar in connection with hard-discs-drives and / or solid-state- drives and that the mere fact that an implementation particu- larly difficult to be achieved is disclosed is in no way to imply the invention must be restricted to such memory. In con- trast, the average skilled person will understand that in many instances described, the particular type of memory will not even be important.

[0035] Reference is had to — processors as described and disclosed in PCT / EP 2009 / 007415; PCT / EP 2011 / 003428 and PCT / EP 2012 / 000713 as well as in DE 11 007 370.7 (often called ZZYX processors hereinafter) . Reference is also had to the memory architec- tures as described in PCT / EP 2010 / 003459, in particular for the processors referred to as well as multi-core processors known in the state of the art, e.g. from Intel, AMD, MIPS and ARM) ; aanndd rreeffeerreennccee is had to exemplary methods for operating ZZYX processors aanndd the like as described in PCT / EP 2010 / 007950 .

[0036] Furthermore, reference is made in particular to the following related patent applications. WWhheerree applicable, priority is claimed to the following ppaatteenntt applications PCT / EP 2009 / 007415 corresponding to WWC022001100 / / 004433440011 (ZZYX07) ; PCT / EP

[0037] 2010 / 003459 corresponding to WWOO22001100 / / 114422443322 (ZZYX08) ; PCT / EP 2010 / 007950 (ZZYX09) ; PCT / EP 22001111 / / 000033442288 (ZZYX10) ; PCT / EP 2012 / 000713 (ZZYX11) ; DE 1111 000077 370.7 (ZZYX12) ; PCT / US2015 / 065418 (ZZYX16) ; PPCCTT / / UUSS22001155 / / 006655441188 (ZZYX16) ; US

[0038] 2014-0351563A1 (ZZYX13) . Also, the content of these documents is enclosed herewith for the purpose of reference and disclo- sure. Note that some of these documents have not been pub- lished themselves, but have been included in published files as priority documents, The applications listed above are fully incorporated by reference for detailed disclosure.

[0039] Reference is also had to the following documents, describing details of an architecture and data processing methods which are particularly suitable to be adapted for an implementation of the present invention. Accordingly, the disclosure in these documents is enclosed in its entirety herein: wo 2016 / 100142A2"Advanced Processor Architecture" ; wo 2013 / 098643A2 "Advanced Processor Architecture" ; wo 2012 / 167933A9 "Tool-Level and Hardware -Level Code Optimiza- tion and respective Hardware Modification"; WO 2012 / 123061A9"Paralell Memory Systems" ; WO

[0040] 5

[0041] SUBSTITUTE SHEET (RULE 26) 2011 / 079942Al"Optimisation ooff Loops and Data flow sections"; US 2024 / 0054097Al"High Performance Processor"; US 2023 / 0409334A1" Providing Code Sections for Matrix Arithmetic Logic Units in a Processor"; US 202 / 10406027A1 "Advanced Pro- cessor Architecture"; US 2021 / 0286755Al"High Performance Pro- cessor"; US 2021 / 0026637A1 "Providing Code Sections for Matrix Arithmetic Logic Units in a Processor"; US 2020 / 0241879A1 "Is- suing Instructions To Multiple Executions Units"; US 2020 / 0042492Al"Cptimization of Loops and Data Flow Sections in Multi-Core Processor Environment""; US 2019 / 0377580Al"Execution of Instructions based on Processor and Data Availability"; US 2019 / 0197015Al"Parallel Memory Sys- tems "; US 2019 / 0171449Al"Tool-Level and Hardware-Level Code Optimization and respective Hardware Modification "; US 2019 / 0079769Al"Providing Code Sections for Matrix Arithmetic Logic Units in a Processor "; US 2018 / 0181403A1 "Issuing In- structions To Multiple Executions Units "; US 2018 / 0039576Al"System and Method for a Cache in a Multi-Core Processor "; US 2018 / 0004530A1" Advanced Processor Architec- ture "; US 2017 / 0364338A1" Tool-Level and Hardware-Level Code Optimization and respective Hardware Modifications "; US 2017 / 0262406A1 "Optimization of Loops and Data Flow Sections in Multi-Core Processor Environment" "; US 2016 / 0306631Al"Providing Code Sections for Matrix Arithmetic Logic Units in a Processor "; US 2016 / 0048394Al"Issuing In- structions To Multiple Executions Units "; US 2016 / 0004639Al"System and Method for a Cache in a Multi-Core Processor "; US 2015 / 0301983Al"Optimization of Loops and Data Flow Sections in Multi-Core Processor Environment"; US 2014 / 0351563A1" Advanced Processor Architecture "; US 2014 / 0310696Al"Tool-Level and Hardware-Level Code Optimization and respective Hardware Modification "; US 2013 / 0191817A1" Op- timization of loops and data flow sections"; US 2012 / 0216012Al"Sequential Processor comprising an ALU Array "; US 2012 / 0137075A1" System and Method for a Cache in a Multi- Core Processor"; US 11797474 B2" High performance processor"; US 11687346B2" Providing code sections for matrix of arithme- tic logic units in a processor"; US 11061682B2" Advanced pro- cessor architecture"; US 10908914B2" Issuing instructions to multiple execution units"; US 10409608B2" Issuing instructions to multiple execution units"; US 10331615B2" Optimization of loops and data flow sections in multi-core processor environ- ment"; US 00010031888B2" Parallel memory systems"; US 9898297B2" Issuing instructions to multiple execution units"; US 9734064B2" System and method for a cache in a multi-core processor"; US 9703538B2" Tool-level and hardware-level code optimization and respective hardware modification"; US 9672188B2" Optimization of loops and data flow sections in multi-core processor environment"; US 9348587B2" Providing code sections for matrix of arithmetic logic units in a pro- cessor"; US 9152427B2" Instruction issue to array of arithme- tic cells coupled to load / store cells with associated regis-

[0042] 6

[0043] SUBSTITUTE SHEET (RULE 26) tens as extended register file" ; US 99008866997733BB22"" System and method for a cache in a multi-core processor" ; UUSS 9043769B2" Optimi zation of loops and data flow sections iinn multi-core processor environment" ; GB 2549883A" Advanced processor archi- Lecture" ; EP2996035A1" Data Processing Device " ; EP2791789A2" Advanced Processor Architecture " ; EP2718859A1" Level Code Op- timi zation and respective Hardware Modi fication" ; EP2519876Al"Optimi zation of Loops and DDaattaa Flow Sections" EP2366144Bl"Sequential Processor comprising an ALU Array " ; EP2366144A2"Sequential Processor comprising an ALU Array" . Note that some of these documents have not been published themselves , but have been included in published files as pri- ority documents . Also , some of the documents cited may belong to the same patent families . The documents listed above are fully incorporated by reference for detailed disclosure . In certain instances and j urisdictions and where applicable , pri- ority from some or all of the documents mentioned above may be claimed .

[0044] In the present disclosure , both the term "execution unit" and the term "processing element" are used .

[0045] Both terms have also been used extensively in the prior art , such as when referring to arrays of process cores ; actually, both terms have been used in the prior art to refer to com- plete cores including inter alia instruction fetch / decode / issue units and other circuitry necessary common to implement a complete core , such as a dedicated Level 1 cache .

[0046] In contrast , an execution unit or processing element of an ar- ray or processing element ; as provided in the maj or aspects of the present disclosure , will not have their own instruction f etch / decode / issue units , Level 1 caches as f . each, but will share e . g . instruction fetch / decode / issue unit with the entire array or will at least share a plurality of instruction fetch / decode / issue units as provided for multithreading within an array . Accordingly, there may be a signi ficant di f ference between the execution units of processing elements mentioned in some of the prior art and any execution units or processing elements referred to in the disclosure of the invention .

[0047] Furthermore , it is noted that in previous applications of in- ventor Vorbach in particular in behal f of Hyperion company and / or those mentioned above , reference is frequently had to an array of ALUs , that is arithmetic-logic-units . Nonetheless , it could be easily understood that the ALUs referred to in those documents are not j ust have e . g . editor circuitries , multipliers circuitries , comparator circuitries but will also have some additional circuitries such as input registers , out- put registers , bus interface circuitries and so forth . In or- der to emphasi ze this more strongly, the present disclosure

[0048] 7

[0049] SUBSTITUTE SHEET (RULE 26) uses the term "processing element" or "execution unit" rather than the term ALU and does so in a manner where the two terms are used exchangeably .

[0050] Applicability to known designs

[0051] It will be understood that applicability of certain techniques may obviously depend on the exact architecture resulting from the ideas laid out here and that even where a data processing design is conventional , certain novel techniques and methods described and disclosed herein may be used for certain conven- tional architectures while certain of the novel architectural designs can be implemented without making partial or full use of each and every method .

[0052] It should be understood that the technology and disclosure de- scribed in herein will be particularly applicable on ZYXX pro- cessors as described in the extensive list of documents cited above and included herein in their entirety by reference as well aass inter alia in PCT / EP 2009 / 007415 ; PCT / EP 2011 / 003428 ; PCT / EP 2012 / 000713 and DE 11 007 370 . 7 , with memory architec- tures as described in PCT / EP 2010 / 003459 . Also , ideas present- ed and disclosed here will also be applicable directly or with little amendments on multi-core processors known in the state of the art ( e . g . from Intel , AMD, MIPS and ARM) ; and exemplary methods for operating ZYXX processors and the like as de- scribed in PCT / EP 2010 / 007950 .

[0053] The above mentioned documents are fully incorporated by refer- ence for detailed disclosure of the speci fic designs architec- tures , pprroocceedduurreess aanndd mmeetthhooddss for which the present applica- tion iiss ppaarrttiiccuullaarrllyy uusseeffuull .. In certain iinnssttaanncceess and j uris- dictions aanndd wwhheerree aapppplliiccaabbllee ,, priority from ssoommee or all of the documents mentioned above may be claimed .

[0054] Accordingly, while the present invention and description re- late inter alia to a novel architecture for processing of da- ta, to one or more novel aspects of novel and known architec- tures for processing of data, and / or to as well as to novel methods of processing data using data processing devices and of preparing for the processing data, it will be understood that a large number of functionalities , methods and / or ele- ments known per se in the art might be used and / or modi fied, and that any such modi fication might be inventive even i f ap- plied to a conventional data processing device such as an FPGA, DSP, CPU, GPU and the like .

[0055] 8

[0056] SUBSTITUTE SHEET (RULE 26) PRIOR ART OF SPEEDING UP PROCESSORS

[0057] Processing data faster has been a longstanding problem and ac- cordingly, techniques are known in the art to speed up pro- cessing . Such known techniques might be applied, improved, modi fied, implemented and / or adapted for the purposes of the present invention . Therefore , a brief overview will now be given about several techniques important to the invention for either being applicable as is or for being adaptable or im- provable for the purposes of the invention .

[0058] This being said :

[0059] What a CPU basically needs to do for processing data is get instructions , the corresponding data, process the data and write back the data to the correct location . Operand data can be stored in and retrieved from a limited number of registers , with the number depending on the exact architectural implemen- tation of the processor, but data of course needs to be trans- ferred to the registers and results must eventually be output- ted to a memory, interface or the like . Note that the number of registers will be inter alia dependent from the instruction set executable by the architecture as any instruction set ar- chitecture will make use of a speci fic, but limited number of registers , ( as will be seen hereinafter, the absolute umber of registers provided for a given core can be higher than the number of registers in the instruction set so as to allow exe- cution of several threads ; also , where multiple cores are pro- vided, each core may have its own register set ) .

[0060] In more detail , to process e . g . a mathematical operation, the corresponding instruction needs to be fetched from the loca- tion in the (program) memory te which the program pointer points . The instruction fetched must be decoded to the level where the processor can actually process the data ( for exam- pie , correct registers must be defined where operands are re- trieved from or results are to be stored temporarily) . Then, the ( decoded) instruction has to be issued . Note that typical- ly, reference is made to " fetching an instruction" from memory and thereafter " issuing the instruction" even though some mod- i fication to the instruction as fetched such as with respect to register allocation has been ef fected prior to issuing, so that the instruction issued will not be the exact same bit pattern fetched . In most cases , the present disclosure will not deviate from that denomination practice .

[0061] Then, the necessary IM; s for executing the instruction must be loaded and the actual mathematical or logical operation must be carried out ( executed) and the result must be written to a predefined location so that it can be used later .

[0062] 9

[0063] SUBSTITUTE SHEET (RULE 26) For executing a mathematical operation, a variety of di f ferent mathematical-logical components such as an adder, a multipli- er, a comparator, a divider as f . will be provided in an execu- tion unit and one of those components will be selected for the actual operation . FFuurrtthheerrmmoorree ,, di f ferent stages may also be provided for the ootthheerr steps such as an instruction fetch stage , an instruction decoding stage , a load stage , an execu- tion stage and a write stage can be provided .

[0064] While a processor can be operated at higher frequencies , addi- tional measures to increase performance can be taken . Now, for the ef ficient operation of the processor, it is of course best i f all of these units are kept busy all of the time .

[0065] Also , the parts other than those actually processing data but needed for correct and / or fast operation such as program pointers , registers , caches and so forth should be operated in a manner that allows to minimi ze the area needed to implement them and / or the energy needed to operate them- in this way, for a given si ze of a chip, mmoorree aarreeaa rreemmaaiinnss available for actually processing operands aanndd lleessss hheeaattiinngg of the device occurs due to such processes . IInn tthhiiss ccoonntteexxtt ,, it will be un- derstood that modern pprroocceessssoorr ddeessiiggnnss uussee a plurality of cores , the cores having most of the units described above for only their own use while sharing some of the resources such as higher levels of a cache with other cores . Also , for the cores to communicate with other cores , communication paths must be provided, but the area needed for this again should be mini- mi zed and the communication should be suited for the algo- rithms the device will typically be used for . Note that the communication paths may depend on the available hardware im- piementation, e . g . because where chiplets of core arrays are stacked into a plurality of stacks , communication within a single layer might be extremely fast , communication within a stack might still bbee fast and have a high bandwidth whereas the communication bbeettwweeeenn di f ferent stacks might be compara- tively slow .

[0066] Accordingly, the use both of all ccoorreess and of their respective parts ( or elements or units ) sshhoouulldd bbee ooppttiimmii zzeedd wwiitthh a view to ef ficient processing . Obviously, pprroocceessssiinngg wwiillll be more ef ficient i f units do not have to wwaaiitt ffoorr ooppeerraannddss and i f units are kept as busy as possible . NNoonneetthheelleessss ,, keeping all units busy at all times is possible oonnllyy bbyy cceerrttaaiinn extra measures requiring additional hardware aanndd (( ssuuppppoorrttiinngg)) pro- cessing, e . g . suitable preparation of instructions . According- iy, there is a trade-of f and care should be taken to optimi ze use of the di f ferent techniques in a manner minimi zing the ad- ditional costs .

[0067] Regarding the di f ferent techniques developed to increase over- all performance :

[0068] 10

[0069] SUBSTITUTE SHEET (RULE 26) Latency, Caches , Preloading

[0070] One reason for a processor to not process data is that the processor has to wait for operands or instructions . This is called latency . With respect to latency, each data processing architecture will have suitable I / O circuits and inevitably, latencies will occur during their use . This is well known and given that a lot of the periphery such as hard disks , solid state devices , ( DRAM) memory as f . will be conventional for the present invention, so will be the I / O circuits used or usable in the present invention may be fully or at least in part con- ventional ; furthermore , as the periphery known in the art is known to create problems such as latencies , use will prefera- bly be made of techniques such aass caching and preloading and while the strategies and hardware for implementing caches , cache hierarchies or of preloading will di f fer from that in the art , it will be considered that the average skilled person has a basic understanding of these techniques . Note that both preferred caching strategies or cache architectures as well as preferred preloading strategies may di f fer .

[0071] Caches make data stored in or to be written to a memory having large latencies available closer to the processor and thus ac- cessible with smaller latencies ; preloading allows to load op- erands from memory into local registers or caches before the actual instruction needing them is scheduled . In both instanc- es- caches and preloading, it must be prevented that altering data by preceding instructions will not be noticed when re- trieving data , Corresponding techniques for this will be de- scribed as needed and will be compared to those known in the art . So , techniques such as caching and preloading can be ap- plied but must be implemented with care .

[0072] However, for an algorithm having a multitude of steps , not all operands will have to be retrieved as data from outside of the processor . Rather, intermediate results are frequently used that can be temporarily stored in a local register or cache and retrieved therefrom . Still , some time may still be needed for such internal retrieval , even i f intermediate results are produced in one unit and are trans ferred via a - preferably- short and / or direct path to a unit executing subsequent steps in an algorithm . RReeaassoonnss iinntteerrnnaall accesses cause latencies in- clude length of ddaattaa ccoonnnneeccttiioonnss ,, availability of busses and so forth .

[0073] Furthermore , techniques have been suggested to cover up wait- ing times and to improve performance of the single units , Some of the techniques known in the art interact to further improve the performance .

[0074] Instruction prefetching

[0075] As latencies may become important even for fetching instruc- tions , suggestions have been made to improve ef ficiency and

[0076] 11

[0077] SUBSTITUTE SHEET (RULE 26) speed thereof , In particular, it has been suggested in the art that rather than fetching single instructions during each fetch operation- chunks of subsequent instructions are fetched .

[0078] Accordingly, as long as the next instruction to be executed is part of the chunk ( or " lies within the window retrieved" ) , no signi ficant additional latency will occur within a chunk once retrieved, although j umps in the sequence of instructions might incur additional latencies as a j ump might necessitate to discard the remainder of the chunk , So , waiting times dur- ing fetching may still occur, in particular because programs branch at certain points in aa mmaannnneerr d deeppeennddiinngg oonn results ob- tained during processing . SSttiillll ,, rreettrriieevviinngg aa nnuummbbeerr o < f in- structions from memory at aa ggiivveenn ttiimmee wwiillll bbee eenneerrggyy ef fi- cient and improvements when decoding and / or issuing tthhee in- structions might be achieved, e . g . by detecting certain struc- tures in a sequence to be executed, even though it will be un- derstood that ever increasing the window si ze will not be helpful .

[0079] Occasions where a larger window might increase performance per se is when other techniques come into play, for example , by making improved use of instruction level parallelism as will be explained hereinafter .

[0080] Pipelining

[0081] Generally, preventing parts of a processor to idle increases the performance and in this context , it is helpful to remember that in a modern CPU, both a plurality of di f ferent execution units and a plurality of other stages ( or elements serving other purposes other than actual operating on operands ) are present so that several methods of preventing such units from being in an idle state may be applied .

[0082] For example , a processor includes parts other than the execu- tion units to e . g . process a mathematical operation in a pro- cess , the corresponding instruction needs to be fetched from a program memory, the instruction fetched must be decoded to the level where the processor can actually be made to process the data, then the ( decoded) instruction has to be issued, then the necessary operands for executing the instruction must be loaded and the actual mathematical operation must be carried out ( executed) and the result must be written to a predefined location so that it can be used later . Where in this kind of operation a new instruction is issued for every new data, a simple way of processing results called scalar processing .

[0083] While di f ferent units such as for example an instruction fetch stage , a decoding stage , a load stage , an execution stage and

[0084] 12

[0085] SUBSTITUTE SHEET (RULE 26) a write stage can be provided, for an efficient operation of the processor, it is of course best if all of these units are kept busy. To this end, registers can be provided between the different stages resulting in a pipeline of different stages. This allows to fetch a new instruction while the previously fetched instruction is decoded; to load data while a previous- ly instructed operation is executed and so forth. This is called pipelining; while it helps to increase performance be- cause instructions can be issued every clock cycle, pipelining alone will not help to keep a plurality of execution units busy at a given time, as proper instructions would have to be issued different separate execution units such as adders, mul- tipliers, comparators and dividers would have to be issued which is not the case.

[0086] So, even with pipelining, the remaining execution units simply remain idle. Even if pipelining is applied to scalar proces- sor, this obviously is inefficient and latencies may further reduce perfomance.

[0087] Scheduling: Multitasking

[0088] Techniques have also been developed that reduce latencies by making use of the fact that on the one hand, the instructions for a modern CPU form sequences which in turn together form a program, and that on the other hand a user (or, more general, a computer such as a server) might need to execute several in- dependent programs in parallel. Certain programs or even parts of a program may however operate fully or at least somewhat independent of other parts or programs, as a given part might not need operands that are produced by execution of other parts .

[0089] Such separable parts allow to start execution of one part (or "component" of a program) before another part of the program has finished. This is highly desirable because it allows to keep the processor working even though a program part started earlier has to wait, e.g. until operands have been loaded from memory .

[0090] One strategy for this would be to isolate the different pro- cesses executed from each other in the most complete way fea- sible . This is known as multitasking and allows the concurrent execution ooff mmuullttiippllee ttaasskkss ((oorr " "pprroocceesssseess"")) oovveerr aa certain period of time by regularly changing between them. The separa- tion allows to provide different page tables, different task state segments, different protection rings, different I / O per- missions for different tasks.

[0091] For this, instead of waiting for a task previously started to end, new tasks can interrupt a previous task instead of wait-

[0092] 13

[0093] SUBSTITUTE SHEET (RULE 26) ing for them to end . As aa rreessuulltt ,, a computer can execute seg- ments of multiple tasks iinn aann iinntteerrlleeaavveedd manner, while the tasks share common processing resources such as central pro- cessing units ( CPUs ) and main memory . Accordingly, multitask- ing interrupts a running program, then has to save the state thereof , (partial results , memory contents and computer regis- ter contents ) load the saved state of another, previously in- terrupted program and trans fers control to the new program . Several strategies to implement switching between tasks have been developed, in particular pre-emptive multitasking and co- operative multitasking .

[0094] In a preemptive scheduling scheme , a CPU is given a task and later on, the task will be taken away from the CPU, for exam- pie when a higher priority task requests tthhee CCPPUU .. Switching tasks is done by initiating a context switch routine , so that once the task is given back to the CPU, the task can resume its operations at the point and in the state it left .

[0095] Context switching can be ef fected in several ways .

[0096] For example , a round-robin-strategy can be implemented where tasks are switched after fixed time . However, this is not well suited for real-time systems since a time critical task cannot get hold of the CPU when it needs to . Also , a long task might be stopped shortly before it completes its operations which is disadvantageous .

[0097] In real-time systems , a scheme may be used where the tasks are assigned di f ferent priorities , with time critical tasks having higher priorities . Then, a higher priority task can be allowed to request that a lower priority is stopped and grab and use the CPU until it releases it .

[0098] Also , it is possible to implement a cooperative multitasking strategy where a running program is coded to signal to a su- pervisory software (which can be the operating system or part thereof ) that it can be interrupted .

[0099] Note that di f ferent strategies can be combined .

[0100] Frequently changing between di f ferent tasks increases the overhead due to the frequent switching, but will give a user the impression that multiple tasks would be performed at the same time , even though actually, only one task i s performed any given moment .

[0101] However, to keep a central processing core processing data, it is not always necessary to fully isolate di f ferent processes from each other in the way multitasking suggests . This allows additional techniques of improving performance .

[0102] 14

[0103] SUBSTITUTE SHEET (RULE 26) Improving Performance in view of Instruction-Level-Parallelism

[0104] Keeping components of a processor as busy as possible can be achieved in ways other than fully isolating di f ferent sequenc- es of execution from each other in the way multitasking sug- gests .

[0105] Now, as stated above , in a core , a variety of di f ferent mathe- matical- logical components are provided to actually execute instructions , such as adders , multipliers , comparators and di- viders . - i f these are to be used in parallel , instructions must be found that do not generate operands used by other in- structions simultaneously issued .

[0106] I f this is ensured, instructions may be executed in parallel . Of course , the processor architecture must support issuing several instructions per clock cycle . The form of parallelism obtained is called instruction-level-parallelism and a proces- sor capable of this is called superscalar processor . Obvious- ly, the throughput can be higher compared to a case where only one instruction at a time is executed .

[0107] It must be understood however that this way of operation re- quires to select instructions such that no dependencies be- tween them exist which would prevent them from being executed in parallel . SSoo ,, the dependencies need to be considered prior to issuing instructions and only i f independently issuable in- structions are found can they be issued .

[0108] Therefore , a situation may occur that in a sequence of in- structions currently retrieved ( the "chunks" or windows re- ferred to earlier ) , t all instructions depend on proceeding in- structions yet ttoo bbee executed . In this situation, the paral- lelism of the ccooddee iiss too low and no simultaneous execution can be scheduled . This problem becomes smaller however in case the window of instructions is enlarged .

[0109] This is a case where larger instruction fetching windows will improve performance . Of course , the processor must have enough resources such as registers for the additional operands , re- sults as f . to make use of the parallelism i f present .

[0110] Improving Performance by considering several Threads

[0111] Instead of considering completely independent ttaasskkss that re- quire a complete separation of all resources aanndd hence task switching, a software process may be considered which in turn may have several parts that can be managed independently while still sharing resources . In the art , reference is had to the smallest sequence of programmed instructions that can be man- aged independently as a thread of execution .

[0112] 15

[0113] SUBSTITUTE SHEET (RULE 26) Such threads may share the code , data and file the process be- longs to . Since they can be managed more or less independent- ly, but do not require separation of all resources , it is pos- sible to switch between the threads by simply issuing instruc- tions that belong to di f ferent threads . This not only spares the task switching, it is also useful where di f ferent threads need to exchange data- allowing a tight coupling thereof , alt- hough it becomes nneecceessssaarryy to ensure that no problems (haz- ards ) arise from tthhee sharing of rreessoouurrcceess ,, in particular memory hazards .

[0114] Performance can be increased when considering more than one thread . There are several possibilities to do so .

[0115] Improving Performance by considering several Threads A Interleaved Multithreading

[0116] A first possibility is to issue iinnssttrruuccttiioonnss from di f ferent threads in an interleaved manner so as to hide latencies . For this , the so- called barrel processor has been developed that switches between threads of execution on every cycle . This is also known aass " interleaved" oorr " fine-grained" temporal multi- threading .

[0117] An initial motivation for the development of barrel processors was to reduce overall hardware costs . In the case of an early example , the CDC 6x00 PPUs , the digital logic of the processor was much faster than the core memory, so a number of separate core memory units could share a single set of processor logic in a round-robin manner without loss of ef ficiency for the single thread . In the processor, for executing the di f ferent threads , each thread of execution was assigned its own program counter and other hardware registers so that each thread hat its own independent architectural state . The design guarantees that each thread will execute one instruction every n cycles , Unlike the preemptive multitasking machine described above , where a signi ficant overhead is caused by task switching and thus a thread typically runs for a very large number of cycles while all other threads wait their turn, threads in a barrel processor will be changed frequently .

[0118] There have been several problems , most importantly that the state of each thread must be kept on-chip, typically in regis- ters , to avoid costly of f-chip context switches . This requires a large number of registers compared to typical processors . On the other hand, designing an n-way barrel processor with an n- deep pipeline is much simpler than designing a single-tasking processor because a barrel processor never has a pipeline stall and doesn ' t need feed- forward circuits . Also , a barrel processor can guarantee that aa " real-time" thread can execute with precise timing, no matter what happens to the other

[0119] 16

[0120] SUBSTITUTE SHEET (RULE 26) threads , even i f some other thread locks up in an infinite loop or is continuously interrupted by hardware interrupts .

[0121] It is noted that in the coarse-grained and the f ine-grained interleaved multithreading, only one instruction will be is- sued; accordingly, only one instruction pipeline per core is needed for interleaved multithreading . HHoowweevveerr,, performance can be increased i f more than one instruction iiss issued per cycle . Accordingly, the instructions issued are not truly exe- cubed simultaneously .

[0122] Now, while the above mentioned barrel processor allows for a fine grained interleaved multithreading, it is not absolutely necessary to alter between di f ferent threads every instruc- tion . Rather, switching between threads could be ef fected only in case the current executing thread causes some long latency events e . g . due to page faults etc . TThhiiss results in coarse- grain multithreading where less context switching between threads is necessary . This has been employed iinn commercial products such as Intel ' s Montecito processor .

[0123] Improving Performance by considering several Threads B Simultaneous Multithreading

[0124] However, despite the fact that additional circuitry such as the aforementioned larger registers are needed, temporal mul- tithreading does not allow execution of multiple instructions in a cycle . Therefore , the performance can be increased fur- ther by allowing simultaneous execution of multiple instruc- tions .

[0125] Per definition, this also means that the processor is su- perscalar, that is , the processor has to have a plurality of instruction pipelines . Then, advantage can be taken of simul- taneous multithreading where instructions to be executed in parallel are selected from di f ferent threads , This allows to exploit additional forms of parallelism, with thread level parallelism ( TLP ) not facing the same restriction described above for instruction level parallelism .

[0126] As in thread level parallelism instructions from di f ferent threads are executed at the exact same time , a true simultane-

[0127] OUS multithreading results . Unlike conventional superscalar processors , where performance suffers from a lack of per- thread instruction-level parallelism, simultaneous multi- threading uses multiple threads to compensate for low single- thread instruction-level parallelism . The additional threads in simultaneous multithreading allow di f ferent instructions in the same clock cycle by using those process units that the first thread left .

[0128] 17

[0129] SUBSTITUTE SHEET (RULE 26) Simultaneous multithreading can use multiple threads to issue instructions each cycle , So , whereas in certain multithreaded hardware architectures , only a single hardware context or thread is active on any cycle , simultaneous multithreading supports all thread contexts to simultaneously compete and share processor resources . Again, several changes to the basic processor architecture are required to implement simultaneous multithreading, such as the ability to fetch instructions from multiple threads in a cycle and a suf ficiently large register file to hold data from multiple threads .

[0130] While simultaneous multithreading can be implemented without overly complicated maj or changes to a processor architecture , given that a larger register file to hold data from multiple threads and to fetch instructions for di f ferent threads must be considered when designing the chip and accordingly, the number of concurrent threads needs to be set . Two concurrent threads per CPU core are common, but some processors already support up to eight concurrent threads per core .

[0131] Improving Performance by considering several Threads C Hyperthreading

[0132] An even more advanced architectural way to make use of several threads is Hyper-Threading . A processor with Hyper-Threading Technology is arranged such that two logical processors per physical core can be defined and used, with each of the logi- cal cores having its own processor architectural state ( and thus , the corresponding hardware to define its own processor architectural state ) , but still sharing some physical elements of the physical core such as the ( typically complex and thus large ) execution engine , caches , and system bus interface . Such sharing of resources allows the two logical processors to work with each other more ef ficiently, and in particular al- lows a logical processor to "borrow" resources from another logical core waiting for data requested .

[0133] Improving Performance by considering several Threads D Chip-level multiprocessing

[0134] It is possible to provide more than one processor ( core ) on a chip and to then execute di f ferent threads on the chip inde- pendently . In this manner, Chip-level multiprocessing is achieved .

[0135] 18

[0136] SUBSTITUTE SHEET (RULE 26) Improving Performance by considering several Threads

[0137] E Combinations

[0138] Note that it is possible to combine the di f ferent techniques mentioned, e . g . Chip-level multiprocessing and Simultaneous multithreading .

[0139] Other techniques for improving performance

[0140] Inclusion of Field Programmable Gate Array ( FPGA)

[0141] In some algorithms , the same sequence of mathematical-logical operations needs to be executed repeatedly for a very large number of operands . For certain cases , e . g . where a stream of input operand data is to be manipulated at bit-level , the most ef ficient way for this would be to use a Field Programmable Gate Array ( FPGA) configurable at runtime . For this , a typically rather small- Field Programmable Gate Array ( FPGA) can be integrated into a processor .

[0142] Techniques for tightly coupling reconfigurable arrays into a processor are known . One suggestion to tightly coupling has been to use configurable special instructions accessible through the RISC-V Instruction Set Architecture ' s custom in- struction extension . Such custom instructions can be used to configure the runtime configurable FPGA, in particular by in- voking a routine that loads a speci fied configuration into the FPGA. In a multithreading environment , i f a thread other than the one which configured the FPGA attempts to access FPGA, an illegal-configuration exception might be triggered . Alterna- tively or in addition the FPGA might be used in some embodi- ments to implement routines that are available for all threads , such as interface special purpose routines to periph- eral devices . Configured for such a purpose , any thread might use the FPGA interface function without triggering an excep- tion .

[0143] Note that per se , the FPGA can be coarse-granular with pro- cessing elements having ALUs or can be fine granular having logic gates only or could be a mix of both . However, where the remainder of the processor core into which an FPGA is to be integrated already allows to ef ficiently execute the same se- quence of complex mathematical-logical operations repeatedly for a very large number of operands as in the case of loops , it can be preferred to use fine-grained FPGAs because of the smaller space requirement compared to more complex components of the core that would otherwise have to be used .

[0144] 19

[0145] SUBSTITUTE SHEET (RULE 26) Branch prediction to improve performance :

[0146] In most modern processors , instruction pipelines are used now- adays . As explained above , an instruction pipelines is used to prepare instructions for upcoming instruction issues . However, a situation might arise where the instructions in the pipeline cannot be used because a result j ust produced might cause a branching to another part of tthhee program currently executed . In this case , no instruction ccaann be issued until the correct instructions that the branching necessitates have passed thru the pipeline . Accordingly, ssuucchh a situation reduces the per- formance of the processor .

[0147] Therefore , it has been suggested to predict the most likely outcome of an upcoming branching and to prepare for the execu- tion of those instructions that are considered to be most the likely . However, while such predictions are possible , the most likely outcome of a branching is di f ficult and quite complex circuitry are necessary for a high prediction hit rate . Thus , there is a sweetspot regarding the improvement due to higher prediction hit rates versus the area needed to implement the prediction hardware .

[0148] Loop related improvements

[0149] Other known techniques for improving performance relate to the execution of loops . In a loop, instructions need to be repeat- edly executed, e . g . in a loop . Conventionally, this necessi- tates to retrieve , decode and issue the same instructions over and over again; obviously, this consumes more energy than nec- essary . Accordingly, it has been suggested to cache decoded instructions ( so-called micro-op code ) , saving both on decod- ing time and energy, but again adding to the complexity of the hardware design .

[0150] Also , many operations in a loop will not depend on each other , This allows to eexxppllooiitt parallelism bbyy ssoo ccaalllleedd loop- unrolling, where tthhee ppaarrtts that can be eexxeeccuutteedd in parallel during execution of the loop body are identi fied .

[0151] Out-Of-Order execution and Tomasulo algorithm

[0152] As explained before , programs consist of a sequence of in- structions . It has been suggested above with respect to in- struction level parallelism that some instructions could be executed in parallel , so that an instruction and its subse- quent instruction would be executed at the same time . 11 is also possible to go one step further and to execute a subse- quent instruction before all preceding instructions have is- sued .

[0153] 20

[0154] SUBSTITUTE SHEET (RULE 26) In more detail , in processing a sequence of stored instruc- tions , a situation may occur where operands for a first in- struction are needed but are not yet available due to laten- cies while subsequent instructions forth might be found later in the sequence that do not depend on results generated by the first instruction waiting for its operands . In such a case , it would be possible per se to execute the instructions not ac- cording to the sequence in which they are stored, but inde- pendent thereof and thus not in the same order they are stored . I f this is done , the instructions are executed out-of- order .

[0155] There is a caveat to Out-Of-Order execution Even though a sub- sequent instruction may not need input generated by a proceed- ing instruction, data hazards may still exist . In more detail , several hazards should be considered when instructions execut- ed out-of-order access the same registers or memory locations . Such access will inevitably occur, for example because the number of registers usable by instructions is limited . The da- ta hazards to be considered are :

[0156] Read-after-write (RAW)

[0157] I f an operand is read from a register for a given instruc- tion, the value read must be that generated by the PRECED- ING instructions . I f a later instruction would write into this register, the value retrieved at a later time but for use by a preceding instruction would be incorrect . This is known as a read-after-write (RAW) hazard .

[0158] Write-after-write (WAW)

[0159] I f data is written repeatedly into the same register or memory location, the value that should finally be found in that register must be that of the last instruction writing into the array . Now, i f a later instruction is out-of- order executed early on, a possibility exists that an ear- lier instruction executed after the later instruction overwrites the result . This would result in wrong data . This is known as Write-after-write (WAW) hazard .

[0160] Write-after-read (WAR)

[0161] A Write-After-Read hazard occurs i f an issued instruction found later in the sequence already tries to write to a register although the register has not yet been read by an instruction earlier in the sequence , as - even though the instruction found earlier in the sequence had previously issued- could not yet complete until then .

[0162] I f data is read from a register or memory location must return the last prior value written to that location, and not one written programmatically after the read . This is a sort of false dependency that can be resolved by renam- ing . WAR dependencies are also

[0163] 21

[0164] SUBSTITUTE SHEET (RULE 26) A method to deal with these hazards has been developed by To- masulo in the late 1960s and has been used - with an amendment for branch prediction- ever since . The basic idea behind the Tomasulo algorithm is to provide more registers than needed according to the register set provided according to the in- structions set . These additional physical registers are then used to resolve data hazards by allocating physical registers to the logical registers found in the sequence of instructions and to make use of this allocation in a manner resolving the hazards .

[0165] Obviously, this in turn necessitates to keep track of the al- location of the physical registers and their use , so this in- creases the overall amount of hardware needed to implement out-of-order processing in the art .

[0166] In more detail a number of di f ferent elements are needed : be- fore issuing an instruction, the registers used by this in- struction must be renamed and for this , a number of so- called "reservation stations " are provided and used as buf fers which fetch and store instruction operands as soon as they are available .

[0167] Each reservation station corresponds to one instruction and holds a plurality of fields , namely an indication whether the reservation station is in use ; the actual operation to be exe- cuted, the actual value of the source operands i f available , the reservations stations that will produce the values needed as operands for the operations i f these values are not yet available , and an of fset or ef fective address for a load / store operations i f the operand is to be loaded ( obviously, an oper- and can only be either loaded or generated by another instruc- tion, so only one of the corresponding two fields ( other res- ervation station or of fset ) can be used . The reservations sta- tions are used according to the instruction order of the pro- gram .

[0168] From this , it will be understood that a reservation station holds either the information about the location from where op- erands can be obtained until the operands themselves are available or alternatively holds the operands themselves .

[0169] Once all operands needed for the operation are available , the instruction can be executed . It will be understood that new instructions will be entered into the reservation stations as per the program order, the execution of the instructions will only depend on the availability of the resources needed for execution .

[0170] Then, by executing an instruction, results are generated and by broadcasting the results on the common data bus , the reser-

[0171] 22

[0172] SUBSTITUTE SHEET (RULE 26) vation stations waiting for the results will receive it so that processing of the next instructions can proceed .

[0173] However, results generated will not only be consumed by other processes ; results also need to be stored in physical regis- ters or in memory . As explained above , the problem with out- of-order executions is that two subsequent instructions might write to the same register, with the instruction found earlier in the stored sequence of program steps being executed later due to the out-of-order scheduling so that an outdated result could be found in memory or a physical register .

[0174] To avoid this , the instructions are not only sent to a reser- vation station, but also , a list of instructions in program order by adding each instruction according to the program or- der at the end of a so-called reorder buf fer .

[0175] Rather than immediately writing results to a memory or its fi- nal destination register, any result generated will then first be entered in the reorder buf fer at the location corresponding to the respective instruction that has generated the result .

[0176] Any entries in the reorder buf fer will only be written to memory or a register once the corresponding buf fer entry has reached the top end of the list and the corresponding instruc- tion has completed . Once the result is written to memory, the entry in the reorder-buf fer is deleted and the subsequent en- try in the buf fer can be processed . In this manner, it is made sure that for any entry reaching the top of the reorder buf f- er, all preceding instructions have been completed and no write hazard exists . This also allows to execute predicted branches : i f the prediction is found to be incorrect , it is possible to simply delete all subsequent entries in the reor- der buf fer .

[0177] As can be understood from this , out-of-order execution in the art necessitates both a signi ficant amount of hardware ( reser- vation stations , common data bus , reorder buf fer ) and energy for the operation thereof .

[0178] Simultaneous multithreading from a practical viewpoint

[0179] Simultaneous multithreading ( SMT ) is a further technique for improving the overall ef ficiency of superscalar CPUs by ex- ploiting hardware multithreading . Simultaneous multithreading permits multiple independent threads of execution to better use the resources provided by modern processor architec- tures . Because a vast number of measuring methods increase the conflict on shared resources , measuring or even agreeing on the meaning of certain measurements can be di f ficult .

[0180] 23

[0181] SUBSTITUTE SHEET (RULE 26) However, measured energy ef ficiency of SMT with parallel na- tive and managed workloads on historical 130 nm to 32 nm Intel SMT (hyper-threading) implementations have shown that in 45 nm and 32 nm implementations , SMT is extremely energy ef ficient , even with in-order Atom processors .

[0182] For more modern systems , it can be understood that SMT also exploits concurrency ef fectively while requiring very little additional dynamic power . It is found that even where perfor- mance gains are minimal , power consumption savings can be con- siderable . It is also noted that while in most current cases , SMT is about hiding memory latency, increasing ef ficiency and increasing throughput of computations per amount of hardware used, it would also be possible to provide redundant computa- tion by SMT to allow for some level of error detection and re- covery so as to increase safety, in particular in a mission critical environment such as in medical or defense applica- tions .

[0183] The device according to the present invention will now be de- scribed.

[0184] In a practical embodiment , the device is a multicore proces- sor .

[0185] Each core has a frontend including circuitry for instruction fetch, branch prediction, register aliasing and instruction issue as well as an execution backend including processing el- ements arranged in an array-like manner and interconnected by a bus system and special function units . As will be understood from the description hereinafter, the device of the practical embodiment is adapted to support various computing paradigms such as reconfigurable , out-of-order, data- flow, in- / near- memory and neuromorphic computing and the cores are capable of operating in parallel at least in certain types of operation modes .

[0186] In more detail , each core contains 5 thread-independent pro- cessing units , namely one 64-bit RISC-V in-order processor and four arrays of processing elements s interconnected by a bus system with additional circuitry in the core in particular to handle operation therof . It will be understood that instead of a RISC V instruction set architecture , the processor could al- so be built to execute instructions from other instruction sets . Accordingly, while using a Risc-V - embodiment processor is considered advantageous as it is well established in the industry .

[0187] The array may have a si ze of 16x 16 execution units , which is signi ficantly larger than the number of registers addressable by the instruction set of the multicore processor .

[0188] 24

[0189] SUBSTITUTE SHEET (RULE 26) It is well known that multicore processor are produced in a variety of variants for different performance levels. Here, independent of the performance level needed, the array size may be the same for all different variants, with processor variants having a higher performance being provided with a larger number of cores and / or a larger number of arrays per core. This allows to optimize certain code parts for execution on e.g. a 16x16 processing elements and compilers and software developers can optimize for such .e.g. 16x16 arrays. Unless upcoming algorithms will require larger arrays, it is likely that future core architectures will then maintain use multi- ples of 16x16 so as to allow better execution of legacy code. It will be understood that the number of processing elements in the 16x16 array is larger than the number of registers used in the Risk-V-instruction set; in particular, it is several times larger. There is no need that the number of processing elements is an integer multiple number of the register number of the Rise V instruction set, but it is preferred if it is at least 3 times larger, in particular at least 4 times larger, more preferred at least 5 times larger. Also, in case the de- vice would be designed as an architecture implementing a dif- ferent instruction set with a different number of registers, care will be taken that the number of processing elements in the array remains substantially larger than the number of reg- isters used by the respective instruction set, for example at least 3 times larger, in particular at least 4 times larger, more preferred at least 5 times larger. Reference is however had to the use of so-called extended registers which allow to use a very limited number of processing elements in the array as has been described by applicant's co-founder Vorbach else- where before, emp . the documents cited above.

[0190] Where the overall integrated device includes a plurality of cores - as in the case described-, each core may have a number of arrays of processing elements, but preferably, all arrays will have the same size, so that an 16X16 array of processing elements is used and preferably, the size of the array is maintained both over all devices within a series ("generation" of products) as well as from generation to generation so as to allow use of the same code and the use of legacy code. Should future processing needs increase very significantly, it would however be possible to enlarge the array e.g. to a 32x32 array and to then use only a fraction thereof for legacy code, even though this would not be particularly preferred.

[0191] The processing elements will be suitable to execute the most common operations such as integer multiplication, addition, subtraction and logical operations, e.g. for comparing oper- ands ( a<b, a>b, a=b, a=0 asf.) .

[0192] 25

[0193] SUBSTITUTE SHEET (RULE 26) In addition to the 16x16 array of processing elements where each processing element comprises the same arithmetical logi- cal unit to implement the same arithmetical logical opera- tions , a number of processing elements for operations less frequently instructed such as floating point multiplication units , dividers as f . are provided as additional elements .

[0194] The processing elements execute di f ferent threads and compete for I / O and memory bandwidth . For each core , the instruction cache , LI data ccaacchheess aanndd aa TLB are shared among the pro- cessing units . AAllssoo ,, iinn tthhee practical embodiment described here , cores are ccoonnnneecctteedd ttoo a shared highest level cache and a variety of peripheral interfaces .

[0195] Note that the arrays can be used for di f ferent types of pro- cessing such as out-of-order processing and loop acceleration as will be explained hereinafter ; thus , the arrays are called "Universal" Processing Arrays . Examples of the arrays Univer- sal Processing Arrays and ALU-Blocks are described e . g . in the patents mentioned above . In the current implementation, a UPA is compatible with the RISC-V Instruction Set Architecture .

[0196] With respect to the front end, the cores share some elements such as the Memory Management Unit (MMU) and the respective Translation Lookaside Buf fers ( TLB ) , which are necessary be- cause the device is to be used with an operating system, so that during operation, the computer ' s operating system, through a combination of hardware and software will map memory addresses used by a program to virtual addresses which need to be translated into physical addresses in computer memory . It will be understood that here , the translation lookaside buf fer ( TLB ) may be a conventional memory cache that stores the re- cent translations of virtual memory to physical memory . However, each core will have its own instruction fetch unit for requesting instructions from the memory hierarchy, e . g . the first level instruction cache , as well as its own instruc- tion decoder for decoding the instructions received from the memory hierarchy, and its own issue unit for issuing the re- ceived and decoded instruction to the respective processing elements , and a thread scheduler for scheduling the use of the front end resources for the thread being executed on the re- spective processing elements .

[0197] In the practical embodiment discussed, threads can be sched- uled for example based on the availability of resources , data and instructions : Those threads are scheduled for using the frontend ( or other shared resources such as the data memory hierarchy) which processing element is ready and can accept new instruction, which input data is available and output data can be transmitted, and which instructions are available in the memory hierarchy for immediate access .

[0198] Each core will also have a record used as a register position- ing table where for all registers according to the instruction

[0199] 26

[0200] SUBSTITUTE SHEET (RULE 26) set , an entry is made indicating to what processing element an instruction writing into the register was issued .

[0201] Then, a machine control unit is provided that keeps track of the state of execution of instructions by the processing ele- ments and is connected for this to the processing elements .

[0202] In the practical embodiment discussed, the bus system for transmitting data between the processing elements is prefera- bly capable to concurrently transmit multiple data words hav- ing associated therewith IDs uniquely identi fying a data source processing element . The bus system is arranged for broadcasting data on the bus and for snooping the bus system so that results generated by one processing element for use as input data by other processing elements can be identi fied and stored in the corresponding operand input registers of the processing elements needing those . Note that in an array, hor- i zontal and vertical interconnections are needed and thus , hori zontal and vertical busses are provided .

[0203] Regarding additional elements :

[0204] As additional elements , special function units can be provided to implement functions and opcodes that are either not data processing related or are rarely used and expensive in hard- ware , such as double precision floating point , 64-bit wide multipliers and dividers . General purpose load / store units are also located at the side .

[0205] In the practical embodiment described here , the additional el- ements may also comprise a programmable gate array ( element ) , GA ( or GAE ) , in at least one core . In the practical embodiment described, the programmable Gate Array provided is arranged adj acent to core 1 and at the side thereof and is controlled by the tasks executed on Core 1 .

[0206] As will be understood hereinafter, the processor core of the present invention will be perfectly suitable to process data in a dataflow manner in its loop acceleration mode using the coarse grained processing elements of the 16x16 array of pro- cessing elements . Accordingly, in order to be of additional use , the programmable gate array provided is fine grained so that for example input data streams received via a sseerriiaall in- terface in a dataflow manner can be handled in a bitwise man- ner . Thus , the programmable Gate Array, includes a programma- ble FPGA-like Gate Array, namely runtime configurable LUT4 logic cells , each including a Flip-Flop, with a total of around 6000 logic gates that can be used; but the programmable Gate Array in the practical embodiment described here will al- so comprise eight SRAM blocks , each 512x8 large , and four 16-

[0207] T1

[0208] SUBSTITUTE SHEET (RULE 26) bit multiply-accumulate DSP elements all embedded in the Gate Array as well as a PLL is provided for frequency synthesis .

[0209] In the practical embodiment described, the programmable gate array is arranged such that the configuration is retained when switching tasks .

[0210] Furthermore , the programmable Gate Array is arranged such that it is accessible in two di f ferent manners , namely one the one hand as a custom instruction as defined in the RISC-V ISA and, alternatively on the other hand, aass a peripheral device .

[0211] Consequently, the programmable Gate Array, GA has interfaces to the core ' s front end and register file and in addition is addressable in the 10 address space .

[0212] In the practical embodiment described, the programmable gate is connected to 24 configurable 10 pads and further to a bridge to the peripheral bus of the processor so that it has access to certain on-chip peripherals through that bridge .

[0213] As mentioned, other additional elements can be provided adj a- cent the array of processing elements . For example , it is not useful to provide every single processing element in the array with the capability to execute operations such as divisions that are needed rarely but require a large area for their im- plementation . One or several division units could thus be pro- vided at the side of the 16X16 array and in case a division needs to be calculated, the corresponding unit can be used . Note that even in case where the ( smaller ) processing elements of the 16x16 array only have one set of input registers and one output register, it would be possible to provide a plural- ity of input register sets and a plurality of output registers to such larger units as this would allow to re-use such large area units for data processing in a shared manner very much in the same way as known per se in the art even though it has not yet completed a previous use . Implementing such rarely-used processing elements outside the actual 16x16 array has a num- ber of advantages . Inter alia, the processing elements of the 16x16 can be smaller, reducing the overall si ze of the array and thus the time needed for transporting data ( such as in- struction data, operand data and, i f necessary, processing el- ement status data ) within the array- accordingly, the array can be operated at higher frequencies even without providing pipeline registers in a bus .

[0214] It should be understood by now that other additional units might also be provided, e . g . units that are speci f icially de- signed or suitable to process branching intensive algorithms . The skilled person will understand that such additional units are known in the art . Also , the mere fact that only a limited number of di f ferent additional units is mentioned is not to imply that other additional units should be excluded .

[0215] 28

[0216] SUBSTITUTE SHEET (RULE 26) Regarding additional circuitry : interfaces

[0217] In the practical embodiment described here , the multicore pro- cessor has a number of peripheral interfaces to memory such as DDRS , SD3 . 0 ( Secure Digital Memory Cards ) and SLC / MLC Single- Level-Cell ( SLC ) / Multi Level Cell NAND memory . In addition, in the practical embodiment described here , the multicore pro- cessor has an interface for a number of PCIe x4 bus system, at least one of which can be configured to operate as 4 independ- ent PCIe xl interfaces and the multicore processor has a syn- chronous 16-bit bus as well as several serial data I / O inter- faces such as 4USB3 . 2 interfaces , a CS I interface , 2 Gigabit Ethernets and 2 SATA3 interfaces . Furthermore , in the practi- cal embodiment described here , both the processor cores and, separately, the programmable Gate Array have access to a num- ber of interfaces , namely in the practical embodiment de- scribed here , one UART interface , two I3C interfaces , 4 QSPI QSPI interfaces , two CAN interfaces all as Low / medium data rate serial interfaces and a SERDES 16Gb / s interface as a con- figurable serial .

[0218] Regarding additional circuitry : CACHES

[0219] A cache system is provided as additional circuitry for the processor . The cache system is organi zed as a three-Level cache system . In more detail , in the embodiment described here , each core may include a level 1 cache system of a 64KB I-cache and a 64KB D-cache . Then, a level 2 cache is provided which in the embodiment discussed here is 512KB large and physically integrated in the universal processing arrays .

[0220] The cache system iiss arranged ssuucchh that in the Loop- Acceleration-Mode to bbee described llaatteerr,, processing elements can directly access portions of the LL22 cache . For this , the processing elements are allowed direct access to dedicated ad- dress windows , bypassing the LI cache system . Data trans fer to external memory is under control of the L2 cache controller so that data coherence can be ensured .

[0221] Finally, the shared L3 cache iiss 66MMBB llaarrggee and consists of a plurality of cache banks , each of which is concurrently acces- sible .

[0222] MODES OF OPERATION

[0223] OUT OF ORDER MODE

[0224] The device of the present invention can be operated in di f fer- ent modes .

[0225] In particular, it is capable of executing instructions in or- der as well as out-of-order and to accelerate the execution of loops in a particular loop-acceleration mode .

[0226] 29

[0227] SUBSTITUTE SHEET (RULE 26) To simpli fy the explanation of the out-of-order operation and to allow for a better understanding of the di f ferences to op- erating a processor according to the Tomasulo algorithm, a situation will be described during which no data need to be loaded from outside , where instructions have already been fetched and where all operands of the units considered both come from within the array and will only be sent to processing elements in the array . Furthermore , it is assumed that no branching will occur .

[0228] In this manner, the basic principle of out-of-order operation can be briefly explained in general terms so as to give a short overview to the reader . After operation has been ex- plained for a simple example , it will be explained in some more detail , outlining possible variations and improvements . Accordingly, further explanations can be made how instructions are fetched, how data is loaded and stored, how branches are predicted as f .

[0229] Regarding the basic principle of out-of-order operation in a nutshell :

[0230] For out-of-order operation, instructions are fetched in order and issued in this order to processing elements of the array that currently are not in use . I ssue stalls until the required resources become available , e . g . currently used processing el- ements become free again .

[0231] Rather than storing the results obtained by a given processing element in a speci fic register and to then retrieve from this speci fic register the previously obtained result as operand for a subsequent instruction, the results are transmitted from a generating processing element to any processing element that subsequently needs these results as operand input and once it is determined that a processing element will not have to pro- vide its previously generated results any longer, it will be re-used .

[0232] For this to work properly, it is inter alia necessary to keep track which speci fic register currently relates to what pro- cessing element ; the PEs are interconnected by a bus system for transmitting data, preferably capable to concurrently transmit multiple data words . Trans ferred data have associated IDs uniquely identi fying its source PE so that data can be transmitted by broadcasting rather than routing the infor- mation .

[0233] Accordingly, source registers are renamed at issue with the unique ID of the processing element producing the source data and each processing element will snoop the bus system for its source data which it will store into its operand ( input ) data

[0234] 30

[0235] SUBSTITUTE SHEET (RULE 26) registers . Once all operands are available , the PE performs the operation, stores the result in its result output register and transmits it together with the PE ' s ID to the bus system as necessary . Data remains in the PE ' s result register until the respective logical register has been reassigned as data target register to a later instruction and the content thereof has been provided as necessary to all processing elements needing these as operands . The processing element can receive a new instruction as soon as it can be considered free for this . Note that it might be possible to pre-load a new in- struction into an instruction register of a processing element i f it has finished its execution but has not yet provided its result to all processing elements subsequently needing the re- sults . Preloading an instruction into a processing element that has completed its execution has the advantage that the processing element could already listen to the bus system for obtaining its next operands , further reducing latencies .

[0236] For proper operation, the correspondence of processing ele- ments with registers and the current state of a processing el- ement is managed by a central unit .

[0237] Also , it is necessary to determine when a processing element can be re-used .

[0238] Then, for a high performance of the device , care must be taken to issue enough instructions and to provide both suf ficient I / O-bandwidth and suf ficient bandwidth within the device . Nonetheless , j ust for the explanation of the principle of out- of-order operation, any instruction- fetching arrangement would do per se , so it will not be explained in speci fic detail for the time being, although it will be understood that instruc- tions need to be fetched suf ficiently fast to achieve a high performance system; therefore , preferred methods and arrange- ments for fetching instructions will be described hereinafter, including arrangements that allow to predict most likely next instructions to be fetched .

[0239] In some more detail :

[0240] Still supposing the array is already in use and is producing results that need to be processed further by other units . In this situation, a free unit is to be instructed about the next operation it is to execute , has to wait for its operands , pro- cess the operands , provide the results to all units that need the results and once all this is done , be issued a further in- struction .

[0241] Also suppose that the instructions to be issued combine two input operands and produce one output result . In the typical

[0242] 31

[0243] SUBSTITUTE SHEET (RULE 26) command structure , an instruction such as "a+b" will need two input operands a and b and will generate one result .

[0244] It will be understood from what has been said above , that a number of instructions has already been fetched from memory in the sequence of the program and that a first part of the se- quence of instructions has already been issued to units in the array that have previously been available and that now, the next instruction in the sequence needs to be issued to the next unit that becomes available .

[0245] As the units to which previous instructions in the sequence depend on when their operands became available , there is no pattern, in particular no predictable pattern to what pro- cessing unit preceding instructions have been issued or what processing element generates the operands necessary . ( The only exception to this rule would be a highly speciali zed pro- cessing element of which only a single unit is provided for an array, e . g . a processing element having a floating point di- vider . These cases are disregarded for the moment )

[0246] Therefore , a record of the output registers is kept that indi- cates for all previously issued instructions that are still relevant to what processing element the instruction has been sent and what output register the result of the corresponding instruction should be written . In this record, the processing element can be identi fied by an identi fier, e . g . a set of two coordinates indicating the position of the respective pro- cessing element in the two-dimensional array or a correspond- ing linear number that is fixedly associated with the respec- tive array . Note that where the array is three-dimensional be- cause several dices each carrying a two dimensional array of processing element arrays have been stacked, the respective identi fier might be a 3-tuple of values or a corresponding ( larger ) linear number . Similar records have been suggested in the art and have been e . g . designated as register alias ta- bles .

[0247] While the number of processing elements and hence the width of each entry in the register increases with the number of pro- cessing units in the field, whereas the length of the record, that is the number of entries in the record is dependent only on the number of registers necessary to implement the register set selected . Accordingly, i f for example the processor is de- signed to use a Rise V instruction set where 31 di f ferent reg- ister can be accessed, the record should have a length of 31 , one entry for each register . As this record is actually stor- ing the position of the processing element in the array, it is considered to be a register LOCATION table similar to the more conventional register ALLOCATION table .

[0248] 32

[0249] SUBSTITUTE SHEET (RULE 26) In case multithreading techniques are used where a plural set of input registers and correspondingly more than output regis- ter is provided for an processing element so that one output register can be provided for each thread executable , the num- ber of record can be correspondingly longer, e . g . providing two times or four times 31 entries . This is particularly im- portant where large processing elements such as dividers are implemented with a plural set of input / output registers to op- erate them in a multithreading manner . Note that it would even be possible to implement this for the entire array of pro- cessing units to adapt it to multithreading . MARTIN : SOLL DAS REIN? ODER SPAREN WIR DAS AUF?

[0250] Now, even though operation takes place out of order mode , the instructions will still be issued in order to the array during operation . For this , immediately prior to being issued, any instruction to be transmitted to an processing element will be altered in that instead of indicating the registers where the operands could be obtained, the output registers of the pro- cessing elements producing the result is entered into the cor- responding fields . Also , the output register of the instruc- tion is entered into the corresponding record .

[0251] It is important to note that in the record kept for previously issued instructions , a situation will inevitably occur where a register designated as output register has already been used by a previously issued instruction . As a matter of fact , given the limited number of registers available in a register set and the fact that the array will contain signi ficantly more processing elements than registers provided for in the in- struction set the architecture is implementing, this situation is likely to occur on a regular basis .

[0252] In such a case , the previous entry in the central record may be overwritten .

[0253] Overwriting a current entry will not impair operation : Given that the instructions are issued in order, this means that any future instructions will need to know the output register of the processing element having received the most recent in- struction that is to write to that register . Also , any in- struction previously sent to the array and still having the old (now to be overwritten) information about the processing element providing its operands either has received its oper- ands or will still wait for its operands from the correct , namely previously designated

[0254] PLACING INSTRUCTIONS IN PROCESS ING ELEMENTS

[0255] Basically, the instructions which previously have been fetched in order are now placed in order processing elements , and for

[0256] 33

[0257] SUBSTITUTE SHEET (RULE 26) this , whatever processing element of the array currently is available can be used; in a device which can be implemented particularly easy- will be used, a queue of processing ele- ments currently available will be defined by a first-in- first out buf fer, that is , the instruction will issue to the pro- cessing element that has become available the longest time ago . Note that this is not an absolute necessity . It would al- so be possible to issue instructions in a manner taking into account factors such as proximity to the elements providing the operands . Proximity in this context can be construed to be a temporal proximity, that is , to relate to an expected time of trans fer of operands from the providing element to the ele- ment the current instruction is issued . However, when taking into account proximity, an evaluation of the proximity would be needed, for example taking into account the physical dis- tance and / or next-neighbor interconnections between processing elements i f such are provided, grouping of processing ele- ments , e . g . by being arranged on the same row interconnected by a bus as f . Also , as typically more than one processing ele- ment will provide operand data for an element providing a sub- sequent instruction, the relevant distances to all operand providing processing elements would have to be taken into ac- count , e . g . by issuing the instruction to that processing ele- ment for which the longest interconnection to processing ele- ments providing its operand is shortest . What is to be under- stood in this context is that such optimi zation will require both additional hardware and consume energy . Only i f perfor- mance is correspondingly improved will such additional expense be j usti fied . Therefore , it might be more useful to either j ust issue the instruction according to a First- In-First-Out principle or to at least reduce the amount of analysis needed, e . g . by only checking whether a processing element is availa- ble for being issued the next instruction within the same row as at least one processing elements providing operands . Note that such ef forts of improving performance are necessary for each instruction issued, but may result in only a limited ad- vantage with respect to performance . Therefore , checks may be simpli fied, e . g . where next-neighbor connections are provided by checking whether a next neighbor-processing element provid- ing operands is available for being issued a new instruction .

[0258] For the instruction to be executable by the element selected, the respective element chosen either needs to be informed what operation is to be executed and, unless routing between ele- ments is provided for, which other processing elements gener- ate ( s ) the input data needed for the execution of the instruc- tion .

[0259] To this end, the instruction is transmitted to the respective element together with the information relating to the elements providing the operands .

[0260] 34

[0261] SUBSTITUTE SHEET (RULE 26) In a simple implementation, transmission is ef fected by broad- casting the instruction together with the additional infor- mation relating to the processing elements from which the op- erands are to be received and an ID of the processing element for which the transmission is intended . All processing ele- ments or at least those available will listen to the broadcast and the respective one for which it is intended will react ac- cordingly .

[0262] This can be done either by speci fically routing the instruc- tion to the processing element or by broadcasting the instruc- tion to a plurality of elements

[0263] As routing for a single instruction requires vastly more ex- pensive hardware and control overhead, it is considered more useful at the time of this application to broadcast the oper- ands ( at least for the out-of-order-mode ) . So , the instruc- tions issued should include

[0264] - the address or ID of the processing element that is to exe- cute the instruction

[0265] -the addresses or IDs of the processing elements that pro- vide ( s ) the operands

[0266] - the actual instruction to be executed .

[0267] Accordingly, the instruction issuer will alter an instruction accordingly so that the respective information is included in an instruction broadcast packet .

[0268] The instruction broadcast packets are preferably broadcast on a separate instruction broadcast bus system . Note that this is not necessary and that in certain cases , the instruction broadcast packets could be transmitted over the same bus sys- tem as operand data . This holds in particular for low- budget applications where slow processing is of no concern, though .

[0269] The instruction bus system could be segmented in several ways . In particular, the processing elements can be arranged in a multitude of rows and each row can have a bus row segments with each bus row segment being connect to a column collection bus ( rows and columns could be changed, obviously) . Where sev- eral layers of stacked dies are provided, such an arrangement could be provided for each layer of the stack . It would be possible to mask the broadcast such that only those

[0270] Providing OPERANDS to processing elements

[0271] Then, once the instruction broadcast packet has reached its target processing element , several things need to be ef fected at the target processing element . For this , the data packet

[0272] 35

[0273] SUBSTITUTE SHEET (RULE 26) received must be evaluated and ensured that the operands can be received; for this , for example a mask can be set that en- sures that only operands broadcast together with the correct ID of the correct sender will be stored in an input register ( the execution element needing operands may snoop the bus to this end) . Also , the arithmetic-logic units within the pro- cessing element needs to be prepared for executing the opera- tion, e . g . by setting multiplexers internal to the processing element correctly .

[0274] The processing element so instructed then waits for the pro- cessing elements generating its input data to broadcast the data, stores the broadcasted input data received in its inputs register and once all input data has been received, executes the operation instructed . It then writes the result data into its own output register, broadcasts the result on the bus and keeps the result in its output register until it is determined that the data in the output register will no longer be needed .

[0275] For this , the operands need of course to be sent in the first place . Several strategies exist to ensure that this is the case .

[0276] In this respect , as busses will not be configured in out-of- order mode- given the real time randomness of processing ele- ment placement- the input operands are to be received by lis- tening to any information broadcast on the ( data ) bus and by identi fying the correct data as input operands .

[0277] In case the processing element j ust having received its in- struction would only passively wait for operand data to be sent by the processing element generating this data, basically three situations might occur :

[0278] The input operands the processing element is waiting for have been produced a while ago and have been broadcasted a while ago .

[0279] The input operands the processing element is waiting for might not have been produced yet . I f this is the case , the processing element would j ust have to continue waiting .

[0280] The input operands the processing element is waiting for might have j ust been produced and are broadcasted shortly after the processing element became ready for identi fying the correct broadcast address and for storing any needed operand . This is the ideal case , obviously .

[0281] In view of the first case , the processing element must be able to request transmission of its operand data from the pro- cessing element generating its input operands , or the trans- mission must be requested on behal f of the processing element . However, it probably would not be necessary to do this all of the time . Rather, the processing elements generating results could broadcast all results on the data bus once the results

[0282] 36

[0283] SUBSTITUTE SHEET (RULE 26) have been generated and have been written into the output reg- ister .

[0284] This has two advantages . First , frequently the timing of issu- ing an instruction that generates a result to be used as oper- and data in a subsequent instruction and of issuing the subse- quent instruction needing the result as operand data is fre- quently such that the processing element executing the subse- quent instruction is perfectly capable of obtaining the result from the bus . This holds in particular where results are broadcast on a bus so that an processing element having pro- duced a result might have to wait a couple of clock pulses un- til the bus is free for broadcasting . Where a large number of processing elements are waiting to broadcast their operands on the bus - and hence where the bus is busy- it might therefore be even advisable - or necessary, for that matter- to wait a couple of clock pulses before requesting operand data in the bus so as to not clog the busses by requests for data that are about to be broadcast anyhow . For particularly large arrays of processing elements , it would even be possible to implement a counter counting a speci fic number of clock pulses before re- questing operand data is allowed so as to reduce traf fic on the busses . Second, on average , waiting time can be shortened by broadcasting because in most cases , at least some receiving units might be ready to accept the input data, so better use can be made of the overall hardware . Obviously, the larger the array is , the larger the improvement is .

[0285] Yet , even then, i f no operands are received soon or immediate- ly after an instruction has issued to the processing element , broadcast of operand data of the processing element must be requested .

[0286] In a first implementation, it would be possible to broadcast any result data as an operand for subsequent operations from any processing element once this processing element has fin- ished its operation . In this manner, the corresponding operand is made available to processing elements executing subsequent instructions as soon as possible ; however, unless all operands broadcasted are buf fered in each processing element -which for the time being is not considered useful both in view of energy consumption and in terms of hardware needed for buf fering - immediate broadcasting once result data have been produced will only be of any use i f at least one receiving processing element is ready to accept the operand- which in turn necessi- tates that , by the time the result data reaches the pro- cessing element which is to execute the subsequent instruc- tions , the subsequent processing element has already received the subsequent instruction and has then been prepared for re- ceiving the operand . I f this is the case , then sending out re- sult data as soon as they have been generated might improve performance .

[0287] 37

[0288] SUBSTITUTE SHEET (RULE 26) Also , even then a case may arise where a result is used re- peatedly by di f ferent processing elements that receive the in- struction to be executed only some time after the execution of a preceeding operation which has generated the operands needed as input data . In that case , a mechanism must exist to ensure that the operands will be received by the corresponding subse- quent processing elements . This can be ef fected in several ways .

[0289] A first method would be a request from the receiving pro- cessing element once it has received its instruction . In that way, it is ensured that the operands will not be sent out be- fore the subsequent processing element is ready for reception of the operand; in particular, the subsequent processing ele- ment must be ready to snoop the bus and except the operand . However, preparing the processing element for snooping the bus and receiving its operand does not require signi ficant delays . Therefore , another-and more preferred-method would be to let the instruction sent to a given subsequent processing element also trigger transmission of the operands needed or to let the issue of the respective instruction rather than the instruc- tion itsel f trigger transmission of the operands , this context , it should be understood that several possibili- ties exist to issue instructions . Instructions could be issued via a broadcast indicating in the transmission the processing element for which an instruction is intended . However, in that case it is necessary to snoop the bus so as to determine whether or not a transmission is intended for the respective processing element . For data transmission, the bus system is snoop anyhow; accordingly, snooping circuitry would be availa- ble and would need to be altered only to some minor extent to also allow snooping the bus system for instructions . Yet , in particular where the array is provided for multithreading and thus a plurality of instructions should issue to the array each clock, broadcasting the instructions via a bus connected to all processing elements does not seem useful as the band- width of such bus system would have to be high enough to allow both data and instructions transmission over the same buses . The problem is ameliorated where a separate instruction bus system is provided; nonetheless , even in such a case issuance of several instructions per clock cycle might become necessary for multithreading . Therefore , it might be preferred to pro- vide direct connections from an instruction issue stage such as a plurality of instruction issue units ( e . g . one for each thread) . I f this is done , instructions can be directed from the issue stage to the respective processing element con- cerned . Where this is the case , it might be preferred to also provide direct connections to trigger broadcasting of operands to be transmitted from the processing elements that generate the results needed to execute the respective instruction . To this end, either a separate trigger network could be provided

[0290] 38

[0291] SUBSTITUTE SHEET (RULE 26) in addition to the instruction network, with direct intercon- nections from each instruction issue unit or instruction issue stage to each processing element ; or the same direct intercon- nections used for transmitting the instructions are also used for triggering output data broadcasting .

[0292] Where a network of direct connections for triggering output data broadcasting is provided, it is not necessary that each processing element evaluates whether a given instruction broadcast relates to a result it has recently generated or not and, i f it does relate , to automatically schedule sending of the results- or, where results are not yet available because the processing element that is to provide its results as oper- ands to an processing element executing a subsequent instruc- tion has itsel f not yet completed execution, prepare for transmission as soon as available . To this end, a flag could be set which once set , will be reset only after a requested broadcasting has been ef fected . In this way, the processing element receives several requests to broadcast its results , only one transmission is needed . When the transmission has been ef fected, the flag can be reset so that further requests will result in additional transmissions . It will be understood by a person skilled in the art that setting the flag should have an ef fect such as requesting the bus for operand trans- mission, signaling the need for transmission when polled, ac- cepting a token on the bus for operand transmission and so forth .

[0293] Where a separate trigger network is used, it is not necessary that each processing element evaluates whether a given in- struction (broadcast ) relates to a result it has recently gen- erated or not . It is anticipated that this saves energy over- all , even though the direct path for the transmission trigger must be set .

[0294] However, where broadcasting results is triggered by broadcast- ing requests to all processing elements , i f such request does relate to the speci fic processing element , the speci fic pro- cessing element could automatically schedule sending of the results- or, where results are not yet available because the processing element that is to provide its results as operands to an processing element executing a subsequent instruction has itsel f not yet completed its operation and where also no automatism for sending out results exists , a transmission could be scheduled after completion .

[0295] It will be understood that where a trigger network is used, it might be preferred to not automatically broadcast results once the results have been generated . This holds in particular where a large number of threads are simultaneously executed in an array and the case may occur where for a given thread the

[0296] 39

[0297] SUBSTITUTE SHEET (RULE 26) next instruction has not yet issued-accordingly, to prevent clogging of the bus system, broadcasting output data might on- ly be provided for after a trigger to reduce overall data traf fic .

[0298] In some embodiments , scheduling of transmission could be post- poned at least one or two clock cycles so that even where the path between instruction issuer and receiving processing ele- ment is signi ficantly longer than the path between instruction issuer and a processing element having to provide operands , the operands will not be sent before the receiving processing element is prepared for accepting the transmission . It will be understood that such a situation might occur where the arrays become particularly large , in particular signi ficantly larger than the 16 X 16 processing element-arrays currently consid- ered . Nonetheless , it is the current understanding that for the hardware design currently considered, such measures are not necessary . Also , depending on the overall number of pro- cessing elements that have to transmit or re-transmit their operands , situations might arise anyhow where the transmission of operands will hardly be delayed by waiting a couple of clock cycles . For the record, it should also be noted in this context that the busses need not be clocked with the same clock as the processing elements but could be clocked either slower or faster .

[0299] Another way to ensure that operands do in fact reach the re- spective processing element would be to let the processing el- ement request its operands . In this manner, it can be ensured that the processing element will definitely receive the oper- ands needed, independent of the timing of the preceding pro- cessing element . This may result in a somewhat slower data throughput compared to triggering via a dedicated point to point network . Nonetheless , situations may arise where overall processing ef ficiency is not or not signi ficantly adversely af fected by processing elements having to request broadcast of the operands ; in particular, a limiting factor to array per- formance might be power dissipation and heating of the respec- tive structures within the array . In such a situation, the slower data throughput will not reduce overall performance or will at least not reduce overall performance signi ficantly .

[0300] Still , a method where all operands need to be requested can reduce traf fic on the data busses as no data that currently is not needed yet will be sent ; otherwise , data could in particu- lar be send out too early, because the results obtained by one processing element would only be needed much later in the al- gorithm . On the other hand, results from executing an instruc- tion are frequently needed for executing an instruction imme- diately following .

[0301] 40

[0302] SUBSTITUTE SHEET (RULE 26) Accordingly, after results have been generated, they might be needed immediately afterwards- and accordingly, in such situa- tions , it might be useful to broadcast them rather than wait until they are requested . Whether or not this is to be pre- ferred may depend on typical algorithms executed in the array, on the number of threads simultaneously handled within the ar- ray and the like .

[0303] From this simple consideration, it can be seen that a determi- nation as to what way of triggering transmission of result is most favorable may strongly depend on the algorithms executed ( together with other factors such as bandwidth of busses , pow- er considerations as f . ) .

[0304] It should thus be understood by a person skilled in the art that the speci fic selection of a preferred method of operand transmission triggering may not j ust depend on the use case and parameters such as the overall number of processing ele- ments , the relative clocking speeds of instructions busses ( i f provided separately from the operand busses ) , operand busses , processing elements , instruction issuer and from parameters such as typical number of clocks for generating a result from operands , but also on the overall physical si ze of a device , the number of processing elements in a device , segmentation of the bus system i f any, the use case as f . Nonetheless , provid- ing a separate output data transmission trigger network in particular with point-to-point interconnections or point-to- few-points- interconnections ( such as from the instruction is- sue stage ( s ) to processing elements ) will be clearly preferred according to the present understanding of the inventors .

[0305] It will be understood that di f ferent methods of triggering transmission of operands could be combined . For example , even where operands are automatically transmitted by the generating processing element , it would still be possible for a receiving processing element to request a re-transmission as the initial transmission might have occurred to early and has been missed . Also , some results may be particularly important as they are likely to be needed by a lot of di f ferent processing elements already waiting for them and thus , immediately broadcasting them could be useful .

[0306] Furthermore , in particular where a round-robin access to the bus system for requesting ( i f done by the processing element ) and / or for broadcasting operands is given, it would be possi- ble for a processing element that has only recently received its instruction but has not yet received all of its operands to detect the operands needed on the bus even prior to re- questing them, as e . g . a case might occur where other pro- cessing elements have already requested the same operands or the operand generating processing element has only recently generated the operands and is transmitting them before the

[0307] 41

[0308] SUBSTITUTE SHEET (RULE 26) subsequent processing element needing the operands has re- quested the corresponding operands .

[0309] I f the operands have been generated already by the correspond- ing processing element to which the request is addressed, it will simply re-send the data in its output register . Other- wise , that is , i f the data has not yet been generated, the re- quest can simply be ignored, because once they are generated, they will be broadcast anyhow , (unless no provision is made to broadcast , all results as soon as they are generated - in that case , i f a request for a result has been sent to a given processing element , a flag should be set there to trigger transmission of newly generated results ) .

[0310] It may be useful that an processing element snoops the bus not j ust to identi fy the data operands it needs but also to listen to requests for these data operands as in case other pro- cessing elements need the operands as well , no additional re- quest is necessary and the data operands will be sent even without such request . Again, this keeps data traf fic on the busses low and thus enables to use a simple bus system even for rather large arrays , thus increasing possible array si ze and / or reducing consumption of power .

[0311] Accordingly, at least some time after issuing the instruction to the processing element , all operand data will be available , the operation instructed will be executed, the result will be written into the output register of the processing element and will be broadcast on the bus system for use by processing ele- ments to which subsequent instruction will be issued .

[0312] Thereafter, a decision must be made whether and when to re-use the processing element , that is when to consider the pro- cessing element to be available again .

[0313] To do so , states are used and these states are explained here- inafter . Note that while other and / or additional states might be used, the present set of states is considered to be imple- mentable and handled particularly easy, so while not restric- tive , the set of states suggested here is useful at least as a simple example and as an embodiment preferred for the ease of its implementation .

[0314] For deciding whether a processing element can be re-used, it basically must be made sure that all processing elements that need a result generated by the processing element as input op- erand have received or will receive their input operands . Also , the information must be processed in a manner where no problems occur because information is wrong or outdated, in particular to avoid that decisions are made in view of outdat- ed information-

[0315] 42

[0316] SUBSTITUTE SHEET (RULE 26) It will be understood that a decision whether input operands have been provided from a given processing element to all oth- er processing elements that might need them cannot be made without knowledge of the sequence of instructions as a case might arise where actually all processing elements to which instructions have currently been issued already have been pro- vided with their input data, but additional instructions that might be issued in the future might still need the correspond- ing information . As a matter of fact , such later instructions might not even have been fetched at a given time .

[0317] The only case when it can be known that in an upcoming se- quence of instructions no further future instructions will need to have access to a given output operand is once a later instruction re-uses the register the output register of that processing element currently represents . Thereafter, no later instruction will need the speci fic register content .

[0318] I f , as explained above , a central record is kept which indi- cates which instruction set register currently is represented by a given processing element output , once the entry of the central record is updated when preparing for issuing new in- structions , it is clear that no additional later instructions will need access to this corresponding output register- from that point of view, the processing element ( or processing ele- ment ) can be released . This information is thus available from issuing instructions and as fetching and issuing of instruc- tions happens in a centrali zed manner, this information is al- so held centrally .

[0319] Still , even though a processing element has been released, several cases can occur :

[0320] The processing element which has been released might not yet have executed its own operation, in particular because it has not yet received its own input data (note that sometimes , op- erands must be provided from outside , so there may be signi fi- cant delays ) . I f an operation has not yet been executed, this also implies that the respective processing element has not yet sent its output to all processing elements which need in- put from this processing element .

[0321] Alternatively, an processing element which already has been released might have finished its operation, but might not yet have broadcasted its result on the bus or might not yet have its result on the bus following a corresponding request . This might be the case e . g . because busses were not yet available .

[0322] Now : as an processing element can only be re-used after it has actually finished its operation, has broadcasted its output as needed and once it has been made sure that no upcoming in- structions will need the output , and as the information wheth-

[0323] 43

[0324] SUBSTITUTE SHEET (RULE 26) er future instructions might still need the output is availa- ble centrally in the corresponding record, status information relating to the execution of the instruction and / or the transmission of results is sent from each processing element to a central control that also has access to the register po- sitioning record so as to check whether the corresponding pro- cessing element has been released . Note however that status information about having finished an instruction need not be transmitted to all other processing elements in the array .

[0325] Accordingly, rather than sending such information via a sepa- rate bus , status could be transmitted from the processing ele- ments to the respective central control instance via a sepa- rate connection arrangement , e . g . one bus or a bus system com- prising several bus lines for collecting information from all processing elements and leading to the machine-control unit provided to monitor the processing elements state . It should be understood that instead of a control signal bus system, point-to-point interconnections from each processing element to the central control instance could be provided . That obvi- ously has the advantage that completion signals and the like can be transmitted immediately and thus without delay, result- ing in an overall higher throughput .

[0326] In a simple implementation, where the amount of status infor- mation transmitted is to be kept to a necessary minimum, the machine control station could distinguish the following states :

[0327] FREE The free processing element can receive new in- structions and the output register do no hold content that that might still be needed else- where .

[0328] The processing element can be cleared and any instruction and any content of an output regis- ter can be invalidated .

[0329] NOTE : The invalidation of output register con- tent can help to prevent that in response to a request from another processing element for re- sults in the output register, outdated results are still propagated .

[0330] EXECUTING The executing processing element executes an instruction that has been issued to it .

[0331] This execution includes waiting for a bus to become available for sending a request to those processing elements that are to provide the op- erands from their output registers ; sending the request waiting for the operands once the request has been sent , receiving the operands ,

[0332] 44

[0333] SUBSTITUTE SHEET (RULE 26) combining the operands and writing the result into the output register . NOTE The state is relevant because the pro- cessing element is no longer free and the pro- cessing element should not be disturbed while in this state .

[0334] RELEASED The output register of the released processing element has already been released in the regis- ter positioning table , indicating that no fu- ture instruction to be issued subsequently will still need the result therein; however, those processing elements that already have received their instructions might still request the re- sult as operand .

[0335] A processing element can thus be "released" even i f it has not even executed the operation it has previously been instructed to execute . It will be understood however that even though it has been released, no new instruction is al- lowed to be issued to and executed by such re- leased unit because a situation might arise where other elements might still need the re- sults .

[0336] Accordingly, being "released" might also in- clude

[0337] Still waiting for a bus to become available for sending a request to those processing elements that are to provide from their output registers the operands the released processing elements needs ; sending the request ; waiting for the operands once the request has been sent , receiving the operands , combining the operands and writing the result into the output register and sending out the results to processing elements that have received their own instructions prior to the release and are thus still waiting for their own operands .

[0338] COMPLETED The processing element has finished its execu- tion of the previously issued instruction, that is , the result has been produced and stored in the output register .

[0339] Other processing elements might still request that the results currently stored in the output register be ( re- ) broadcasted in the future .

[0340] 45

[0341] SUBSTITUTE SHEET (RULE 26) Therefore , the result has to remain in the re- sult register until the logical target register is released in the register positioning table .

[0342] TRAP A problem may occur in case an operation is not allowed, for example a divison by zero or an overflow .

[0343] In this case , the instru tion will not generate a useful result and thus , subsequent instruc- tions relying on these results would never com- plete . Therefore , the machine control unit should be informed so that e . g . an interrupt routine can be entered .

[0344] As the reaction to an overflow or the like cer- tainly must be di f ferent from the reaction to an error- free execution of an instruction, so should be the corresponding status transmitted to the machine control unit .

[0345] Note that while a trap state could be transmit- ted in the same manner as other states , it typ- ically necessitates to handle the condition that led to the trap state and thus where sta- tus transmission is arbitrated accordingly, would preferably be transmitted with a particu- larly high priority .

[0346] With these states , the most interesting aspect might be when and how a transition between states may occur .

[0347] Obviously, the central management unit can easily determine that a FREE state of a processing element is to be altered to a state of EXECUTING . This is simple , because as soon as it is determined that an instruction is to be issued to it , it should not be issued another instruction prior to having fin- ished (Note that in certain embodiments , a preloading of one or more instructions might be allowed; here , instructions to be executed later might be issued to the executing unit , but might not be executed before a corresponding ( trigger ) infor- mation has to be sent to the processing element preloaded in that way so as to trigger actual execution of a next preloaded instruction . Another possibility to use preloaded instructions would be a multithreading embodiment which however would need a plurality of input and output register sets for a given pro- cessing elements . However, in a simple embodiment as is more easy to understand, no provision is made to issue multiple in- structions to the same processing element that is still oper- ating- with the sole exception that a single instruction might be preloaded .

[0348] Now : The central management however has no per se knowledge as to whether or not an execution has completed, that is whether

[0349] 46

[0350] SUBSTITUTE SHEET (RULE 26) a given processing element has received its operands , has com- bined them as needed and has written the result into its out- put register . Yet , even though it might theoretically be pos- sible to provide for a more local management of states of each processing element in the processing elements themselves , it is likely that any such method would necessitate an extremely large communication overhead for exchanging information be- tween units so that even i f a theoretical approach would be found, it is unlikely to be of any practical use in view of energy and silicon area . Therefore , track of the states should be kept at one single location : a machine control .

[0351] For this , information relating to the state of the processing elements must be managed centrally .

[0352] Information relating to the issuance of instructions is easy to obtain centrally, as instructions are issued from a some- what central unit , but for keeping track of the states men- tioned above , information must also be transmitted from the processing elements to the central machine control unit so that decisions can be made when an processing element can be considered to be available again . Note that where such infor- mation is managed centrally, this can be done in one of sever- al units ; for example , such information could be included in the register positioning table , in a separate file or at the instruction issuer . The later - keeping a file separate from the register positioning table is particularly useful where the separate file is arranged close to or at the instruction issuer as it is there the information about the state is real- ly needed and where in particular TRAP state information should be made available as soon as possible . The separate file and any circuit associated therewith can be construed and understood to be part of a central machine control unit which however need not be arranged in a single contiguous location on the chip .

[0353] For this , certain status information needs to be sent from the executing units to the central machine control unit which of course must receive the status information in a manner allow- ing to relate the status to the speci fic processing element sending the status information . This information could be an ID of the instruction that the processing element is executing and / or the address of the processing element . Note that as long as only one instruction is held in a processing element at any given time , it is suf ficient to j ust hold the infor- mation relating to the instruction . However, above , multi- threading in conventional processors has been described, not- ing that by providing a multiple set of registers for an ALU in a core ( and other measures ) , multithreading could be ena- bled; accordingly, it might be necessary to also send an ID of the instruction or process for which the state of the pro- cessing element has changed .

[0354] 47

[0355] SUBSTITUTE SHEET (RULE 26) It should be understood that a multithreading-like approach could be taken in the present invention, in particular with processing elements having particularly large ALUs such as those including dividers . Here , multiple instructions could be issued to a respective corresponding processing element and execution could take place once the respective operands are available . Again, whether or not this is useful will depend on parameters such as si ze of ALUs vs si ze of additional in- put / output registers per processing element , additional commu- nication overhead as f . Also , it might be easier to implement a device and its toolchain without support of such multithread- ing .

[0356] Nonetheless , it should be understood that this is only an ex- ample of a particularly simple implementation of multithread- ing, namely by providing a plurality of output registers with- in a given processing element and by handling additional ID relating to the actual thread to which the output register re- lates . What is more is that even in a simple implementation, it might be necessary to not only provide a plurality of out- put registers , but to also provide the respective processing element with a plurality of input register sets for each thread the processing element can handle . This necessitates a rather large overhead per processing element which in turn is not useful in case a maximum of processing power is to be di- rected onto a single thread for certain applications . In par- ticular for applications necessitating a very high perfor- mance , it would be more useful to have rather simple pro- cessing elements so that the number of simple and thus small processing elements per chip area can be increased . It should be understood that while managing a larger number of pro- cessing elements may incur a penalty e . g . because a larger number of processing elements need to be managed, at some point the number of bus lines might have to be increased to handle the additional data traf fic, more instruction fetch / decode / issue units as f . might be needed, such penalty will typically be small and will not of fset the gain due to a vastly higher number of processing elements .

[0357] One method to allow multithreading would be to provide a plu- rality of arrays in one microprocessor chip so that each array can operate on one single thread . In that case , each array may have its own register positioning table and preferably its own instruction fetch / decode / issue circuitries . I f plural ar- rays are provided in one microprocessor chip, it will obvious- ly be preferred to allow coupling thereof ( coupling only some or coupling all and coupling preferably only under certain conditions such as in loop acceleration mode ) so that when processing data in a loop acceleration mode , an overall array si ze available for processing without altering the array set up becomes larger . It is noted and suggested however that even

[0358] 48

[0359] SUBSTITUTE SHEET (RULE 26) then, not all arrays need to be coupled together for pro- cessing data in a loop acceleration ( dataflow) mode ; rather, it would be possible to process data in a dataflow manner on a single array of a multi-array chip or on only some of the ar- rays ; it would also be possible to process data in a dataflow manner in di f ferent threads executed on a multi-array chip with each or some arrays executing di f ferent threads ; it is however noted that it might be preferred to have at least one array in the multi-array chip that is not operated in a data- flow manner at a given time , but remains in out-of-order mode at least while all other arrays of the multi-array chip are used in a dataflow manner . The reason this is preferred is that interrupt handling in response to e . g . traps typically relies on sequentially issued instructions and keeping one ar- ray in out of order mode reduces the time until a trap can be attended to .

[0360] Nonetheless , while a plurality of arrays provided in a given chip or chiplet system can be used for multithreading, an even better and more straightforward way to implement multithread- ing in a device according to the present invention exists , namely executing multiple threads on one and the same array .

[0361] For this , it is first of all suggested that a register posi- tioning table is provided for each thread that the arrangement of the present invention shall be able to execute simultane- ously .

[0362] Accordingly, i f an array is to be able to execute for example

[0363] 5 threads , 5 separate register positioning tables would be necessary .

[0364] In a typical implementation, the array might have a si ze of 16 times 16 processing elements . The microprocessor could be an implementation of a Risk V-instruction set architecture so that a maximum of 31 registers might be needed per thread ( as long as no out-of-order processing ins necessary) . Therefore , it would be easily feasible to handle 8 threads in the 16 times 16 processing elements-array . Accordingly, 8 separate register positioning tables would be necessary .

[0365] In some of the patent documents cited in the introductory por- tion ( including the patent documents cited explicitly as well as any patents issued from applications therefrom, any divi- sionals and / or continuations therefrom as well as patent docu- ments claiming the same priority) , it has been suggested to use eviction registers so that the number of registers that can be handled could be higher than the actual number of pro- cessing elements in the array .

[0366] For example , i f a processing element has finished the execu- tion of a given instruction, rather than storing the results

[0367] 49

[0368] SUBSTITUTE SHEET (RULE 26) within the processing element itsel f and thus blocking the en- tire processing element until the release of the register, the result could be trans ferred to a processing element having at least the register for holding the result and some additional circuitry allowing to snoop the bus system for requests of the corresponding result and further allowing to transmit the re- sult via the bus system to any elements requiring the results . In this manner, some chip area could be saved by providing such processing elements basically for use as " intelligent" registers and accordingly, i f this is done , the number of threads per actual executing processing element within the ar- ray could be further increased, provided of course that a suf- ficient number of register positioning tables would be provid- ed . Note that such intelligent registers could either be part of a somewhat more complex processing element or could be pro- vided separately from a processing for use by di f ferent pro- cessing elements or for alternative use in modes of operation requiring pipeline registers ( in such case and for such modes , snooping could in particular be switched of f ) . Also , where a processing element has a particularly complex arithmetic logic unit such as a divider, a more conventional processing element having a less complex arithmetic logic unit could be used as such " intelligent register" ; such processing element receiving results can be referred to as being a "proxy processing ele- ment" and a typical way to operate such proxy processing ele- ment will be described below . With suf ficiently advanced hardware design and technology, a large number of proxy pro- cessing elements could be placed in a separate die layer in particular where a processor has a 3dimensional arrangement of arrays . It can also be anticipated that a proxy processing el- ement will have a power consumption smaller than a convention- al processing element having an arithmetic logic unit consum- ing a lot of power . This will typically hold despite the fact that the proxy processing element will snoop the bus for some time for its data to arrive and will have to react to retrans- mit triggers . Given the lower power consumption and the fact that certain under certain conditions , the performance of an array might be limited by maximum power dissipation considera- tions , it is feasible to distribute proxy elements across the array to reduce the heat load density . Also , it would be fea- sible to prefer processing elements having reached a particu- larly high temperature for use as proxy elements as this may allow to cool down somewhat . To this end, a plurality of heat sensors can be distributed across an array and a selection of proxy processing elements can be made in view of a current temperature , either of a region the processing element is in or, i f measured, of the processing element itsel f . Note in that context that in out of order mode , it might be advanta- geous to select a processing element in view of its current temperature and / or in view of an expected power consumption of executing an instruction .

[0369] 50

[0370] SUBSTITUTE SHEET (RULE 26) Now, processing elements for di f ferent threads can be easily distributed at random to basically any place within the array without disturbing each other ; only when instructions are is- sued to the array must care be taken that no two threads issue an instruction simultaneously to the same processing element . This can be easily achieved, for example by a process of re- questing the processing element to accept the new instruction; in case the processing element accepts the request , it will deny further requests and any later requestor will have to at- tempt to send an instruction to another processing element . Another way could be to keep a list of processing elements that are free and thus have both completed the instruction previously issued to this processing element (note that this previously issued instruction could belong to by any of the threads ) and have been released by their threads . From this , it will be understood that the thread a processing element is used by may change from instruction to instruction issued to this processing element . Note that the list of processing ele- ments to which new instructions of di f ferent threads could is- sue is rather short , namely no longer than all processing ele- ments in the array .

[0371] It will be understood that for each thread, additional hard- ware other than the register positioning table will also be provided such as for example one program pointer per thread- and of course , it will be necessary to fetch and decode an is- sue a suf ficiently large number of instructions so that ade- quate use of the increased processing capabilities of multi- thread processor is made .

[0372] It will be understood by a person skilled in the art that re- trieval and issuing of a suf ficiently large number of instruc- tions necessitates suf ficiently powerful circuitry, for exam- pie with respect to instruction fetch and decode , memory ac- cess bandwidth and so forth .

[0373] Note that it is possible that for every thread, separate front ends each having instruction fetch, decode and issue units can be provided or that several threads might share a front end . Even with front end sharing, several instruction fetch, decode and issue units will typically be provided to achieve a suf fi- ciently high performance . Furthermore , it will be understood that for example , one program pointer per thread will be pro- vided .

[0374] In case certain particularly complex operations are executed by only some of the processing elements , for example divisions executed only by speciali zed ( large ) divider processing ele- ments , such speciali zed processing elements could have multi- ple input / output register sets , preferably at least one per set and / or a queue could be provided so that when you instruc- tions ( from di f ferent threads ) would only issue to such spe-

[0375] 51

[0376] SUBSTITUTE SHEET (RULE 26) ciali zed processing element once previously issued instruc- tions have completed . The later solution would however be less preferred in that a long period of waiting for operands of previously issued instructions might slow down the entire pro- cessing of the array . Accordingly, where speciali zed pro- cessing elements are to be used, a reservation-station-like unit could be provided in front of the speciali zed processing element , with the reservation station like unit receiving in- structions from all threads , waiting for the respective oper- ands of each instruction received and forwarding the received instructions to the speciali zed processing element only i f such forwarding can then be done together with the operands . In this manner, blocking of a speciali zed processing element in its entirety by particularly slow threads can be avoided . Note however that by proper use of microcode , by providing very small amounts of memory within the array and by providing suitable circuitry to execute sequences of microcode , complex instructions can be easily executed by a large number of pro- cessing elements having such circuitry and access to such small amount of memory . Accordingly, the need for highly spe- ciali zed processing elements might be rather low in practice .

[0377] It has been described above that each thread will need a mini- mum number of processing elements as defined by the number of registers in the register set of the instruction set architec- ture the processor is implementing . However, providing such bare minimum number of processing elements will not allow the thread to execute in out-of-order mode ; for this , more pro- cessing elements will be needed . Such additional processing elements need not be statically and equally assigned to the di f ferent threads . Rather, in order to maximi ze overall per- formance of a processor, it is advantageous to take into ac- count to which degree the threads can currently take advantage of additional resources . In other words , where more than the bare minimum number necessary to simultaneously process n threads is provided, the di f ferent threads may compete for the additional resources .

[0378] To this end, it should be kept in mind that even with a suf fi- ciently powerful circuitry for fetching, decoding and issuing instructions as described above and allowing out-of-order pro- cessing of some or all of the threads , the highest overall performance of the data processor might frequently be obtained when it is taken into account that di f ferent threads will be executed with di f ferent speeds , for example because some threads require more access to nonlocal data whereas other threads might have a higher rate of access to cached data or data stored in other processing elements ( or registers ) .

[0379] Also , a case may occur where di f ferent threads -even i f they show basically the same performance - have di f ferent priori- ties or where some threads , during out-of-order execution,

[0380] 52

[0381] SUBSTITUTE SHEET (RULE 26) make more ef ficient use of additional resources whereas other threads will not benefit from providing a particularly large number of additional resources e . g . because they have no or hardly any branching instructions . All these factors might in- fluence the frequentness with which for a given thread, new instructions will issue .

[0382] In this context it is also noted that frequentness of new in- struction issuing might also take into account a depth of cur- rent speculative execution by penali zing deeply speculating executions ( that is , multiple predicted branching can be pe- nali zed) .

[0383] Note that either an average frequentness can be considered for each thread, taking into account how many times an execu- tion has completed in some past clock cycles such as the 2 , 4 , 8 or 16 past cycles or that only the number of completions in the last clock cycle could be considered .

[0384] Whether the number of processing elements exceeding the bare minimum for executing n threads should be rather high ( result- ing in a number n of threads simultaneously executed in an ar- ray of a given si ze being rather low) or not may depend on factors such as I / O bandwidth of the microprocessor design, structure of internal busses , algorithms executed as f .

[0385] Furthermore , it will be understood that the question whether data needed as input for data processing is available easily with low latency and a high access bandwidth or whether the latency of data access is high and the access bandwidth is low, can typically not be answered at compile time but can on- ly be assessed during runtime . For a thread where latency is high and data access bandwidth is low, it may not be useful to issue a large number of instructions that will block a large number of processing elements and thus prevent a large number of processing elements from processing instructions of other threads ; rather, issuing instructions for other threads could be preferred in particular over issuing additional out-of- order instructions for such other threads that do neither com- plete their instructions fast nor have a particularly high priority . The decision of how many instructions a given thread should issue in a particular period should thus be made by a scheduler at runtime to make optimum use of the array . Di f ferent measures could be assessed by the scheduler to de- termine a frequentness of instruction issuance for any given thread . For example , the time -or number of clock cycles , more precisely- between issuing an instruction in a given thread and the time until the instruction has completed could be de- termined . This time length takes into account in particular the duration until operands have been received; where the time or number of clock cycles is short , so is latency . Another measure could be whether or not a processing element is re-

[0386] 53

[0387] SUBSTITUTE SHEET (RULE 26) leased and / or free soon after an instruction has issued there- to , preferably measured again by a clock cycle statistic . Such statistic could average over all instructions issued in a thread since thread start , over all instructions within the last n instructions issued, such as the last 2 , 4 , 8 , 16 , 32 , 64 etc . instructions issued -with binary numbers of last in- structions being preferred for obvious reasons , although it is by no means necessary to restrict the number of last instruc- tions considered to only a binary number . It is possible to disregard outliers in the statistic or to not disregard outli- ers ; rather than using averages , other statistical values such as median values could be used to assess the frequentness with which instructions should issue in a given thread . Note that issuing instructions in any given thread need not and most likely will not be strictly periodical , for example be- cause interrupts need to be handled or because where a sliding average or median values calculated, such sliding value chang- es over time . Accordingly, the term frequentness is preferred over the term frequency .

[0388] Obviously, other methods to take into account the time needed for completion of an instruction or the time needed until re- lease of a processing element currently used by a speci fic thread or the time until a processing element is free could be used; the above methods are favorable in that the assessment or evaluation of time refers to signals from the actual pro- cessing elements itsel f within the array directly indicating that the ALU is available again .

[0389] Taking into account frequentness of completion of an instruc- tion is considered inventive and advantageous per se , even in a context other than the processor described here having an array of processing elements and a register positioning table .

[0390] For example , it is considered that completion signals or other status signals from ALUs could be used to determine the fre- quentness or relative frequentness of instruction issues for di f ferent threads in a conventional multithread processor as well , albeit it is considered that in the present invention, the possible number of threads that can be allowed at a given hardware expense is likely to be signi ficantly higher than in the prior art . For example , where 16 arrays of 16* 16 pro- cessing elements in a 31-register- RISC V instruction set im- plementation are provided, the number of threads would easily be at least 16* 8 threads , basically at the expense of only 16* 8 * ( 31 +x ) registers- with x being the maximum number al- lowed for out-of-order processing of a thread - plus the addi- tional circuitry for program pointers , instruction fetch and decode and managing the register positioning tables . Note in particular, that thus , the additional hardware overhead for handling these threads is rather marginal and that the large number of possible threads warrants measures to manage the in-

[0391] 54

[0392] SUBSTITUTE SHEET (RULE 26) structions issue frequentness of each thread for optimi zed performance .

[0393] Note that even where branch prediction as described elsewhere in this application requires additional copies of a register positioning table , the overall expense for multithreading with a large number of threads still remains low . Also note that allowing a rather large number of threads in an array will al- low to make particularly ef ficient use of hardware such as ad- ditional processing elements for out-of-order operation . For example , where only a single thread is allowed, a very large number of processing elements will typically be used only in deeply predicted branching having a high likelihood of a mis- prediction, whereas i f a large number threads use the same processing elements for executing only instructions belonging to a respective first branching, the overall probability of mispredictions is lower . It will be understood that this will have an influence on the most preferred number of threads exe- cuted in an array . It will also be understood that the number of threads simultaneously executed may be reduced below the maximum number possible in view of the given hardware , depend- ing e . g . on software requirements , priority as f . It is antici- pated that a register positioning table provided for a sepa- rate thread could also be used in executing a predicted branch with only some amendments to the circuitry involved and the method of its operation .

[0394] Note that in another embodiment of thread instruction issue frequentness management , the number of processing elements assignable to any given thread might ( also or exclusively) be determined in view of a priority assigned to the speci fic thread or a maximum number could be assigned independent of the priority; defining a rather high maximum here may be use- ful to allow execution of predicted branches , in particular with subsequent (nested) predictions - obviously, for such a case , the respective thread must make use of signi ficantly more processing elements than the number of registers provided for in the instruction set as the speculatively executed in- structions might need a particularly large number of addition- al processing elements . However, it is possible that only when e . g . a predicted branch is executed will the maximum number of processing elements be assigned . Otherwise , the number could be smaller without signi ficant adverse ef fects , but again, when allowing for out-of-order execution, the allowed number might still be higher than the number of registers according to the instruction set .

[0395] During operation, whenever the respective threads requires an additional processing element before the maximum number of processing elements allowed or guaranteed for that thread have been put in use by this thread, a new instruction may issue as soon as possible , that is once the next processing element be-

[0396] 55

[0397] SUBSTITUTE SHEET (RULE 26) comes FREE ; this allows use by other threads and in particular competition for resources as in the preferred implementation where issuing instructions for a thread will depend on the frequentness of completion .

[0398] It is possible that - once a thread has reached the maximum allotted number of processing elements it may use , a pro- cessing element currently used by this thread would first have to change its status to FREE before a new instruction may is- sue for this thread . Note that this does not mean that the very processing element becoming free will be used by the same thread again . On the one hand, other competing threads might also already be waiting for such a processing element as well and could be allowed to first issue their instruction because their frequentness is higher ; on the other hand, several pro- cessing elements from di f ferent threads might become free at the same time so that the thread continues to be executed on randomly varying processing elements in the array .

[0399] Note that even though restricting the number of processing el- ements assignable to a thread may reduce overall performance somewhat arti ficially, this might still be a preferred ef fect in e . g . cloud computing where a user should pay for higher performance .

[0400] In this manner, that is by giving the next processing element becoming free to a thread having not yet reached its ( allot- ted) maximum number of processing elements , it is not neces- sary to completely block the maximum number of processing ele- ments assignable to a given thread . This will allow for some competition for resources among threads ; nonetheless , enough processing elements can of course still be assigned to a spe- ci fic thread under speci fic conditions such as when a predict- ed branch is to be executed . Therefore , basically, it is still easily possible to run every thread in an array in an out-of- order mode despite simultaneous multi-threading .

[0401] Again, it will be understood that even for in order execution, the maximum number of processing elements assigned to a thread must be not smaller than the number of registers provided for in the instruction set to prevent a situations where the thread cannot FREE a processing element because all processing elements available are still needed . Obviously, it would be possible to use the proxy processing elements mentioned before instead of actual processing elements to alleviate this prob- lem to some degree .

[0402] Note that di f ferent methods could be combined for deciding to what thread additional resources should be assigned, for exam- ple where even though the maximum number of processing ele- ments assignable to a thread is limited, the sum of such maxi- mum numbers of allowed processing elements is still larger than the number of processing elements actually available as overhead (beyond the mere minimum for in-order-execution by

[0403] 56

[0404] SUBSTITUTE SHEET (RULE 26) all threads currently executed in the array) ; here , the number of processing elements actually assigned to a thread might de- pend on depth of speculation and / or on priority and / or on fre- quentness of completion an instruction .

[0405] Also note that it would be possible to provide a plurality of output register for each or some processing elements , e . g . for allowing use thereof by multiple threads with each thread us- ing a di f ferent output register : where the maximum number of registers usable in an array is larger than the number of out- put registers , a di f ferent output register could be used for any thread currently assigned to the processing element ( or proxy processing element ) .

[0406] Accordingly, what has j ust been described is inter alia a mul- tithreading capable data processor adapted for the (multi- threading) out of order execution of a (multithreading- multitude of ) sequences of instructions belonging to an in- struction set , the instructions referring to operands from a limited number of registers and generating results to be stored in one of the registers of the limited number, the processor comprising an instruction issue stage ; a number of processing elements larger than the limited number of reg- isters of the instruction set , each processing element of the number having inter alia a unique identi fier, operand input registers for input operand data and at least one output reg- ister, wherein a result generated by the processing element can be stored for transmission to one or more processing ele- ments processing subsequent instructions until it is no longer needed by subsequent processing element , and state signaling circuitry adapted for actively or passively signaling via suitable connections a state of the processing element indi- cating whether the processing element has received its oper- ands , the processing element has generated a result and / or the processing element is available to accept a further instruc- tion; the processor further comprising a register positioning table arrangement referencing the unique identi fier of the processing element which is to execute an instruction to the result register of the instruction; and a bus system for the interconnection of the processing elements , wherein the in- struction issue stage is preferably capable of simultaneously issuing n instructions per cycle with n >1 , with the instruc- tions being issued only to processing elements having con- firmed by their state signaling circuitry availability for ex- ecuting a further instruction, and the instructions simultane- ously issued belonging to n di f ferent threads ; the number of processing elements being larger than n times the number of registers used in the instruction set or the number of pro- cessing elements together with registers of an extended regis- ter file being larger than at least n times the number of reg- isters used in the instruction set ; the register positioning table arrangement comprises at least n register positioning

[0407] 57

[0408] SUBSTITUTE SHEET (RULE 26) tables with each of the n tables being arranged to store the unique identi fier of a processing element referencing which is to execute an instruction belonging to one of the n threads to the register of the instruction and the instruction issue stage retrieves information relating to the signaling of the state signaling circuitry in determining to which processing elements instructions are to be issued .

[0409] Furthermore , it is understandable that it is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention that the register positioning ta- ble arrangement comprises a processing element state listing common to all threads and listing for each of the processing elements of the number of processing elements at least whether the respective processing element has been released from its previous thread e . g . from noting that the register the pro- cessing element had to write its results to has been reused according to the thread the processing element was assigned to . Using a common or uni fied processing state listing common to all threads is preferred because it simpli fies issuing a plurality of instructions in each clock cycle . However, meth- ods other than using a uni fied processing element state list- ing could be used, for example by keeping a list of the states of the processing elements of the bare minimum processing ele- ment numbers for each thread and a separate common list for all "overhead" processing elements .

[0410] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that in such a processor, the issue stage is adapted to issue instructions only to pro- cessing elements which have been released according to the listing and to which according to the signals obtained from the state signaling circuitry of the processing element a new instruction can be issued .

[0411] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that the bus system has an ar- biter arbitrating between processing elements for operand transmission access and the processor is adapted for operating in at least two di f ferent modes with one mode being an out-of- order mode wherein access of a processing element to the bus system is arbitrated and one mode being a loop acceleration mode allowing for unarbitrated access to the operand bus .

[0412] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that it is determined that a first processing element is available to receive a new in- struction once all processing elements receiving operands from

[0413] 58

[0414] SUBSTITUTE SHEET (RULE 26) the output register of the first register have received there operands therefrom .

[0415] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that it is determined that a first processing element is available to receive a new in- struction once it has been released by its current thread and once it has either signaled that it has transmitted its result onto the bus system and / or all subsequent processing elements needing the result in the thread have signaled that they have executed their instruction .

[0416] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that the bus system comprises di f ferent busses for transmitting operands and instructions and / or status signals .

[0417] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that the n processing elements are grouped into m arrays with at least one array being ar- ranged to operate on a thread di f ferent from at least one thread in another group . In a further preferred embodiment , a pattern of multiple arrays is provided with each array having a number n of processing elements larger than the number of registers provided for in the instruction set implemented by the instruction set architecture processor of the present in- vention .

[0418] It will also be understandable that what is particularly ad- vantageous in a multithreading capable data processor accord- ing to the present invention is that at least one group of processing element , in particular an array in a multi-array pattern processor, is reserved to handle interrupts , in par- ticular traps .

[0419] With respect to cases where e . g . a large divider unit is used in a multi-threading like manner, a situation might arise where it is not enough to refer to such multi-threaded usable processing element with a single processing element ID . Ra- ther, where e . g . several divider executions unit are provided with multithreading or preloading capabilities and thus have to snoop the bus for several input operand data or transmit data from one of several output registers , a separate ID could be used for each set of input / output registers in a processing element used in the shared multi-threading like manner . This avoids that the ID information for all other processing ele- ments has to be changed as well . Where this seems to be too complicated in view of e . g . the necessity to provide plural

[0420] 59

[0421] SUBSTITUTE SHEET (RULE 26) bus snooping circuits , only a plurality of output registers could be provided .

[0422] In addition to this identi fying information, the actual status should of course be transmitted to the central machine control unit . This can again be related to the state speci fic for the respective input / output register set identi fied by one of the several IDs assigned to the more complex processing element .

[0423] Overall , this shows , that a situation might arise where it is not enough to j ust sent out the processing element ID without additional reference to the instruction and where e . g . several divider executions unit are provided with multithreading or preloading capabilities and thus have to transmit both their ID and an ID of an instruction for which the state has changed, both the instruction ID and the processing element ID should be transmitted by all processing elements i f an embodi- ment with a more homogeneous handling of status information is to be designed .

[0424] In addition to this identi fying information, the actual status should of course be transmitted to the central machine control unit . From the states listed above , the central machine con- trol unit cannot determine whether or not an processing ele- ment has completed its instruction or not , given that the cen- tral machine control unit does neither know whether a given processing element has already received all of its operands nor whether it has finished the execution . Therefore , the "completed" status information must be transmitted to the cen- tral machine control unit . In a similar manner, i f errors oc- cur such as an overflow or division by zero error, correspond- ing information must be sent from the processing element con- cerned to the central machine control unit .

[0425] Regarding the release state mentioned above , the information that a register is re-used, that is new content is to be writ- ten in a register by an upcoming instruction in the sequence of instructions , is available centrally and can be transmitted centrally to the central machine control unit . So , the release state can be considered as known in the machine control unit by providing adequate information from instruction fetch / de code / issue .

[0426] Also , it is to be understood that the central machine control unit will have the complete information about the completion state of ALL processing elements it manages . This helps in de- termining when a status of a processing element currently re- leased can be reset to FREE even i f the respective processing element does not report every transmission of operands to eve- ry or at least speci fic processing elements .

[0427] In more detail , as the instructions are issued in order, it can be determined by the central machine control unit whether

[0428] 60

[0429] SUBSTITUTE SHEET (RULE 26) all instructions preceding a release of a register ( that is , all instructions that might still need operands from a given processing element ) have completed; i f this is the case , then all of these processing elements will have received their op- erands and accordingly, no other processing element will still need the operands . So , a central machine control that receives "completed" information also knows whether all processing ele- ments to which instructions have been sent before the re-use of a register will have received their operands and that ac- cordingly, the corresponding processing element to which the re-used register was previously assigned can be re-used .

[0430] This , however, would free processing elements only strictly in order . This may not be economical as a number of processing elements that have received their instructions between assign- ing and re-assigning a given register will not execute in- structions relating to this register . Accordingly, their state is irrelevant for setting the status of a released processing element back to free . Therefore , a check could be made in the central machine control unit for j ust all of those processing elements that would need the results have completed and i f so , the released unit could be reset to free .

[0431] The later approach will speed up freeing processing elements as the number of instructions that must have finished will be smaller ; accordingly, the waiting time will be smaller on av- erage and more processing elements will be available .

[0432] As a downside , the management overhead increases somewhat i f track about the speci fic processing elements is kept that will need certain results . In particular, for obtaining an ad- vantage from keeping track, a list must be provided and gone thru, which costs both silicon area and energy . So , deciding what is more advantageous again needs to be done in view of a given use case .

[0433] It will be understood that where a TRAP state occurs , a rou- tine dealing with the trap state might necessitate additional di f ferent steps . As a TRAP error handling routine should be handled with priority, these speci fic steps should be executed with a high priority prior to issuing other additional in- structions .

[0434] As can be seen above , to allow a transition to FREE , it is enough i f the processing elements report their "complete" state .

[0435] By now, it should have been understood how operands are trans- ferred within an array, how instructions are issued and how the states of processing elements are used .

[0436] What has not yet been explained is how data is input / output and how a non-linear sequence having branches or j umps can be handled ef ficiently .

[0437] 61

[0438] SUBSTITUTE SHEET (RULE 26) Specif ic Problems of OUT OF ORDER OPERATION : BRANCH PREDICTION

[0439] Above , it has been assumed - for a single-thread-case of course- that the instructions fetched form a linear sequence of instructions and are issued as such linear sequence to the processing elements in the array . However, a case may occur where the further execution of the program depends on a result obtained and accordingly, the sequence BRANCHES ( or j umps ) in a manner not predictable at compile-time .

[0440] Now, it would of course be possible to wait until the result which is used to decide about the branch to be executed has been calculated- but in this case , the number of processing elements that would be executing instructions obviously de- creases before the branching decision can be made , reducing performance . To avoid this , where a branching is based on a simple decision between two branches , both branches could be executed in an out-of-order manner until the instruction by which the deci- sion is made is actually executed . Then, the results from the wrong branch could be discarded once the decision is made while the results from the correct branch could be maintained . However, this approach would not solve the problem completely as cases might occur where a branching decision is to be made between more than two possibilities or where several branching decisions are to be made in a short time , leading to a multi- tude of (nested) branches that might become relevant .

[0441] While a theoretical possibility might exist at compile time to determine that at a branching point only a limited number of options exist without additional branching points following soon in the sequence of instruction so that the few branches identi fied could indeed be executed in parallel i f needed, ra- ther than splitting the available number of processing ele- ments more or less evenly between several branches , it is fre- quently better to make assumptions at runtime about the upcom- ing decisions and to then execute that branch which is consid- ered the most likely . Then, i f the assumption is found to be correct , the results obtained in the meantime (by instruction processing elements FREE at branching point ) can be used- oth- erwise , the results can be discarded and the respective pro- cessing elements that have been put into use in this branch can be returned to FREE .

[0442] Now, it is well known in the art how a prediction what branch is to be selected can be made . Corresponding known branch pre- dictor circuitry to implement such known prediction methods could be easily implemented .

[0443] However, it can be anticipated that a mechanism how to deal with branch or j ump prediction misses , in particular when nested branching or multiple j umps occur, must also be provid-

[0444] 62

[0445] SUBSTITUTE SHEET (RULE 26) ed and it can be anticipated that providing such a mechanism encounters signi ficant problems , the solution of which is to be considered as inventive per se . Here , in particular, care must be taken that the information in the central machine con- trol unit remains useful .

[0446] In this context and in view of the above discussion of multi- threading, it is noted that executing a predicted branch will require a number of processing elements higher than the number of registers in a given instruction set architecture . Where in a multithreading environment the guaranteed number of regis- ters assignable to a given thread is limited, it may be advan- tageous to increase the maximum number of processing elements assignable for a given thread once it enters a predicted branch, thus resulting in a dynamical maximum number of regis- ters assignable to a given thread .

[0447] It is also noted that both allowing a maximum number of regis- ters assignable to a given thread to be a dynamic number and measures relating to the frequentness of completion and / or the duration until completion and / or similar measures are used to decide on issuing new instructions to a thread as set out herein, in particular above , is considered inventive each on its own and might be advantageous even in a more conven- tional processor design as known in the art and having e . g . reservation stations and so forth .

[0448] In particular, even in a conventional processor using multi- threading, the number of new instructions issued for a given thread might be determined in a manner dependent on the fre- quentness of completion and / or the duration until completion and / or the depth of a speculation or such that once a pre-set plurality of instructions for a thread have issued that all await execution, a new instruction will issue only once an in- struction has completed for this thread . Accordingly, it is suggested that deep speculations might be penali zed in in- struction issue for standard multithreading processors as well .

[0449] Note that this might require a corresponding feedback, such as a corresponding feedback line or status line for indicating a completion of an instruction issued for a given thread to e . g . an instruction issue stage .

[0450] Before dealing with prediction misses , a preferred branch prediction arrangement will now be described.

[0451] In a nutshell , for a processor according to some additional and / or favorable concepts and embodiments of the present in- vention, a branch-prediction at least largely analogous to ex- isting designs such as in particular Berkley' s RISC-V standard out-of-order front-end implementation can be used .

[0452] 63

[0453] SUBSTITUTE SHEET (RULE 26) In particular, as shown in Fig 1 , a front-end preferred for the present invention can be provided which fetches instruc- tions and makes predictions throughout the Fetch stage to re- direct the instruction stream . This front-end is referred to as HCRV front-end in the drawings , in particular in figure 1.

[0454] I f a misprediction is detected in the Back-end ( execution pipeline ) , or one of the predictors wants to redirect the pipeline in a di f ferent direction, a request is sent to the Front-end and it begins fetching along a new instruction path . The Front-end retrieves a Fetch Packet of instructions from instruction memory and puts them into the Fetch Buf fer to give to the rest of the pipeline . The Fetch Packet also contains other meta-data, such as a valid mask ("which instructions in the packet are valid?" ) and some branch prediction information that is used later in the pipeline . Additionally, the PC and branch prediction information is stored inside of the Fetch Target Queue which holds this information for the rest of the pipeline .

[0455] In more detail , at least the following components can be pro- vided

[0456] I-Cache

[0457] The I-Cache is a virtually indexed, physically tagged set- associative cache .

[0458] Fetch Buf fer

[0459] Fetch Packets coming from the I-Cache are placed into a Fetch Buf fer . The Fetch Buf fer helps to decouple the instruction fetch Front-end from the execution pipeline in the Back-end . The Fetch Buf fer is implemented as a " f low-through" queue .

[0460] Fetch Target Queue

[0461] The Fetch Target Queue is a queue that holds the PC received from the i-cache and the branch prediction info associated with that address . It holds this information for the pipeline to reference during the executions of its Micro-ops . It is dequeued by the ROB once an instruction is committed and is updated during pipeline redirection / misspeculation .

[0462] Branch Prediction

[0463] Reference is now had to the figure already referred to above showing a Front-end usable in the present invention . This Front-end usable in the present invention uses two levels of branch prediction - a fast "next-line predictor" (NLP ) and a slower but more complex "backing predictor" (BPD) .

[0464] In this case , the next-line predictor is a Branch Target Buf f- er and the backing predictor is a more complicated structure like a GShare predictor . A gshare share uses the global history hashed with the PC to index into a table of 2-bit counters . The

[0465] 64

[0466] SUBSTITUTE SHEET (RULE 26) high-order bit makes the prediction . An example is shown in fig- ure 2 .

[0467] Next-line Predictor (NLP )

[0468] The a front-end preferred for the present invention fetches instructions and predicts every cycle where to fetch the next instructions . I f a misprediction is detected in the Back-end, or the backing predictor wants to redirect the pipeline in a di f ferent direction, a request is sent to the Front-End which consequently begins fetching along a new instruction path .

[0469] The next-line predictor (NLP ) takes in the current PC being used to fetch instructions ( the Fetch PC ) and predicts combi- nationally where the next instructions should be fetched for the next cycle . I f predicted correctly, there are no pipeline bubbles .

[0470] The next-line predictor is an amalgamation of a fully- associative Branch Target Buf fer (BTB ) , Bi-Modal Table (BIM) and a Return Address Stack (RAS ) which work together to make a fast , but reasonably accurate prediction .

[0471] NLP Predictions

[0472] For the NLP Predictions , reference can be had to figure 3 .

[0473] The Fetch program counter PC first performs a tag match to find a uniquely matching BTB entry . I f a hit occurs , the BTB entry will make a prediction in concert with the RAS as to whether there is a branch, j ump, or return found in the Fetch Packet and which instruction in the Fetch Packet is to blame . The BIM is used to determine i f that prediction made was a branch taken or not taken . The BTB entry also contains a pre- dicted PC target , which is used as the Fetch PC on the next cycle .

[0474] Next-line Predictor (NLP ) Unit .

[0475] The Fetch PC scans the BTB' s "PC tags" ffoorr aa mmaattcchh .. I f a match is found ( and the entry is valid) , the BIM and RAS are con- suited for the final verdict . I f the entry is a "ret" ( return instruction) , then the target comes from the RAS . I f the entry is a unconditional "jmp" ( j ump instruction) , then the BIM is not consulted . The "bidx" , or branch index, marks which in- struction in a superscalar Fetch Packet is the cause of the control flow prediction . This is necessary to mask of f the other instructions in the Fetch Packet that come over the tak- en branch

[0476] The hysteresis bits in the BIM are only used on a BTB entry hit and i f the predicting instruction is a branch .

[0477] I f the BTB entry contains a return instruction, the RAS stack is used to provide the predicted return PC as the next Fetch PC . The actual RAS management ( of when to or the stack) is governed externally .

[0478] 65

[0479] SUBSTITUTE SHEET (RULE 26) NLP Updates

[0480] Each branch passed down the pipeline remembers not only its own PC, but also its Fetch PC ( the PC of the head instruction of its Fetch Packet ) .

[0481] BTB Updates

[0482] The BTB is updated only when the Front-end is redirected to take a branch or j ump by either the Branch Unit ( in the Exe- cute stage ) or the Backing Predictor ( later in the Fetch stag- es ) .

[0483] I f there is no BTB entry corresponding to the taken branch or j ump, an new entry is allocated for it .

[0484] RAS Updates

[0485] The RAS is updated during the Fetch stages once the instruc- tions in the Fetch Packet have been decoded . I f the taken in- struction is a call , the return address is pushed onto the RAS . I f the taken instruction is a return, then the RAS is popped .

[0486] Superscalar Predictions

[0487] When the NLP makes a prediction, it is actually using the BTB to tag match against the predicted branch' s Fetch PC, and not the PC of the branch itsel f . The NLP must predict across the entire Fetch Packet which of the many possible branches will be the dominating branch that redirects the PC . For this rea- son, we use a given branch' s Fetch PC rather than its own PC in the BTB tag match .

[0488] The Backing Predictor (BPD)

[0489] When the next-line predictor (NLP ) is predicting well , the processor' s Back-end is provided an unbroken stream of in- structions to execute . The NLP is able to provide fast , sin- gle-cycle predictions by being expensive ( in terms of both ar- ea and power ) , very small ( only a few dozen branches can be remembered) , and very simple ( the BIM hysteresis bits are not able to learn very complicated or long history patterns ) .

[0490] To capture more branches and more complicated branching behav- iors , the front-end preferred for the present invention pro- vides support for a "Backing Predictor" , or BPD .

[0491] The BPD' s goal is to provide very high accuracy in a (hopeful- ly) dense area . The BPD only makes taken / not-taken predic- tions ; it therefore relies on some other agent to provide in- formation on what instructions are branches and what their targets are . The BPD can either use the BTB for this infor- mation or it can wait and decode the instructions themselves once they have been fetched from the I-Cache . This saves on needing to store the PC tags and branch targets within the BPD .

[0492] The BPD is accessed throughout the Fetch stages and in paral- lel with the instruction cache access and BTB . This allows the BPD to be stored in sequential memory ( i . e . , SRAM instead of

[0493] 66

[0494] SUBSTITUTE SHEET (RULE 26) flip- flops ) . With some clever architecting, the BPD can be stored in single-ported SRAM to achieve the density desired . Reference should be had to Fig . 3 . 1 : The front-end preferred for the present invention .

[0495] Here you can see the BTB and Branch Predictor on the lower portion of the diagram . The instructions returning from the instruction cache are quickly decoded; any branches that are predicted as taken from the BTB or Backing Predictor will re- direct the front-end from the F4 stage . Prediction snapshots and metadata are stored in the branch rename snapshots ( for fixing the predictor after mispredictions ) and the Fetch Tar- get Queue ( for updating the predictors in the Commit stage ) .

[0496] Making Predictions

[0497] When making a prediction, the backing predictor must provide the following : is a prediction being made? a bit-vector of taken / not-taken predictions

[0498] As per the first bullet-point , the BPD may decide to not make a prediction . This may be because the predictor uses tags to inform whether its prediction is valid or there may be a structural hazard that prevented a prediction from being made . The BPD provides a bit-vector of taken / not-taken predictions , the si ze of the bit-vector matching the Fetch Width of the pipeline ( one bit for each instruction in the Fetch Packet ) . A later Fetch stage will will decode the instructions in the Fetch Packet , compute the branch targets , and decide in con- j unction with the BPD' s prediction bit-vector i f a Front-end redirect should be made .

[0499] Jump and Jump-Register Instructions

[0500] The BPD makes predictions only on the direction ( taken versus not-taken) of conditional branches . Non-conditional "j umps" ( JAL ) and "j ump-register" ( JALR) instructions are handled sep- arately from the BPD .

[0501] The NLP learns any "taken" instruction' s PC and target PC - thus , the NLP is able to predict j umps and j ump-register in- structions .

[0502] I f the NLP does not make a prediction on a JAL instruction, the pipeline will redirect the Front-end in F4 .

[0503] Jump-register instructions that were not predicted by the NLP will be sent down the pipeline with no prediction made . As JALR instructions require reading the register file to deduce the j ump target , there ' s nothing that can be done i f the NLP does not make a prediction .

[0504] Updating the Backing Predictor

[0505] Generally speaking, the BPD is updated during the Commit stage . This prevents the BPD from being polluted by wrong-path information . However, as the BPD makes use of global history, this history must be reset whenever the Front-end is redi- rected . Thus , the BPD must also be (partially) updated during

[0506] 67

[0507] SUBSTITUTE SHEET (RULE 26) Execute when a misprediction occurs to reset any speculative updates that had occurred during the Fetch stages .

[0508] When making a prediction, the BPD passes to the pipeline a "response info packet" . This " info packet" is stored in the Fetch Target Queue ( FTQ) until commit time . 11 Once all of the instructions corresponding to the " info packet" is committed, the " info packet" is set to the BPD ( along with the eventual outcome of the branches ) and the BPD is updated . The Fetch Target Queue ( FTQ) covers the FTQ, which handles the snapshot information needed for update the predictor during Commit . Re- name Snapshot State covers the BPD Rename Snapshots , which handles the snapshot information needed to update the predic- tor during a misspeculation in the Execute stage .

[0509] Managing the Global History Register

[0510] The Global History Register ( GHR) is an important piece of a branch predictor . It contains the outcomes of the previous N branches (where N is the si ze of the GHR) . When fetching branch i , it is important that the direction of the previous i-N branches is available so an accurate predic- tion can be made . Waiting till the Commit stage to update the GHR would be too late ( dozens of branches would be inflight and not reflected ! ) . Therefore , the GHR must be updated specu- latively, once the branch is fetched and predicted .

[0511] I f a misprediction occurs , the GHR must be reset and updated to reflect the actual history . This means that each branch (more accurately, each Fetch Packet ) must snapshot the GHR in case of a misprediction . 13 There is one final wrinkle - exceptional pipeline behavior . While each branch contains a snapshot of the global history register, any instruction can potential throw an exception that will cause a Front-end redirect . Such an event will cause the GHR to become corrupted . For exceptions , this may seem ac- ceptable - exceptions should be rare and the trap handlers will cause a pollution of the GHR anyways ( from the point of view of the user code ) . However, some exceptional events in- clude "pipeline replays" - events where an instruction causes a pipeline flush and the instruction is refetched and re- executed . 14 For this reason, a commit copy of the GHR is also maintained by the BPD and reset on any sort of pipeline flush event .

[0512] The Fetch Target Queue ( FTQ)

[0513] The Reorder Buf fer ( see The Reorder Buf fer (ROB ) and the Dis- patch Stage ) maintains a record of all inflight in- struc- tions . Likewise , the Fetch Target Queue ( FTQ) maintains a rec- ord of all inflight branch predictions and PC information .

[0514] These two structures are decoupled as FTQ entries are incredi- bly expensive and not all ROB entries will contain a branch instruction . As only roughly one in every six instructions is a branch, the FTQ can be made to have fewer entries than the ROB to leverage additional savings .

[0515] 68

[0516] SUBSTITUTE SHEET (RULE 26) Each FTQ entry corresponds to one Fetch cycle . For each pre- diction made , the branch predictor packs up data that it will need later to perform an update . For example , a branch predic- tor will want to remember what index a prediction came from so it can update the counters at that index later . This data is stored in the FTQ .

[0517] When the last instruction in a Fetch Packet is committed, the FTQ entry is deallocated and returned to the branch predictor . Using the data stored in the FTQ entry, the branch predictor can perform any desired updates to its prediction state . There are a number of reasons to update the branch predictor after Commit . It is crucial that the predictor only learns correct information . In a data cache , memory fetched from a wrong path execution may eventually become useful when later executions go to a di f ferent path . But for a branch predictor, wrong path updates encode information that is pure pollution - it takes up useful entries by storing information that is not useful and will never be useful . Even i f later iterations do take a di f ferent path, the history that got it there will be di f ferent . And finally, while caches are fully tagged, branch predictors use partial tags ( i f any) and thus suf fer from de- constructive aliasing .

[0518] Rename Snapshot State

[0519] The FTQ holds branch predictor data that will be needed to up- date the branch predictor during Commit ( for both correct and incorrect predictions ) . However, there is additional state needed for when the branch predictor makes an incorrect pre- diction and must be updated immediately . For example , i f a misprediction occurs , the speculatively- updated GHR must be reset to the correct value before the processor can begin fetching ( and predicting) again .

[0520] This state can be very expensive but it can be deallocated once the branch is resolved in the Execute stage . Therefore , the state is stored in parallel with the Rename Snapshots . During Decode and Rename , a branch tag is allocated to each branch and a snapshot of the rename tables are made to facili- tate single-cycle rollback i f a misprediction occurs . Like the branch tag and rename Map Table snapshots , the corresponding branch predictor "rename" snapshot can be deallocated once the branch is resolved by the Branch Unit in Execute . Reference should be had to figure 4 . This figure shows the Branch Pre- dictor Pipeline .

[0521] Although simple , the diagram of the Branch Predictor Pipeline helps to show the I / O within the Branch Prediction Pipeline . The Front-end sends the "next PC" ( shown as req) to the pipe- line in the FO stage . Within the Abstract Predictor, hashing is managed by the Abstract Predictor wrapper . The Abstract Predictor then returns a "BPD response" or in other words a prediction for each instruction in the Fetch Packet .

[0522] 69

[0523] SUBSTITUTE SHEET (RULE 26) The Abstract Branch Predictor Class

[0524] To facilitate exploring di f ferent global history-based BPD de- signs , an abstract "BrPredictor" class is provided . It pro- vides a standard interface into the BPD and the control logic for managing the global history register . This abstract class can be found in the above mentioned diagram labeled "Abstract Predictor" . For a more detailed view of the predictor with an example , reference should be had to figure 5. This shows the GShare Predictor Pipeline .

[0525] Global History

[0526] As discussed in Managing the Global History Register, global history is a vital piece of any branch predictor . As such, it is handled by the abstract BranchPredictor class . Any branch predictor extending the abstract BranchPredictor class gets access to global history without having to handle snapshot- ting, updating, and bypassing .

[0527] Operating System- aware Global Histories

[0528] Although the data on its benefits are preliminary, the front- end preferred for the present invention does support OS-aware global histories . The normal global history tracks all in- structions from all privilege levels . A second user-only glob- al history tracks only user-level instructions .

[0529] The Two-bit Counter Tables

[0530] The basic building block of most branch predictors is the "Two-bit Counter Table" ( 2BC ) . As a particular branch is re- peatedly taken, the counter saturates upwards to the max value 3 ( Obl l ) or strongly taken . Likewise , repeatedly not- taken branches saturate towards zero ( ObO O ) . The high-order bit speci fies the prediction and the low-order bit speci fies the hysteresis (how " strong" the prediction is ) .

[0531] These two-bit counters are aggregated into a table . Ideally, a good branch predictor knows which counter to index to make the best prediction . However, to fit these two-bit counters into dense SRAM, a change is made to the 2BC finite state machine - mispredictions made in the weakly not-taken state move the 2BC into the strongly taken state ( and vice versa for weakly taken being mispredicted) . The FSM behavior is shown in Fig . 3 . 6 . Although it ' s no longer strictly a "counter" , this change al- lows us to separate out the read and write requirements on the prediction and hystersis bits and place them in separate se- quential memory tables . IInn hhaarrddwwaarree ,, the 2BC table can be im- plemented as follows :

[0532] The P-bit :

[0533] Read - every cycle to make a prediction

[0534] Write - only when a misprediction occurred ( the value of the h-bit ) .

[0535] 70

[0536] SUBSTITUTE SHEET (RULE 26) Reference should be had to figure 6 .

[0537] A gshare predictor uses the global history hashed with the PC to index into a table of 2-bit counters . The high-order bit makes the prediction .

[0538] The H-bit :

[0539] Read - only when a misprediction occurred .

[0540] Write - when a branch is resolved (write the direction the branch took) Reference should be had to the figure 7 .

[0541] This figure shows a Two-bit Counter State Machine

[0542] By breaking the high-order p-bit and the low-order h-bit apart , we can place each in 1 read / 1 write SRAM . A few more assumptions can help us do even better . Mispredictions are ra- re and branch resolutions are not necessarily occurring on every cycle . Also , writes can be delayed or even dropped alto- gether . Therefore , the h-table can be implemented using a sin- gle Irw-ported SRAM by queueing writes up and draining them when a read is not being performed . Likewise , the p-table can be implemented in Irw-ported SRAM by banking it - buf fer writes and drain when there is not a read conflict .

[0543] A final note : SRAMs are not happy with a "tall and skinny" aspect ratio that the 2bc tables require . However, the solu- tion is simple - tall and skinny can be trivially trans formed into a rectangular memory structure . The high-order bits of the index can correspond to the SRAM row and the low-order bits can be used to mux out the speci fic bits from within the row .

[0544] The GShare Predictor

[0545] GShare is a simple but very ef fective branch predictor . Pre- dictions are made by hashing the instruction address and the global history ( typically a simple XOR) and then indexing into a table of two-bit counters . Fig . 3 . 5 shows the logical archi- tecture and Fig . 3 . 7 shows the physical implementation and structure of the Gshare predictor . Note that the prediction begins in the FO stage when the requesting address is sent to the predictor but that the prediction is made later in the F3 stage once the instructions have returned from the instruction cache and the prediction state has been read out of the Gshare ' s p-table .

[0546] Reference should be had to the following figure already shown above : Figure 5 .

[0547] The TAGE Predictor

[0548] Reference should be had to figure 9 .

[0549] The requesting address ( PC ) and the global history are fed in- to each table ' s index hash and tag hash . Each table provides

[0550] 71

[0551] SUBSTITUTE SHEET (RULE 26) its own prediction ( or no prediction) and the table with the longest history wins .

[0552] The front-end preferred for the present invention also imple- ments the TAGE conditional branch predictor . TAGE is a highly- parameteri zable , state-of-the-art global history predictor .

[0553] The design is able to maintain a high degree of accuracy while scaling from very small predictor si zes to very large predic- tor si zes . It is fast to learn short histories while also able to learn very, very long histories ( over a thousand branches of history) .

[0554] TAGE ( TAgged GEometric ) is implemented as a collection of pre- dictor tables . Each table entry contains a prediction counter, a usefulness counter, and a tag . The prediction counter pro- vides the prediction ( and maintains some hysteresis as to how strongly biased the prediction is towards taken or not-taken) . The usefulness counter tracks how useful the particular entry has been in the past for providing correct predictions . The tag allows the table to only make a prediction i f there is a tag match for the particular requesting instruction address and global history .

[0555] Each table has a di f ferent ( and geometrically increasing) amount of history associated with it . Each table ' s history is used to hash with the requesting instruction address to pro- duce an index hash and a tag hash . Each table will make its own prediction ( or no prediction, i f there is no tag match) . The table with the longest history making a prediction wins . On a misprediction, TAGE attempts to allocate a new entry , It will only overwrite an entry that is "not useful" (ubits = 0 ) .

[0556] TAGE Global History and the Circular Shi ft Registers ( CSRs ) Each TAGE table has associated with it its own global history ( and each table has geometrically more history than the last table ) . The histories contain many more bits of history that can be used to index a TAGE table ; therefore , the history must be " folded" to fit . A table with 1024 entries uses 10 bits to index the table . Therefore , i f the table uses 20 bits of glob- al history, the top 10 bits of history are XOR' ed against the bottom 10 bits of history .

[0557] Instead of attempting to dynamically fold a very long history register every cycle , the history can be stored in a circular shi ft register ( CSR) . The history is stored already folded and only the new history bit and the oldest history bit need to be provided to perform an update . Listing 3 . 1 shows an example of how a CSR works .

[0558] Reference should be had to figure 10 . This Figure shows the circular shi ft register .

[0559] 72

[0560] SUBSTITUTE SHEET (RULE 26) When a new branch outcome is added, the register is shi fted ( and wrapped around) . The new outcome is added and the oldest bit in the history is "evicted" .

[0561] Each table must maintain three GSRs . The first GSR is used for computing the index hash and has a si ze n=log (num_table_entries ) . As a GSR contains the folded histo- ry, any periodic history pattern matching the length of the GSR will XOR to all zeroes (potentially quite common) . For this reason, there are two GSRs for computing the tag hash, one of width n and the other of width n- 1 .

[0562] For every prediction, all three GSRs ( for every table ) must be snapshotted and reset i f a branch misprediction occurs . Anoth- er three commit copies of these GSRs must be maintained to handle pipeline flushes .

[0563] Usefulness counters (u-bits )

[0564] The "usefulness" of an entry is stored in the u-bit counters . Roughly speaking, i f an entry provides a correct prediction, the u-bit counter is incremented . I f an entry provides an in- correct prediction, the u-bit counter is decremented . When a misprediction occurs , TAGE attempts to allocate a new entry . To prevent overwriting a useful entry, it will only allocate an entry i f the existing entry has a usefulness of zero . How- ever, i f an entry allocation fails because all of the poten- tial entries are useful , then all of the potential entries are decremented to potentially make room for an allocation in the future .

[0565] To prevent TAGE from filling up with only useful but rarely- used entries , TAGE must provide a scheme for "degrading" the u-bits over time . A number of schemes are available , One op- tion is a timer that periodically degrades the u- bit coun- ters . Another option is to track the number of failed alloca- tions ( incrementing on a failed allocation and decremented on a success ful allocation) . Once the counter has saturated, all u-bits are degraded .

[0566] TAGE Snapshot State

[0567] For every prediction, all three GSRs ( for every table ) must be snapshotted and reset i f a branch misprediction occurs . TAGE must also remember the index of each table that was checked for a prediction ( so the correct entry for each table can be updated later ) . Finally, TAGE must remember the tag computed for each table - the tags will be needed later i f a new entry is to be allocated .

[0568] Other Predictors

[0569] The front-end preferred for the present invention provides a number of other predictors that may provide useful .

[0570] The Base Only Predictor

[0571] 73

[0572] SUBSTITUTE SHEET (RULE 26) The Base Only Predictor uses the BTBs BIM to make a prediction on whether the branch was taken or not .

[0573] The Null Predictor

[0574] The Null Predictor is used when no BPD predictor is desired .

[0575] It will always predict "not taken" .

[0576] The Random Predictor

[0577] The Random Predictor uses an LFSR to randomi ze both "was a prediction made?" and "which direction each branch in the Fetch Packet should take?" . This is very useful for both tor- turing-testing BOOM and for providing a worse-case performance baseline for comparing branch predictors .

[0578] BRANCH PREDICTION PREPARE FOR / PREDICTION MISS

[0579] This being said, it will now be explained how the speculative execution of a branch is prepared for and what action is to be taken in case of a prediction miss . Once this is understood, nested predictions and function calls / j umps where a return point needs to be defined can easily be understood as well .

[0580] Using branch predictions , it may eventually become necessary to return to the program step where the prediction was wrong . At this step, processing must be resumed, choosing the correct branch rather than the predicted wrong path . For this , a reentry point must be provided .

[0581] However, it is not suf ficient to then execute the correct steps - very obviously, it is also necessary to do so with the correct operands . Now, at a point a program branches , all op- erands that might be needed again are still in the output reg- isters of those processing elements that have generated them . As explained above , when a register is used while instructions of a normal (non-branching) sequence of program steps are exe- cuted, the corresponding processing element can be set in a release state and then be considered FREE .

[0582] During execution of a predicted branch, instructions might al- so be executed that would lead in a similar way to a re-use of registers and could result in changing the state of an pro- cessing element to FREE- this would be perfectly correct , i f f the branch predicted is correct .

[0583] However, where the predicted branch turns out to be the incor- rect one and another branch is to be executed instead, the content of the register might still be needed when executing the correct branch once the prediction has been found to be incorrect .

[0584] Accordingly, a problem would arise i f the operands in the pro- cessing elements would not be available any longer as return- ing to the branching point would not be possible .

[0585] 74

[0586] SUBSTITUTE SHEET (RULE 26) To solve this problem, theoretically, several possibilities exist .

[0587] The first solution would be that where a predicted branch is executed, the data in all registers are read out from the ar- ray of processing elements , so that in case the predicted branch was wrong, the read-out data could be re-written by some additional mechanism into the respective output registers again . This , however, would not only require the additional mechanism for writing back the read-out data into the respec- tive registers when returning to the branching point ; it would also require to read out all information from the registers - and even i f a way was found how to handle the signi ficant amount of data, this would take considerable time even before the predicted branch could be executed . However, waiting a considerable time thwarts the very reason why a branch is pre- dicted in the first place . Therefore , this possibility should remain a theoretical one .

[0588] Accordingly, a way better approach would be a second method wherein, when executing a predicted branch, units currently in use remain in use and within a predicted branch, instead of releasing processing elements unconditionally, the processing elements are released conditionally . Then, for actual release following a conditional release , the additional condition is set that the predicted branch is found to be the correct one - i f the branch executed is found to be the wrong branch, the conditional release is discarded . Therefore , when the branch was found to be the wrong one , a new state for the processing elements that have only been used for the execution of the mispredicted branch, has to be determined- this can be FREE i f the state the processing element had at branching point was FREE .

[0589] However, it is also possible that at branching point , the pro- cessing element was still waiting for its own operands and was able to complete its execution only after the branching . From the fact that for an instruction issued before the branching only operands available before the branching can be possibly used, it can be concluded that the result of the execution is independent of the fact that the branching was mispredicted . Accordingly, i f the status of an processing element which was in use even before branching is changed to "completed" during the execution of the mispredicted branch, this status may be maintained . Of course , where the status was "EXECUTING" before entering the wrong branch and this status has not been changed, this status should also be maintained . In contrast , i f the an processing element was FREE when entering a mispre- dicted branch, in case of a misprediction, it should be set back in a state where re-use is possible - to allow that , one possibility would be to have a "CONDITIONALLY EXECUTING" state

[0590] 75

[0591] SUBSTITUTE SHEET (RULE 26) and to reset them to FREE . In this manner, it can be distin- guished how the state needs to be changed .

[0592] The same basically holds where a status was "completed" even before the wrong branch was entered .

[0593] This status should be maintained in case of a misprediction because obviously, any RELEASE would only be based on new in- structions issued in the (mispredicted) branch and only i f a case

[0594] Then : I f a TRAP condition occurs in an processing element that has been in use prior to executing a predicted branch, the er- ror that lead to the trap condition has been caused by in- structions preceding the branching . Therefore , this must be addressed even i f the branching was predicted correctly, be- cause a reaction is needed in response to such errors .

[0595] However, where a TRAP condition occurs due to instructions is- sued after the branching point , it might well be that the branch taken has not been predicted correctly . Depending on the speci fic steps needed for error handling in the trap, it may be advisable to first determine whether the branching pre- diction was correct and only act in response to the TRAP state in case it was correct . This can and should basically be done as soon as it is obvious that the predicted branch is correct so that corresponding error handling routines can be launched .

[0596] Also , where a completion of an instruction relates to an in- struction issued only after the predicted branching, any pro- cessing element that has been put in use only after branching ( and thus in the due course of executing the branch) should be re-set to FREE in case of a misprediction, independent of the state it then has . By designating any conditions other than FREE "conditional" , it can be more easily distinguished how a state is to change in case of a misprediction . Accordingly, instead of a completed state , reference might be had to a con- ditionally completed state . An exception to this rule is the RELEASE state because i f an processing element is

[0597] In this manner, when returning to a point of mispredicted branching, states of execution should be changed on returning as follows :

[0598] FREE -> FREE

[0599] COMPLETED -> COMPLETED

[0600] EXECUTING -> EXECUTING

[0601] CONDITIONALLYEXECUTING -> FREE

[0602] CONDITIONALLYRELEASED -> NOT RELEASED

[0603] CONDITIONALLY TRAP -> FREE

[0604] 76

[0605] SUBSTITUTE SHEET (RULE 26) In contrast , when a branch has been predicted correctly, the states of processing elements should be changed as follows

[0606] FREE -> FREE

[0607] COMPLETED -> COMPLETED

[0608] EXECUTING -> EXECUTING

[0609] CONDITIONALLY EXECUTING -> EXECUTING

[0610] CONDI T I ONALLYRELEASED -> RELEASED

[0611] CONDITIONALLY TRAP -> TRAP

[0612] Note that for a case where an processing element was released even before entering the branching point but had not yet fin- ished its execution, a further change of status would only af- fect the transition from EXECUTING to COMPLETION and thus , the above list would be applicable . It would even be possible to FREE this processing element during execution by checking in the manner described above whether all instructions issued be- fore the branching and needing the corresponding result have also completed .

[0613] It has been stated above that basically, the management of processing element states can be separated from keeping track of register positioning and thus must not be a part of the register positioning table . This still holds even where branch predictions are allowed and accordingly, the changes suggested above can be ef fected outside the register positioning table .

[0614] Obviously, i f a predicted branching is found to have been pre- dicted correctly, the corresponding condition can be li fted; however, with nested branching, care must be taken to consider only the condition corresponding to the correct nesting level be ful filled . A case might occur with subsequently predicted branches where - while a prediction of a branch predicted ra- ther late was found to be correct - the initial prediction leading to it was incorrect ; therefore , in nested branching, conditional releases can only be made actual releases after all preceding predictions where found to be correct .

[0615] In contrast , where a prediction is found to be wrong and later predictions are still executed, it is obvious that the later prediction are based on wrong assumptions and execution of the corresponding instructions should be ended . Several mechanisms could be used for this , such as setting all processing ele- ments executing corresponding instructions to released and completed and / or by sending a suitable signal to the respec- tive processing elements .

[0616] The disadvantage of an approach where a large number of copies of a register positioning table is held is that even for a correctly predicted branch, the processing elements that hold result data in their output register at the time of branching

[0617] 77

[0618] SUBSTITUTE SHEET (RULE 26) cannot be re-used until the predicted branch is found to be the correct one .

[0619] Also , for safely returning to one of a number of branching points from nested branches , it must be made sure that then, the status of processing elements can be changed as necessary to correct for any misprediction occurring in the process .

[0620] It will be understood that this necessitates to maintain suit- ably organi zed copies of the register positioning table and the status information file and to change the status as needed when it is determined whether a branch prediction is correct or wrong . Also , the program counter at the point of branch prediction must be preserved as in case of a prediction miss , it will be necessary to execute the correct branch and thus re-iterate

[0621] A more detailed explanation of the hardware needed to allow branch prediction and methods to change the status in case the correctness of the prediction has been determined will be giv- en hereinafter . However, it will be obvious that a processor having a number of processing elements signi ficantly higher than the number of registers used by the instruction set im- plemented by the processor and a register positioning table will benefit i f multiple copies of the register positioning are provided together with additional circuitry for predictive branch execution and state adaption circuitry for adapting processing element states in response to a determination of a prediction correctness .

[0622] Note that any information relating to the status and use of processing elements should be held within the processor to avoid signi ficant latencies by exporting such files , the num- ber of branches / nesting levels must be restricted- however, it should be understood that keeping status information and register positioning table at the same place is not necessary .

[0623] Now, reaching the maximum branching nesting will basically not cause a problem as this simple prevents further speculative execution and execution of subsequent instructions will con- tinue ; also , soon as one or more register positioning table copies currently used are freed after determination whether a branch prediction is correct , further speculative execution of branches may continue . The same holds where no more additional free processing elements are available- this does not consti- tute a problem per se as execution will continue with those processing elements already in use , with processing elements becoming available again during the operation of the device . Note that for executing predicted branches , the number of pro- cessing elements can be at least as high as the number of the registers used in the instruction set times the number of al- lowed predicted branches . In this way, in each predicted

[0624] 78

[0625] SUBSTITUTE SHEET (RULE 26) branch all register could be put in new use , allowing to con- ditionally released all registers during execution of a specu- lative branch .

[0626] I f multiple nesting levels are used, the same basic principles explained above will apply . For each of the multiple predicted branch points , a copy of the register positioning table is provided and once a decision can be made as to the correctness of a prediction, the copy is either discarded ( together with any subsequent predicted branchings which will also be incor- rect ) or maintained ( together with maintaining depending gen- erations at least temporarily) . Di f ferent actions may result from determining that a branch has been predicted correctly . I f a branching was found to be correct that was the first branching currently considered, then the same rules apply to a change to the respective states as mentioned above , that is

[0627] FREE -> FREE

[0628] COMPLETED -> COMPLETED

[0629] EXECUTING -> EXECUTING

[0630] CONDITIONALLY EXECUTING -> EXECUTING

[0631] CONDI T I ONALLYRELEASED -> RELEASED

[0632] CONDITIONALLY TRAP -> TRAP

[0633] Also , the corresponding copy of the register positioning table is now a non-speculative register positioning table and thus , it will be the new "actual" copy- this is termed in some in- stances the "Most recent Non Speculative Register Positioning Table or MRNSRPT .

[0634] However, i f a branching found to be correct is not the oldest branching, a determination that this subsequent branching is correct does neither imply that a preceding branching was pre- dicted correctly nor that any subsequent branchings are cor- rect . So , any correction of the state must remain conditional , even though one condition can be li fted .

[0635] Accordingly, the change of status must be handled somewhat di f ferently .

[0636] It can be understood that a FREE processing element remains FREE .

[0637] Furthermore , i f an processing element is currently uncondi- tionally executing it should remain executing . Note that as per the tables given above , "Executing" refers to only Execu- tion units that had been executing when the branch was en- tered .

[0638] However, i f an processing element is "conditionally execut- ing" , a distinction must be made whether it is "conditionally executing" because it was instructed within the nested branch currently determined to be correct - in which case the "condi- tionally executing" should be changed to "executing" - or whether it is "conditionally executing" because some branching predicted later on issued an instruction to that processing element- in which case it should remain "conditionally execut-

[0639] 79

[0640] SUBSTITUTE SHEET (RULE 26) ing" until all branchings up to the respective branching cur- rently determined to be correct have also been found to be correct .

[0641] For a released state , several di f ferent possibilities must be considered . Again, a release could have occurred because it occurred within the branch currently determined to be correct- then, a release should be ef fected as soon as all preceding branchings have also been determined to be correct but should be discarded i f any of the preceding branches is determined to be incorrect .

[0642] Similar considerations can be made for a TRAP state .

[0643] One method to handle this would be to add to the state aann in- dication at what level of predicted branching it was changed .

[0644] For example , tthhee nn--tthh ppoossiittiioonn iinn aa bbiinnaarryy sseeqquueennccee could in- dicate whether a ssppeeccii ffiicc ssttaattee is conditional for the n-th branching, e . g . by an entry of " 1" at the respective position indicating that the entry is not conditional for the branching relating to that position or " 0" to indicate it is . Then, a current state in a "Most recent Non Speculative Register Posi- tioning Table" can be changed i f all lower positions (with a smaller n) are " 1" that is non-conditional any longer , It will be understood that for this , the file of states for all pro- cessing elements simple must have a suf ficient bit length .

[0645] It should be understood that in certain conditions other than branching, providing copies of the "Most recent Non Specula- tive Register Positioning Table" might also be useful . For ex- ample , during operation of the device interrupts unrelated to a program or thread currently executed might have to be han- dled with a high priority; in this case , once the interrupt has been handled, in most cases execution of the program exe- cuted before the interrupt is to continue - for this , the states can be preserved in the same manner as in case of a branch prediction .

[0646] MORE DETAILED EXPLANATION OF BRANCH PREDICTION

[0647] This being said, a more detailed explanation of handling branch prediction and the circuitries and hardware used for this in a preferred embodiment currently designed will now be described in some more detail so as to disclose , by way of ex- ample , a particularly preferred embodiment and features usable together or separately in a preferred implementation .

[0648] A data processing device using an array of processing elements and allowing branch prediction might basically have an ar- rangement as schematically shown in figure 11.

[0649] The typical use of a register positioning table is shown in figure 12 .

[0650] 80

[0651] SUBSTITUTE SHEET (RULE 26) Instructions are fetched from an address defined by the Branch Predictor in the instruction memory space ( e . g . via L2 ) . The instructions are pre-decoded and available processing elements are selected by "PE allocation" which sets the processing ele- ment internal state machines ( T-tables ) accordingly . The the instructions ' target registers in the Register Positioning Ta- ble (RPT ) are updated with the allocated processing elements location . The source registers are renamed with the register positioning table entries . Finally the instructions are is- sued . Processing elements storing required operand data retriggered to resend the respective data on the bus .

[0652] The Register Positioning Table (RPT ) stores the ID for each current logic target register . The ID points to the X / Y posi- tion of the processing element holding the respective data re- sult . For each instruction issue the register positioning ta- ble stores the ID for the instruction' s target register . For each source register the ID of the processing element produc- ing the operand data is provided by the register positioning table . During execution processing elements monitor the bus for data trans fers with the respective source ID token .

[0653] Now, in the preferred embodiment currently designed, the reg- ister positioning table will hold copies for :

[0654] A. Branches - rollback in case of miss-speculation .

[0655] B . LD / ST - In case of page miss , TRAP to OS .

[0656] C . DIV - In case of division by zero , a TRAP interrupt can be sent to the operating system .

[0657] D . In case of TRAP instruction E . Most Recent Non-Speculative register positioning table (MRNS REGISTER POS ITIONING TABLE ) .

[0658] F . Live register positioning table - the current working register positioning table that changes with I ssues .

[0659] As task switches and / or interrupts will inevitably occur dur- ing execution of a program, providing arbitrary register posi- tioning table copies is useful so that the loss of code can be kept low .

[0660] For example , a copy of the register positioning table could be generated every 16 instructions unless a copy has already been generated in the preceding 16 instructions in view of one of the reasons indicated above . Note that the repetition rate for the precautionary generation of register positioning table copies might vary for multi-issue and / or multi-threading ap- plications , depending in particular on the number of instruc- tions concurrently issuable .

[0661] The register positioning table copies could be used as follows

[0662] 81

[0663] SUBSTITUTE SHEET (RULE 26) A. When a speculation is right , the related register posi- tioning table entry is cleared .

[0664] B . When a speculation is wrong, the respective register posi- tioning table entry is restored, all younger ( subsequent ) register positioning table entries are cleared .

[0665] C . When a LD / ST / DIV / etc does not trigger Exception - ( for example hit in TLB, memory) , the related register posi- tioning table entry is cleared .

[0666] D . When a LD / ST / DIV / etc does trigger an exception, the re- spective register positioning table entry is restored, all younger ( subsequent ) register positioning table entries are cleared .

[0667] E . MRNS - when a younger register positioning table becomes non-speculative .

[0668] Each register positioning table has :

[0669] 1 . List of pointers for the arch regs .

[0670] 2 . Order - important for clearing younger on rollback, selecting the next MRNS .

[0671] 3 . Valid

[0672] The copies of the register positioning table allow implementa- tion of a register positioning table recovery mechanism, that is a method to recover the register positioning table at a previous point in time . It will be understood by a person skilled in the art that there are some cases in which the pro- cessor has to revert to an older point . The most obvious case is a branch miss-predict . Misspredicting a branch means that instructions have been issued that should not have have run and consequently, the processor needs to revert . Other such cases are TRAP interrupts for whatever reason .

[0673] The basic premise of the recovery mechanism is to identi fy points execution might have to return to , and then save those points . When the processor has to reset to an older state , i . e . has to return the saved point is restored .

[0674] For this , a saved point needs to be created . The save point can be created by :

[0675] 1 . Saving the Live register positioning table , i . e . the current register positioning table to the register po- sitioning table buf fer . The register positioning ta- ble is a list of pointers to the processing elements where the registers are located . By saving those pointers and restoring them, tracks which registers were stored in which processing element .

[0676] 2 . Not re-allocating used registers in the saved register positioning tables . Since it might be necessary to re- store those old registers , the registers can' t be re- used until it is ensured the old point will no longer be needed .

[0677] 82

[0678] SUBSTITUTE SHEET (RULE 26) 3 . Saving the Program Counter ( PC ) . The correct program counter at the save point must be stored so that in case of a revert processing correctly continues .

[0679] Then, the processing element state and content on restore must also be managed .

[0680] Managing PE state and content on restore :

[0681] Processing elements that have been working on instructions that occur in program order before a Restore Point , that is before a saved point which has to be restored, have issued be- fore the restore point in the in-order-issue-out-of-order- execute processor of the current invention and have to keep their data and state .

[0682] Processing elements working on instructions in program order after the Restore Point have to be invalidated .

[0683] Each processing element and each load-store-queue slot has its register positioning table entry .

[0684] On Restore , the register positioning table entry of the re- stored point ( Its location in the Queue ) and the register po- sitioning table entry of the processing element ( or ESQ Slot ) is checked, and i f the register positioning table entry is younger than the Restore point , it is invalidated .

[0685] The register positioning table entry is set to the processing element on issuing of instruction .

[0686] So , the following is saved :

[0687] 4 . register positioning table entry

[0688] While allocating processing elements only available processing elements which are free to allocate can be used . It is not possible to allocate any processing element that is still in use by any saved register positioning tables . ( See point 2 above ) .

[0689] This necessitates to save the "Targeted"-bit per saved regis- ter positioning table as well . These can be calculated from the entries in the register positioning table , but that is time consuming .

[0690] So , to conclude , one could save :

[0691] 1 . The register positioning table, the data in the pro- cessing elements .

[0692] 2 . The targeted bit (per processing element , per register positioning table entry)

[0693] 3 . The register positioning table entry (per processing element )

[0694] Accordingly, attention should be had to figure 13 and the flow diagram as shown in figure 14 .

[0695] 83

[0696] SUBSTITUTE SHEET (RULE 26) Operation should ensure :

[0697] 1 . New processing elements must be allocated such that they will not override old processing elements that still hold data from previous register positioning tables . Therefore , the T-bit table should be checked and i ) processing elements that still hold data from pre- vious Valid Restore Points should not be allocated, ii ) that will broadcast (have data to broadcast or will in the future ) , and iii ) that are currently targeted .

[0698] This will ensure the old data will not be over-written .

[0699] 2 . Once a register positioning table is no longer valid, i . e . it will definitively not be restored, it is marked in the register positioning table table and the old T-bits are cleared for that register positioning table This will allow Processing elements that held old data to be free for allocation once again .

[0700] Register positioning table saving structure - register posi- tioning table buf fer

[0701] The active register positioning table must be saved ( duplicat- ed) whenever a new save point is reached . This place is called the register positioning table Buf fer . It currently is de- signed as a cyclic buf fer with 2 pointers :

[0702] 1 . Head - pointing to the next free entry .

[0703] 2 . Tail - points to the last known non-speculative reg- ister positioning table . This is the oldest point which might be restored . All points after it are considered com- mitted . This is also ccaalllleedd tthhee MMRRNNSS - Most Recent Non- Speculative .

[0704] Most Recent Non-Speculative

[0705] A newly saved register positioning table is entered at the head, and the head pointer advances . When register positioning tables resolve and become non-specualtive , the tail , that is the most recent non-speculative (MRNS ) advances . On advancing the most recent non-speculative the register positioning table entries per -processing element must be updated to make sure the most recent non-speculative does not overtake them . Pro- cessing elements that have an entry that the most recent non- speculative bypass , will be marked as non-speculative , and will not need to advance again .

[0706] Restoring a point - Rollback

[0707] Restoring a register positioning table requires the following steps :

[0708] LoADing the register positioning table to be restored from the register positioning table buf fer . Exchange it for the Live register positioning table

[0709] 84

[0710] SUBSTITUTE SHEET (RULE 26) Setting the program counter with the program counter stored in the updated Live register positioning table to set the next instruction to fetch .

[0711] LoADing the T-bits for the updated Live register position- ing table entry to the live bits .

[0712] Invalidating Processing elements , BBuuss ,, load-store-queues and other special units that have register positioning ta- ble entries younger or equal to the restored register po- sitioning table entry .

[0713] Moving the Head of the register positioning table buf fer back to one position after the register positioning table entry restored .

[0714] Marking the T-bits for invalidated register positioning table entries as free .

[0715] All the steps must take place in the same clock cycle .

[0716] Register positioning table entries in the load-store-queue load-store-queues entries might be speculative , so they need register positioning table entries as well . Consequently those load-store-queue Entries have to be invalidated when restoring a register positioning table .

[0717] Register positioning table - Tables implementation

[0718] Each register positioning table needs to have to pointers to the arch registers , i . e . which processing element holds ( or will hold) the value of the respective arch register . It also needs to be marked as evaluated or not .

[0719] Order between register positioning table s must be conserved, because order is important for which register positioning ta- bles get invalidated on rollback .

[0720] A particularly preferred solution is to maintain order between register positioning tables in a FI FO . This is a trade-of f : Maintaining order becomes easy and implic- it , but the space of unnecessary register positioning tables ( for example loads that finished, branches that were predicted correctly) cannot be reused until all previous register positioning tables were cleared and the pointers advanced .

[0721] FI FO implementation :

[0722] SRAM width :

[0723] #arch_regs* logs ( #Processing elements ) + flags ( evaluated, val- id) = 31 * log2 ( 256 ) + 2 = 250 ~ 256 .

[0724] Each SRAM line stores a register positioning table copy . The SRAM should be as deep as the number of register positioning table copies one is going to have . Increasing this number

[0725] 85

[0726] SUBSTITUTE SHEET (RULE 26) would allow the machine to run deeper, through more LD / ST / branches , but will cost area.

[0727] This number also appears in the processing element state ta- ble .

[0728] The SRAM implements a FIFO, with 2 pointers:

[0729] FIFO Head - The next available slot. New register positioning table copies will be written here. If no new slots are availa- ble, the machine should stall and stop issuing new commands until a new slot becomes available.

[0730] FIFO Tail

[0731] The Most Recent Non Speculative (MRNS) register positioning table . This is the oldest register positioning table that is still valid, so it might be necessary to rollback to it) .

[0732] Register positioning table

[0733] Issue Unit

[0734] The issue unit has to know wether a processing element is Available for Allocation (AfA) .

[0735] A processing element is Available for Allocation if no valid register positioning table copy uses it, and it is in a Free state. Accordingly

[0736] Afa = NOT (OR (register positioning table usage in table + live processing element state) .

[0737] To do this directly, the machine has to be cycle accurate, since if there is a discrepancy between the table result and the actual status, the Issue unit could re-allocate a busy processing element . For example, if the Status take 2 cycles to update, the issue unit could issue 2 instructions to the same processing element, not knowing it was busy, since the table didn't update.

[0738] Another option, is to have a separate data structure in the Issue unit, maintaining if the Issue unit can allocate to this processing element.

[0739] It set a bit per processing element to zero if it issued an instruction to that processing element.

[0740] Only when the processing element sends a message (1 bit pulse, or on rise) that it is free, does the Issue unit clears the bit. This costs an extra data structure and some logic, but gets rid of the timing dependency.

[0741] Interconnections for loop acceleration mode and out of order mode

[0742] Above, emphasis had been made on the capability of the device according to the invention to process instructions of a se- quence of instructions in a particularly effective manner out of order, while also allowing for rather simple ways of exe-

[0743] 86

[0744] SUBSTITUTE SHEET (RULE 26) cuting predicted branches , executing nested predicted branch- es , executing instructions in simultaneous multithreading and so forth . Some of the structures for this have been described already .

[0745] While certain types of algorithms and program structures prof- it signi ficantly from the out-of-order execution these struc- tures allow and / or from the methods described herein, other algorithms and program structures cannot prof e time . In other words , path balancing can be implemented, further simpli fying the trans fer .

[0746] By such handling of datastreams and loops , it is neither nec- essary to repeatedly issue the same instructions nor is it necessary for a processing element to snoop the BUS for its operands . Therefore , operation will be particularly energy ef- ficient .

[0747] Now, while the ratio of time needed to set up an array for loop operation over the time needed for repeated execution of the loop becomes negligible for large numbers of repetition cycles such as when watching a video stream, there are other examples where only a few loop passes are required by an algo- rithm, such as when calculating the first few terms of a tay- lor series . Then, the time to set up the array becomes rele- vant .

[0748] It is therefore desirable to allow ef ficient handling of loops - both for setting up a loop and for returning from a loop- as well as during execution of a loop .

[0749] Furthermore , it would be desirable to provide a processor chip or processor chip arrangement such as a system on a chip with suitable means to allow easy implementation of a loop acceler- ation mode .

[0750] Furthermore , it would be desirable to provide structures for loop acceleration mode which can make at least partial use of those structures of a processor provided for ef ficient out-of- order instructions .

[0751] Furthermore , it would be desirable to make use or partial use of any structures provided for implementation of an ef ficient loop acceleration mode during out-of-order instructions .

[0752] It would be helpful to allow reaching at least some of these obj ects to at least a partial degree .

[0753] It will be understood hereinafter that some of the suggestions made can be implemented in conventional processors , e . g . a processor not having one or more register positioning table . The implementation could be made without or with only little hardware modi fications and it is noted that certain data flow

[0754] 87

[0755] SUBSTITUTE SHEET (RULE 26) type data processing arrangements such as coarse grained FPGAs will also profit from ideas and concepts disclosed and sug- gested herein . Applicant reserves the right to file corre- sponding divisional applications and the like directed to such ideas and concepts .

[0756] Regarding loop acceleration mode , it is noted that some basic concepts , in particular, but not exclusively relating to com- pilers , are known from the prior art stemming e . g . from pre- sent inventor Vorbach alone or together with others , in par- ticular the documents referred to above . The disclosure of the documents mentioned above is thus enclosed herewith in its en- tirety by reference .

[0757] To this end, loops need to be detected in the program code . This could be done at runtime by analyzing the instructions prior to issuing; however, to make optimum use of loop accel- eration mode , the entire loop body should be considered so that analyzing complex loop bodies necessitates to look at a very large number of instructions . TThheerreeffoorree ,, it is preferred to detect loops at compile time and to prepare all instruc- tions as necessary by the compiler . Then, the compiler identi- fies a loop, analyzes the loop body and rolls it out into a graph; a node in this graph corresponds to a mathematical- logical operation and the edges or interconnections in this graph determine where the results obtained in a node need to be sent to . Accordingly, a processing element must be provided for each node to execute the mathematical-logical operation associated therewith, while interconnections between the pro- cessing elements must be provided to reflect the edge struc- ture . It will be understood that in such a graph, the edges represent oriented connections in that result data from one node will need to be trans ferred to a subsequent node in a di- rection defined by the sequence of instructions .

[0758] Now, in the graph, most of these interconnections will trans- fer data in a predominant "data flow" direction with typically only very few edges going against the predominant direction ( such as when a new pass of the loop is needed) . This predomi- nant data flow direction has consequences with respect to how an arrangement of interconnections provided for loop accelera- tion mode should be designed . Considering for simplicity and better understanding an array of rectangular processing ele- ments arranged in a two dimensional manner having rows and columns , it will be suf ficient for most trans fers to output the result from one row to the subsequent row downwards there- of ; only few trans fers will be needed in the counterdirection . Also , typical loop algorithms will require trans feral of out- put da ta to only some processing elements rather than a great many number of receiving processing elements .

[0759] To allow better understanding of how with this , a loop accel- eration mode can make good use of a BUS system otherwise used

[0760] 88

[0761] SUBSTITUTE SHEET (RULE 26) for out-of-order mode operation and / or how out-of-order mode operation can make use of connection structures provided for out-of-order mode operation, first , the BUS system used for out-of-order mode will be described; then, the additional structures implemented for ef ficient and correct loop acceler- ation will be described .

[0762] Thereafter, it is suggested how use can be made of a combina- tion of both types of structures in out-of-order mode and loop acceleration mode respectively .

[0763] From this , it will be understood that and how inter alia the BUS system used for out-of-order mode should be optimi zed or operated, so that in a preferred embodiment , the BUS system allows both ef ficient out of order execution of instructions as well as ef ficient implementation of the loop acceleration mode .

[0764] This can be achieved in a preferred embodiment of the inven- tion by implementing an interconnection system with a proper set of BUS lines and preferably, some additional direct con- nections from point to point . For the sake of clarity : the term "BUS lines" does not refer to multiple parallel wires to reflect the bit width used in a given BUS, that is the number of bits trans ferable in parallel , but rather refers to the number of interconnections along which data with a given bit si ze such as 32-bit data or 64-bit data ( + overhead infor- mation as necessary) can be simultaneously transmitted) . It will be understood by a person skilled in the art that on the one hand, the number and arrangement of BUS lines must be suf- ficient to implement the most common algorithms in an array, a part of an array or a combination of plural arrays while on the other hand, the number of BUS lines should not become too large , it will be understood that while using a bus system with a single bus line in each row and a single bus line run- ning vertically might suf fice for a chip, low-performance pro- cessor, providing multiple bus lines is highly preferred . As is the case with other design decisions , the decision on the number of bus lines and can for example be made in view of algorithms . When designing a processor architecture , the algo- rithms that should be considered in this context will be the algorithms currently most relevant and an attempt can be made to take into account the most likely architectural needs in view of upcoming or expected future algorithms ; therefore , the communication lines should be both suf ficient and usable in a suf ficiently flexible manner ; however, it should be avoided to provide a BUS system and / or communication line system too com- plex .

[0765] At the same time , care should be taken to allow scaling of an architecture , for example so that data processor chips can be

[0766] 89

[0767] SUBSTITUTE SHEET (RULE 26) designed having vastly di f ferent numbers of arrays of pro- cessing elements to attend to di f ferent performance needs . In view of this , a BUS system - or rather : an interconnection system- useful for a plurality of di f ferent algorithms and suf ficient for the needs of both an ef fective out-of-order- processing as well as an ef ficient loop acceleration mode exe- cution will be disclosed for a preferred embodiment of the present invention .

[0768] BUS system for out-of-order mode operation

[0769] Above , a number of remarks have already been made with respect to the buses .

[0770] In particular, it has been stated that in the out-of-order mode , results can be broadcasted on a bus and that the pro- cessing elements will have the capability ( and thus the cir- cuitry) to snoop the bus for the operands they will need to execute their instructions . Broadcasting is considered simpler than routing the results along predefined interconnections , in particular i f the interconnections need to be defined at runtime in view of the random placement of instructions within the array during out of order mode execution .

[0771] It has also been suggested that for broadcasting, several strategies exist but that in a preferred embodiment , broad- casting or re-broadcasting of previously generated data by processing elements can be triggered and that to this end, a separate trigger network can be provided for directly address- ing the respective processing elements that have to broadcast their data .

[0772] Note that with respect to broadcasting strategies , several possibilities exist not j ust with respect to the timing but also with the reach of broadcasting or the path used to ef fect broadcasting .

[0773] Broadcasting is considered simpler than routing the results along predefined interconnections , in particular i f the inter- connections need to be defined at runtime in view of the ran- dom placement of instructions within the array during out of order mode execution .

[0774] In most parts of the document , an output data transmission broadcasted from one processing element is described to reach each and every other processing element ; this allows a partic- ularly simple broadcasting process in that no care needs to be taken where the output data must be transmitted; also , it is particularly simple to handle a second request to broadcast the same data while the processing element still waits for ac- cess to the bus system for handling the first request .

[0775] 90

[0776] SUBSTITUTE SHEET (RULE 26) On the other hand, the bus system must be able to handle data traf fic from a large number of processing elements simultane- ously having to transmit their output data . To overcome per- formance bottlenecks due to insuf ficient data transmission ca- pabilities of the bus system, the frequency (bandwidth) of the bus system can be increased and / or a plurality of independent- ly operable bus lines can be provided . Increasing the band- width and thus the frequency of the bus system will increase power consumption . Providing a plurality of independently op- erable bus lines will necessitate that the processing elements listen to all bus lines simultaneously for their operands to arrive and must connect them to their respective input operand registers as needed . This requires additional hardware ex- pense , silicon area and energy .

[0777] Bus performance becomes more critical the larger the array is ; however, larger arrays allow for more ef ficient processing out of order mode , in particular, where a large number of pro- cessing elements can be shared among multiple threads executed simultaneously .

[0778] A way to improve bus performance in view of this is to re- strict broadcasting to certain regions . For example , a broad- cast could be restricted to only processing elements within the same rrooww,, to processing elements in the rows above or to processing elements in the rows below a transmitting pro- cessing element , to a row next but one below or next but one above . Furthermore , transmissions could easily be ef fected to only all rows above the own row or to only all rows below the own row . (Note that alternatively, restricted broadcasting to certain columns or groups of columns would also be possible ) . Accordingly, broadcasting can be restricted to certain re- gions . This may greatly reduce the overall data bus traf fic . Also , this can easily be combined with a more advanced design of a bus system having multiple lines ; where desired, it is would even be possible to use bus lines having an increased bus frequency .

[0779] Broadcasting to only certain ( target ) regions is signi ficantly less complex than routing to a speci fic processing element , in particular where such routing would have to be set up before a transmission . While some additional information is needed when triggering a transmission, for example relating to the target region of the target processing element , and while some infor- mation relating to a target region must be provided to those parts of the transmission path involved, the overall hardware expense is low . Therefore , broadcasting to certain target re- gions is highly preferred , the idea and in particular certain advantageous aspects as disclosed herein are therefore consid- ered inventive and applicant reserves the right to draft claims relating thereto , in particular in divisional applica- tions . Nonetheless , while a clear disclosure of targeting

[0780] 91

[0781] SUBSTITUTE SHEET (RULE 26) broadcasting is given in this document , in a number of parts references simply had to "broadcasting" rather than "targeting broadcasting" , "regional broadcasting" or the like so as not to overburden other parts of the description, for example those parts where it is suf ficient to ensure that processing elements will receive their input operands from other pro- cessing elements independent of the particular manner of transmission .

[0782] It will be understood that targeted broadcasting may result in di f ferent bus protocols , di f ferent methods of triggering transmission and so forth . Such changes will become obvious hereinafter .

[0783] Above , it has already been stated that in a preferred embodi- ment , the instruction issue stage can both issue the instruc- tions and trigger the broadcasts - or that at least , the trig- ger to broadcast results can be issued simultaneously with the instructions .

[0784] It has also been stated that instructions can be sent via a separate instruction bus or that direct connections from an instruction issue stage to the respective processing elements might be provided . It has been stated that using direct con- nections for triggering and / or transmitting instructions di- rectly to a processing element can render snooping the bus for instructions unnecessary, thus saving on both the respective circuitry and the energy .

[0785] It has been discussed that even upon multiple requests to broadcast results , broadcasting might be ef fected only once ( obviously : unless requests arrive after broadcast has been ef fected) .

[0786] It should be understood that where broadcasting is targeting a speci fic region, a di f ferent approach might be taken with re- spect to multiple requests . A distinction could be made wheth- er or not the request comes from the same region; where it comes from the same region, handling could be similar to a situation where broadcasting is not targeting a speci fic re- gion but is directed to all processing . Where the request comes from a region not targeted or reached by a present broadcasting process in progress , it could be forced to wait until the current broadcasting of the data has finished so as to avoid clogging of the bus system by single processing ele- ments ; it would be possible to restrict the number of broad- casting transmissions a single processing element may concur- rently have to a limited number ; it would be possible to amend the target direction " in- flight" , for example while output da- ta has already been transmitted from the processing element to a buf fer in a hori zontal bus line where the data is to wait for access to a vertical bus line ; amending the target region

[0787] 92

[0788] SUBSTITUTE SHEET (RULE 26) upon receipt of a further request will also be possible by suitable circuitry with in each processing element , albeit do- ing so in front of a vertical bus line might be more ef ficient as it has to be ef fected only in a small number of elements .

[0789] It will also be understood that repeatedly broadcasting output data to the same or di f ferent targeted regions will not be as large a problem as would be the case where true global broad- casting is used .

[0790] Regarding the clock frequency for the buses , it has been stat- ed already that they might di f fer from the clock frequency of processing elements using either higher or lower clock fre- quencies on the bus . Obviously, in such a case , synchroni za- tion of busses and processing elements might become necessary, complicating the design .

[0791] It has been discussed that too many processing elements might be simultaneously waiting for broadcasting access to the bus- es . It has been suggested above that the preferred way to op- erate a bus in such a case might be algorithm-dependent , how- ever, allowing processing elements a round-robin access to the bus system for broadcasting their result data has been sug- gested . Giving round-Robin access may be preferred in view of the simplicity of its implementation, leading to signi ficantly simpler circuits .

[0792] Furthermore , it has been stated that where a particularly large number of processing elements is creating data traf fic on the buses , the number of bus lines should be increased . While the above remarks already disclose preferred ways of op- erating a bus system for a processor according to the present invention and give hints how to physically implement the bus system, it is believed that still a broad range of possibili- ties exist to implement the bus system suitable for di f ferent needs .

[0793] Hereinafter, a bus system will be more explicitly described which is favored in view of its scalability, its adaptability to both out of order mode and loop acceleration mode of the fact that it allows for high-bandwidths despite only rather small expenses in the hardware and consumption of silicon ar- ea .

[0794] In this context , it is noted that the processing elements con- currently preferred embodiment will be basically rectangular so that they can be easily arranged in a two-dimensional ma- trix having rows and columns ; other tilings would be possible such as hexagons or triangles might lead to a di f ferent inter- connection topology and consequently, a di f ferent bus topolo- gy . It is also noted however that a processor of the present invention and are currently preferred embodiment will have

[0795] 93

[0796] SUBSTITUTE SHEET (RULE 26) multiple arrays of processing elements such as a pattern of four arrays of 16 X 16 processing elementsee each, with the arrays arranged in a 2 x 2 pattern .

[0797] Above , it has been suggested that an array might have rectan- gular processing elements arranged in a matrix with rows and columns . This allows for a particularly simple yet ef ficient bus arrangement . The processing elements of a row can be ar- ranged between lines of hori zontal buses . Assuming without loss of generality a dataflow through each processing element from top to bottom, hori zontal bus lines can be provided above each row of processing elements for transmitting input data into the processing elements ; hori zontal bus lines below each row of processing elements can receive the respective output data generated by the processing elements and to be broadcast to all other processing elements .

[0798] While for special applications , such as low-cost , low energy applications , only a single hori zontal bus might be provided per row, typically, multiple hori zontal busses will be used .

[0799] Accordingly, more than one bus line per row will be provided in the preferred embodiment . For example , in the 16 x 16 pro- cessing element arrays currently favored, at least 7 bus lines may be provided in parallel per row . The bus lines between 2 rows both can receive output data generated by the processing elements arranged in the row above them and will preferably broadcast data to the processing elements in the row below them while the data is propagated hori zontally . Such propaga- tion might be ef fected both to the left side and the right side of a processing element , in particular with arrays so small that they do not need drivers . Accordingly, data gener- ated by a processing element in one row can be easily trans- ferred to a processing element directly below thereof . In the preferred embodiment currently designed, each pro- cessing element in a row can broadcast its data onto a se- lectable bus line of the 7 bus lines . The use of 7 bus lines is preferred in a 16x16 array . It will be anticipated that in a larger array such as a 32 X 32 array, more than 7 bus lines might be necessary to prevent signi ficant bottlenecks created by the larger number of processing elements per row . Even with 7 bus lines per row for 16 processing elements , some of the processing elements will have to wait occasionally until a suitable bus line becomes available .

[0800] To this end, suitable circuitry is provided which can for ex- ample be set by a hori zontal-bus-line-round-robin-access- arbiter .

[0801] Now, obviously, transmitting output data to only the data row below will not be suf ficient to broadcast output data generat- ed by one processing element to all other processing elements

[0802] 94

[0803] SUBSTITUTE SHEET (RULE 26) or all other processing elements needing the output data more precisely . Therefore , a number of the vertical bus lines in- terconnecting the respective bus lines between the rows are provided as well . Access to these vertical bus lines is man- aged by an arbiter .

[0804] The arbiter currently considered most preferable is a round- robin arbiter .

[0805] Using any arbiter to grant access to a bus system implies that broadcasting of output data will be delayed until access granted . While it would be possible to prevent the use of hor- i zontal buses until access to a vertical bus line is granted as well , this would prevent reuse of a processing element i f it has been released but not yet broadcast its data . Also , processing elements in the row directly below might receive the output data faster i f transmission onto the hori zontal bus lines not made dependent on the access to the vertical bus lines , reducing average latencies .

[0806] Therefore , it is preferred to allow access to the hori zontal bus lines independent of further access to the vertical bus lines . As data transmitted on a hori zontal bus line between two rows of processing elements can only propagate vertically once access to the vertical bus line has been granted by the arbiter, registers are provided between the hori zontal bus lines running between the rows and the nodes connecting them to the vertical bus lines . The registers store data generated in the processing elements of the row before access to a ver- tical bus line can be granted by the arbiter . It should be un- derstood that the registers will typically be arranged as first in- first out registers .

[0807] Even with particularly large FI FO, a situation still may occur where the FI FO is full at the time processing element intends to access the hori zontal bus .

[0808] Problems arising from this can be avoided by using a proper protocol where the FI FO has to acknowledge receipt of data transmitted along a hori zontal bus line and / or guarantees proper handling, that is either direct forwarding or buf fer- ing, e . g . because the FI FO controls access to a bus line and allows access only as long as results can be buf fered; for the latter case , prior to transmission of data along the hori- zontal bus line to the FI FO, the FI FO may signal to a pro- cessing element that it may send its output data . The latter variant may be preferred as access to a hori zontal bus line would be automatically arbitrated as well . So , it can be guar- anteed to the processing element that once the data will be handled properly once it has been sent out and accordingly, in case the processing element would have been released -that is an instruction in the thread has been found that reuses the

[0809] 95

[0810] SUBSTITUTE SHEET (RULE 26) register the processing element currently represents -the pro- cessing element might receive its next instruction .

[0811] However, where multiple bus lines are provided, care must be taken that upon triggering a transmission from a processing element , the output data is sent to only one bus line . Note that where multiple bus lines are provided, a separate FI FO will typically be provided for each bus line .

[0812] Access can then be controlled using a common finite state ma- chines for all bus lines in the row; alternatively, a separate finite state machine could be provided for each bus line- in that case , a request from a processing element for access to a bus line could either be directed to only one speci fic bus line of the plurality of parallel bus lines running along the row, with the speci fic bus line being in particular and pref- erably selected according to a target region indicated when triggering transmission or with the speci fic bus line being selected fixedly while for example by corresponding fixed hardwiring . The latter case is not preferred though, as some- times several processing elements from which results are re- quested repeatedly could share the same bus line , again lead- ing to a bottleneck situation otherwise avoidable . It would also be possible to provide a common finite state machine con- trolling operation of at least all FI FOs assigned to the par- allel bus lines running between a row and having arbitrated or otherwise restricted access to a system for vertically or oth- er propagation of output data . Repeatedly transmitting the da- ta onto several bus lines in response to only one single re- quest can also be avoided by circuitry in the respective pro- cessing elements allowing broadcast only once per received trigger .

[0813] When the hori zontal bus is forwarding the data, several things may happen, depending on the actual implementation and thus interconnection topology . A simple versatile topology is one where any vertical line will feed its data onto a hori zontal bus lines in all rows other than the row where the data pack- age comes from . While this is a particular simple implementa- tion, it can be anticipated that a large number of bus lines will thus be blocked by every single transmission . A slightly less simple implementation will allow to forward data to only a single row for broadcasting .

[0814] This can be ef fected by either including corresponding infor- mation in a data package so that each node can decide whether a data package should be further forwarded along the vertical line or whether it should be trans ferred onto a vertical bus line .

[0815] Decoding the direction in this manner is rather simple . It is noted that ALUs in an array broadcasting has already been sug-

[0816] 96

[0817] SUBSTITUTE SHEET (RULE 26) gested for bus systems , compare DE 19 651 075 . Furthermore , in WO98 / 35294 , it has been stated that for conventional buses used in arrays , only one data package can be placed on the bus at any given time , resulting in the suggestion to use a seg- mented bus system that can automatically set up an intercon- nection in an array and which allows to route data in the nodes that also divide the bus into segments and it has also been suggested that data can be broadcast to all or to only someunits in an array .

[0818] In the present case , routing is particularly simple because only a suitable hori zontal row has to be selected . Even in cases where the initially hori zontal bus line (used for trans- fer of data from a processing element into the FI FO and then further to a vertical bus line ) would not be selected accord- ing to a target region among a plurality of parallel bus lines running along a row, only a few bits would have to be used for directing the data package . This is a clear advantage over cases where a complete routing needs to be ef fected in an ar- ray . Therefore , it is considered inventive per se .

[0819] Where a plurality of bus lines is used simultaneously and di f- ferent bus lines lead to di f ferent rows or di f ferent regions of rows , routing is further simpli fied and becomes hardware and energy ef ficient to an extent that applicant also consid- ers inventive per se . It will be seen that the bus line ar- rangement of a preferred embodiment of the present invention allows for such preselection, in particular by selection of a initial hori zontal bus line already according to a target re- gion .

[0820] Reducing the hardware and energy expense for routing also al- lows to scale up the processor by use of a number of arrays , in particular more than 4 , in particular more than 16 arrays per integrated monolithic circuit or integrated system on a chip . For the latter cases cases , additional routing infor- mation may be provided by additional routing or target region indicating bits .

[0821] During operation, a situation might occur where a processing element has forwarded its output data onto a hori zontal bus line and the output data is waiting in one of the FI FO regis- ters to be forwarded to other processing elements via a verti- cal bus line and from there to hori zontal bus lines in the other rows ; during such period of waiting for forwarding the data via the vertical bus lines , several clock cycles might pass and accordingly, a situation might arise where broadcast- ing of the same output data still waiting in the FI FO register is requested again for additional processing elements having received their instructions during the time the data is still waiting in the FI FO register . It can be anticipated that the second processing element on behal f of which the ( re- ) broad- cast has been triggered will take advantage of the first

[0822] 97

[0823] SUBSTITUTE SHEET (RULE 26) broadcast by snooping the bus ; where the broadcast is not global but targeted to certain regions , this obviously only holds as long as the second processing element on behal f of which the rebroadcast has been triggered is in the same target region or is connected to the path leading into the target re- gion . Where this is the case , actually rebroadcasting would be in vain .

[0824] In a similar way, output data from one and the same sending processing element might even be needed by several spaced apart processing elements in one of the same clock cycle .

[0825] Several strategies exist to react to this scenario .

[0826] Where the broadcast is truly global , a single broadcast re- quest can be send to the respective processing element that has generated the data needed . As discussed, this might clog the data busses . Accordingly, with targeted broadcasting, the retrigger could be send multiple times to the sending pro- cessing element , triggering one transmission per requesting processing element . This will inevitably lead to at least ex- tremely long latency is which is unacceptable for performant processors . Accordingly, a more preferred way would be to al- low a targeted broadcast to plural , spaced apart regions , such that all regions in which processing elements needing the out- put data will receive the data by only one broadcast transmis- sion . To this end, broadcast target regions can be defined such as rows 1 - 16 , combinations of rows or certain areas comprising for example processing elements in a square having corner coordinates ( 1 , 1 ) and ( 4 , 4 ) in a rectangular 2 dimen- sional array . Where this is the case , the regions to which a broadcast should be simultaneously targeted can be indicated by a suitable bit pattern . A particularly simple bit pattern would be a bit pattern having one bit per target region . This could be used in nodes along a propagation path to mask out without substantive hardware expense those regions into which the output data are to be propagated, for example by either forwarding data into a hori zontal row attached to the respec- tive node or not . There is at least a twofold advantage of such targeted broadcast , namely first , a reduced use of buses resulting in lower latency is due to lower waiting times until fast access and / or data propagation and second, a reduced en- ergy consumption as the amount of data for which processing elements waiting for their operands have to determine that certain transmissions on the buses they snoop are irrelevant for them .

[0827] While this helps with the problem of simultaneous request , the problem where di f ferent processing elements request data from one and the same processing element from which data currently is in the process of being propagated to a target region re- mains .

[0828] 98

[0829] SUBSTITUTE SHEET (RULE 26) Again, a simple solution would be to ignore that the output data previously requested but not yet propagated to the tar- geted processing element is still " in flight" , for example be- cause it is still in the FI FO register , i f this is ignored, the respective processing element requested to ( re- ) broadcast its output data again simply does exactly that : rebroadcast its output data . This may lead to additional traf fic on the (hori- zontal ) bus lines and may lead to unnecessary filling of the

[0830] FI FOs , in particular in cases where information is broadcast globally or where the target region of the new request re- sponse to the target region of the initial request for which data is stored in the FI FO .

[0831] Nonetheless , in view of the simplicity of hardware implementa- tion, rebroadcast would not be particularly disadvantageous at least as long as bus traf fic constitutes no signi ficant bot- tleneck to high performance of the device .

[0832] A more complex approach to rebroadcast requests would be a feedback between the processing element outputting the data and the FI FO; by such feedback, re-triggering of broadcast could be prevented until the respective output data is actual- ly transmitted onto as vertical bus line or even until it has been received by his first destination target .

[0833] As a mechanism to ensure that even upon multiple requests to broadcast results , broadcasting will be ef fected only once , a feedback signal from the FI FO to the processing element indi- cating whether data previously trans ferred from a processing element to the FI FO has been forwarded on to a vertical bus line or not can be evaluated, preventing re-triggering until the data has been forwarded .

[0834] Also , in case the processing element on behal f of which re- broadcast of output data still stored in the FI FO is triggered is placed in the row directly below, any disadvantage from this could be avoided by actually trans ferring output data from the vertical bus lines to all rows including a row imme- diately below the row transmitting the data . The remaining disadvantage of increasing latency slightly can easily be ac- cepted .

[0835] Alternatively, re-triggering could be allowed despite the data still being stored in the FI FO at least for cases where the processing element on behal f of which rebroadcast is requested is placed in a row directly below the row of the processing element having to transmit the output data- obviously, this would require a suitable trigger management .

[0836] A more complex rebroadcast trigger management could be imple- mented in the instruction issuing circuitry or the correspond-

[0837] 99

[0838] SUBSTITUTE SHEET (RULE 26) ing stage responsible for triggering re-broadcast- but a more complex rebroadcast trigger management is disadvantageous in itsel f exactly for being more complex .

[0839] As another alternative , storing another copy of output data in the FI FO between the hori zontal lines and the vertical bus lines could be avoided i f a check is made before storing the output data in the FI FO- but as it might be necessary to check every single entry before storing it , overall additional power consumption and hardware expense would be signi ficant .

[0840] So , currently the still simple solution of using a feedback as described above might be preferred overall , in particular where the waiting period until access to the vertical bus lines is granted is reduced due to a suf ficiently large number of bus lines . It will be understood that this easily inte- grates with the use of the finite state machines associated with the FI FOs registers and allowing the processing elements access to the bus line in which the FI FO is provided .

[0841] Another possibility would be to wait until the first broadcast has been acknowledged either by an intermediate note of the final receiving processing element . A protocol where receipt is acknowledged-in whatever manner-is generally preferable ; where an acknowledgment is implemented and several broadcast- ing transmissions are allowed from a single processing ele- ment , it is distinction would have to be made between acknowl- edgment of the first transmission and a further transmission . As this complicates both the broadcasting process and the ar- chitecture needed, it is generally preferred to have only one transmission " in- flight" .

[0842] As having to wait for access to the vertical bus lines may in- crease the latency between triggering transmission and recep- tion of the output data by a receiving processing element , it may be preferred to provide at least some connections as feed- back buses to the row above the row onto which a processing element outputs its data . This also allows a processing ele- ment to receive the output data it has generated itsel f as in- put operands and on the other hand doubles the number of pro- cessing elements that will receive output data generated di- rectly without trans fer via vertical bus lines . Again, this reduces latency .

[0843] In a similar manner, bus lines could be provided not j ust to a row directly below the processing element via the hori zontal bus lines could also and / or alternatively be provided to the next but one row . Therefore , in a practical implementation, the bus system might look as shown in figure 15 .

[0844] There is a speci fic advantage of preventing a second transmis- sion while the first transmission is still in flight . The spe- ci fic advantage relates to the load / store queues . For loading data, and address has to be generated, the data has to be re-

[0845] 100

[0846] SUBSTITUTE SHEET (RULE 26) trieved ( loaded) from a memory at the address generated has to be stored in a register to which thereafter, instructions can refer . In the design of the present invention with the output register of the processing element serves to store data in its output register, a particularly ef ficient approach to load da- ta is to instruct a processing element with a single instruc- tion to first generate the address e . g . based on operands it receives , to then transmit-or broadcast by global broadcast or by targeted broadcast-the generated address from its output register to a load store queue and to thereafter receive the data retrieved by the load store queue in its output register . Once the data retrieved actually is stored in the respective processing element having previously generated and broadcast the address generated, other processing elements can request the data from the processing element . However, such requests might come even before the processing element itsel f has re- ceived the data and placed it in its output register . Allowing only one broadcast in flight prevents a situation where in re- sponse to a rebroadcast trigger, address data instead of the data retrieved are broadcast . Note that in such situations care can be taken to prevent the address generating processing element to broadcast the address globally to the load store queue as this would allow snooping processing elements waiting for the actual data to mistake the address data for the data still to be loaded . One particular way to prevent erroneous snooping of address data by conventional receiving processing elements waiting for the actual data rather than the address data is to add a speci fic flag or bit pattern to the ID of the transmitting processing element . Where such additional bit pattern a speci fic flag is used, the following bits that oth- erwise represent the transmitting processing element ID can be construed di f ferently or, where a change of the following bits is allowed between transmission of the address is generated and transmission of the actual data once retrieved, can for example indicate a slot in the load store queue .

[0847] The load store instruction issued to a ( typically : common) processing element might also instruct the processing element to only broadcast the result to a target region only contain- ing load store units . Thus , this will deviate from the stand- ard in natural global broadcast ; another way would be to pro- vide dedicated buses only leading to the load store queues or to add a particular bit in the processing element ID which when evaluated by non-load store queue processing elements would result in a determination that the data is to be ig- nored . , It can easily be avoided that addresses are mistakenly sent out instead of data .

[0848] A practical implementation of the bus system at the output side of processing elements is shown in the following Figure .

[0849] 101

[0850] SUBSTITUTE SHEET (RULE 26) In view of the above explanations relating to broadcasting re- sults , by way of example only, a currently preferred design of an output broadcaster is shown in figure 16 .

[0851] The figure shows a brute force version of the listener . The issuer sends the 1-hot encoded region ID ( 16 regions in the processing ele- ment-array + maybe 4 special regions ) . This version simplifies the issuer and the processing elements at the cost of a bigger global issue bus ( going to each processing element ) . For sending up to 8 retriggers (when issuing 4 instructions ) , that is 8 ** {{ 2200++99 )) = 232 wires . The source ID is encoded in 9 bits , which includes plenty of options for invalid source IDs to express that there is no retrigger request via that bus .

[0852] With respect to the "invalidate processing element Result" signal , it is remarked that if the processing element had an initial broad- cast and got the ack that the address was broadcasted, the "invali- date the processing element result" should be such that it only be- comes valid again after the loaded data ( in case of a load instruc- tion) is received .

[0853] With respect to the "initial broadcast" stage , the Initial Broadcast and the LSQ Region ID serve a special case for load / store instruc- tions in a processing element . At first , this processing element needs to calculate the address and broadcast it to the Load / Store Queue ( Initial Broadcast ) and then, in case of a load instruction, the instruction reuses this processing element to store the loaded data in its output register after it arrived , Whenever some other instruction wants to read that loaded data , it will send a retrig- ger . The processing element must not mistakenly broadcast the calcu- lated address to someone waiting for the data , therefore it needs to separate the initial broadcast { only to the LSQ ) and subsequent re-triggers .

[0854] With respect to "broadcast in flight" stage , aass soon as the broad- cast request is copied to the row FI FO , it must be remembered that the processing element has a broadcast in flight . That is important as this processing element must not be re-allocated for a new opera- tion before that broadcast really arrives at its destination . That is needed to avoid that some other new allocated processing element receives our old packet and thinks it came from the new operation that was allocated to this processing element . Therefore , a Working flag is kept set for as long aass tthheerree iiss a broadcast in flight or some pending broadcasts .

[0855] With respect to the "PE WORKING" signal , the processing element can- not be reallocated when the output still needs to be broadcasted or some broadcast is still in flight .

[0856] With respect to the ''broadcast Target regions" stage , it is noted that as soon as the broadcast request the processing element is cop- ied to the row FIFO, the processing element no longer needs to store the target regions locally . The AND gates shown mas k them out and the subsequent OR gates add new broadcast requests that may have arrived at the same time . A broadcast is only sent to the horizon-

[0857] 102

[0858] SUBSTITUTE SHEET (RULE 26) tai busses if the proces sing element has data to be broadcasted and if the processing element has no earlier broadcast that is still in flight .

[0859] With respect to the "Ack Broadcast copied a Row FIFO" , that signal must arrive in the same cycle where the broadcast target regions were sent out .

[0860] With respect to the "broadcast request" signal , a broadcast is only send out to the horizontal buses if the proces sing element has data to broadcast and i f there is no earlier broadcast that is still in flight .

[0861] Overview over LAM and OoO Busses :

[0862] Reference should now be had to figure 17 and figure 18 . A part initially colored and relating to the loop acceleration mode has been deleted in the second part of these figures . Of course , the actual busses would not be interrupted; it be understood that any interruptions drawn stem from graphically processing figure 17 .

[0863] As can be seen, the horizontal lines running above processing elements comprise forward bus lines fed from proces sing ele- ments in the row above, feedback bus lines from the row below and horizontal bus lines fed from the vertical bus lines run- ning parallel to the inputs of the processing elements respec- tively .

[0864] On the output side of the processing element rows , a plurality of bus lines aarree pprroovviiddeedd .. This can be explained better with respect to figure 19.

[0865] In this figure , a number of horizontal bus lines are shown .

[0866] Note that the number of bus lines is not restricted to 3 bus lines as indicated by the dotted points in the figure . In the figure shown, the vertical bus lines (not shown) to which the horizontal bus lines will transfer the output data of the pro- cessing element will be provided at the right side , as indi- cated by the rightmost arrows . What is shown is FIFO with its finite state machine for controlling the operation thereof and for sending out bus protocol acknowledged data signals to the processing elements and receiving acknowledged signals from the process ing elements . As indicated, each proces s ing ele- ments can be connected to any of the hori zontal bus lines ; the nodes indicated allow to access any of the bus lines selec- tively . In the example shown, broadcast requests can be sent to the finite state machine , again selectively . The width of the of a bus line is such that 32-bit output data can be transferred in parallel to an intra- region ID and to a broad- cast target region ID several bits wide .

[0867] 103

[0868] RECTIFIED SHEET (RULE 91) ISA / EP According to this figure , the data packages sent from a pro- cessing element do not j ust include the actual result from the output register, which in the figure shown is 32 bit wide , but will also include a broadcast target region and in intra- region source ID . The intraregional source ID consists of the least signi ficant bits of the source ID and with a width of 4 bit allows to identi fy the processing element in a 16- processing-element-row . Transmitting the intraregional source ID allows to send an acknowledgment signal ( e . g . according to an RDY / ACK protocol ) from the hori zontal line bus FI FO back to the processing element . These 56 bits can be combined into one data package on a 56 bit wide bus line . There are additional input and output lines for controlling transmission, namely for requesting bus access and for receiving acknowledgment signals .

[0869] Regarding bus request signals , these can be sent either to all hori zontal bus lines allowing access to the vertical bus line or, could be sent only to speci fic hori zontal bus lines in a manner depending on the processing element which requests the output data or on behal f of which output data is requested, thus requesting for example speci fically a feedback bus line , forward bus line or a hori zontal bus line with access to the vertical bus lines . Using a global broadcast request simpli- fies requesting and allows for faster transmission of the out- put data onto a hori zontal bus line . In such a design embodi- ment , the broadcast request will be broadcast to all FSMs con- trolling the FI FOS in the hori zontal bus lines ( or, where ap- plicable be broadcast to a uni fied finite state machine used for controlling operation of all or at least multiple of the FI FOs provided in the hori zontal bus lines ) .

[0870] On the other hand, where true global broadcast is used, it must be ensured that every hori zontal bus line is connectable to the vertical bus lines . Note that it is possible to combine global broadcast and targeted broadcast , in particular in im- plementations where global broadcast of results generated will take place independent of a preceding request . Such global broadcast ef fected at any rate and thus without explicit re- quest for the output data is particularly advantageous in cas- es where several instructions are issued simultaneously . Here , the situation might arise where some instructions issued might already need the results from instructions preceding in a se- quence of instructions but issued simultaneously . Obviously, globally broadcasting results immediately at any rate will re- duce latencies in such cases . It is understood that even there , it would be possible to issue the instruction to the processing element processing the preceding instruction which other instructions simultaneously issued might need together with a signal otherwise used for triggering a rebroadcast and indicating that the processing element processing the preced-

[0871] 104

[0872] SUBSTITUTE SHEET (RULE 26) ing instruction actually has to broadcast its result as soon as possible .

[0873] Any broadcast could be transmitted globally, even a re- triggered transmission . However, this might be neither energy ef ficient nor might result in acceptable access times to bus lines of bus lines unless a very large number of bus lines is provided . Therefore , it is preferred to not broadcast results globally every time other cells need the results . Several techniques and possibilities exist in this respect .

[0874] In a first implementation, each broadcast transmission could be ef fected to one row only as speci fied in a corresponding broadcast request . This is helpful where feedback bus lines and / or forward bus lines are provided, in particular in paral- lel to hori zontal buses connectable to one or more vertical bus lines . Obviously, target information relating to the re- spective row could be transmitted to the broadcasting pro- cessing element when triggering the ( re-- ) broadcasting . Therefore , in an alternative design, the broadcast request the processing element sends out to the bus lines it may transmit- ted onto might be for a speci fic bus line or other type of bus line , such as a feedback bus line or a forward bus line or a general , vertically connected bus line .

[0875] Where this is the case , the forward bus lines and the feedback bus lines would not have to be connectable to the vertical bus lines and thus will not need have a FI FO . This might save some space and allow for more hori zontal bus lines to be provided . On the other hand, requesting a speci fic bus line or a speci f- ic type of bus line necessitates to evaluate the relative po- sition of the transmitting processing element and the receiv- ing processing element . This could be done centrally, thus re- ducing the hardware expense within each processing element , but it will still require to then send corresponding requests to the processing element .

[0876] Therefore , a design decision needs to be made between e . g . the most simple-global-bus request and more speci fic bus requests . It should also be noted that in the di f ferent cases of broad- cast requests , circuitries such as those for handling acknowl- edgment signals and the associated finite state machines will be correspondingly adapted .

[0877] Note that in certain implementations of the invention, it will be possible to connect the feedback bus lines or forward bus lines to the vertical bus lines as well even though of the de- sign implements bus line speci fic broadcast requests ; where this is the case , the number of vertical bus lines might be increased . (Alternatively, no forward buses and / or no feedback buses would have to be provided, thus simpli fying the design to only identical hori zontal bus lines identically having

[0878] 105

[0879] SUBSTITUTE SHEET (RULE 26) FI FOs and identically being connected each to one or more ver- tical bus lines . )

[0880] Increasing the number of vertical bus lines is particularly advantageous where a plurality of processing element arrays are arranged in a pattern and communication between di f ferent arrays is to be provided inter alia via vertical bus lines ex- tending beyond the length of a single array, preferably in a segmentable manner .

[0881] Another way to increase communication in a multi--array pat- tern would be to extend some hori zontal bus lines into the next array, for example feedback bus lines or forward bus lines . The parts extended into a neighboring array might lead to FI FOs there and from there allowed distribution onto verti- cal bus lines in the neighboring array . Where within one array the hori zontal bus lines comprise at least two bus lines used for feedback and / or forwarding and are extended into one and the same neighboring array, one of the 2 lines could be used to ef fect a transmission in a vertically upward direction only while the other of the 2 lines might allow a transmission in a vertically downward direction only . While it would be possible per se to allow processing elements in the neighboring array to also transmit data onto the extended bus lines , this might not be advisable because this would double the number of pro- cessing elements that could transmit onto the bus lines , po- tentially leading to a congestion . Accordingly, it is pre- ferred to simply extend the bus lines into the neighboring ar- ray and to preferably allow direct snooping of these bus lines by those processing elements in the neighboring array which are in the respective rows for snooping . Where an array has neighboring arrays both on the left side and the right side thereof , at least one bus line could be extended into each neighboring array on the left and right side respectively; preferably, at least 2 hori zontal bus lines will be extended into each neighboring array .

[0882] It should be understood that inter-array communication can be expected to be lower than intra-array communication, so that the overall number of (hori zontal ) connections between arrays may be smaller than the overall number of (hori zontal ) connec- tions with in an array , The same holds for vertical intercon- nections , of course .

[0883] Where more than a 2x2 pattern of arrays is used in a higher end device , additional bus lines might be provided and / or bus lines extended into one neighboring array might have further extensions into other arrays .

[0884] It will be understood that providing intra-array connections greatly simpli fies scaling . The additional hardware expense for extending the bus lines is small , and circuitry such as

[0885] 106

[0886] SUBSTITUTE SHEET (RULE 26) that for requesting output data generated by a processing ele- ment in a di f ferent array can be designed without disturbing or signi ficantly altering the arrangement of processing ele- ments per se .

[0887] Regarding the acknowledgment signals , the following is noted : The acknowledgment signals will be received from the FI FO in the hori zontal line bus which has been used for the transmis- sion of the data package . Two di f ferent acknowledgment-signals can be sent from the FI FO : a signal acknowledging that the output data has been received in the FI FO and is safely ac- cepted therein; and a signal indicating that the output data has been forwarded from the FI FO onto a vertical bus . It should be understood that upon receiving the "out of FI FO" ac- knowledgment signal , further actions can be taken in the pro- cessing element , for example allowing a further request for bus access in designs where only one transmission in flight per processing element is allowed on the hori zontal bus lines directly accessible by the processing element or allowing that the respective processing element accepts a further instruc- tion i f released .

[0888] For the record : while at a number of occasions , reference is had to limiting the broadcasts in flight from one and the same processing element to exactly one broadcast , a counter might alter natively be implemented to restrict the broadcasts in flight to a number di f ferent from 1 , This would reduce latency for those requests arriving somewhat later but on the other hand might increase the bandwidth requirements on hori zontal and vertical busses . Where targeted broadcasting is used and the broadcast regions targeted at indicated by a bit pattern, a bit pattern representing the combined regions could be de- termined . This could be done before the broadcast centrally before a rebroadcast is triggered, it could be done within a processing element before access to the bus is requested and it could be done while the output data still kept in the hori- zontal bus line FI FO . Care must be taken not to fuse broad- casts from one and the same processing element that relate to di f ferent output results . This is particularly important where processing element are used for load / store instructions by first having to generate and output the address and by there- after having to receive the loaded data, trans fer the loaded data into the output register and by then having to broadcast the loaded data to processing elements requesting the data .

[0889] By using two acknowledgment signals , one for acknowledgment received in the FI FO and one for acknowledgment of transmis- sion to a destination, the broadcasting processing element cannot only be informed that new broadcast requests from other processing elements should be received, but also premature al- location of new instructions to the broadcasting processing element can be prevented . In more detail , problems might arise

[0890] 107

[0891] SUBSTITUTE SHEET (RULE 26) i f a processing element broadcasts data to a FI FO for trans- ferral to a second processing element , is immediately thereaf- ter a new instruction is issued to the first processing ele- ment while it ' s data from the preceding instruction is still held in the FI FO, and then a third processing element requests the results from the first processing element . In such a situ- ation, even with targeted broadcasts , situation may occur where the third processing element is in the same broadcasting region as the second broadcasting element , consequently re- ceives-and accepts-the first transmission and thus has incor- rect data . By allowing acceptance of new instructions in the first processing element only after the first broadcast has left the FI FO, such problems can be avoided . Thus , a signal is needed to indicate that output data has left the FI FO .

[0892] It is understood that in certain cases , the acknowledgment signal that output data has left the FI FO cannot be construed to indicate that the output data will actually be received at the target region ( s ) in the current clock cycle ; rather, in particularly complex arrangements , additional FI FO' s might for example be placed in nodes or particularly large processors might have pipelined buses . In such cases , it is possible to only send out the second acknowledgment signal once the recep- tion in the targeted region-or each targeted region-has been confirmed to the node at the end of the hori zontal bus line the broadcasting processing element has accessed . Accordingly, the second flag can be construed to be a flag indicating that no more broadcast is in flight .

[0893] The average skilled person will understand that where several broadcasts and flights are accepted, the acknowledgment sig- nals must be adapted correspondingly, for example by multiple 'Ack ( out of FI FO) ' being transmittable to the same processing element at a given time ; that case either a decision needs to be made which of a multitude of acknowledgment signals is to be transmitted to the same processing element first or alter- natively, a bit for every broadcasting flight allowed should be provided . Note that handling multiple broadcasts in flight is simpli fied i f each these multiple broadcasts is using a di f ferent hori zontal bus line , broadcast multiple for the same PE at the same time and then we need to select which of them shall be returned first ( or we need per-FI FO-ack-bits for every PE ) .

[0894] On the input side to a FI FO in a given hori zontal bus line , a multiplexer can be provided for selecting between the output connections leading from the processing element to the FI FO . The finite state machine then select the output connection by the respective input and accordingly also knows to which pro- cessing element the 'Ack ( into FI FO) ' must be sent . Thus , in such a case , tthhee ' Intra-region Source ID' might not be needed to send the 'Ack ( into FI FO) ' , as the FI FO explicitly selects from which PE to accept the input . Note that this can be true

[0895] 108

[0896] SUBSTITUTE SHEET (RULE 26) where a common finite state machine for all bus lines in a hori zontal row is used .

[0897] It should be understood that processing element in the out of order mode must not only receive their operand data but also the instructions and the rebroadcast triggers . For issuing in- structions to the processing elements , a bus system separate from the data bus system is preferred and implemented in the practical embodiment . Also , the issuing interconnection net- work can also be used for triggering rebroadcast . Using the same network for triggering rebroadcast and for issuing in- structions is preferable because in that way, triggering re- broadcast can be more easily ef fected the same clock cycle as issuing the instructions , reducing overall latency .

[0898] To make use of the high performance capability of the array, multiple instructions should issue each clock cycle . These multiple instructions may relate to the same thread, imple- menting a single thread multi-issue method, or they might re- late to di f ferent threads . In both cases , the instruction bus- es-or instruction interconnections-must be arranged such that issuing a plurality of instructions within one (processing) clock cycle becomes possible . Where each instruction has to wait for the same bus lines at the buses are operated with the same frequency, this will not be possible . Also , operating buses at higher frequencies often necessitates the use of pipeline registers in the buses , again increasing latency . Therefore , in a preferred embodiment , instructions will not be broadcast globally or routed along a global bus system to a speci fic processing element ; rather, the instruction buses will be divided into regions , with each of the multiple in- structions to be issued simultaneously being issued into a di f ferent region . Preferably, these di f ferent regions will be non-overlapping . The di f ferent regions might be the same as the broadcast regions defined for targeted broadcast into spe- ci fic regions . This simpli fies management of regions . However, it might be preferred to use regions defined di f ferently . The reason for this resides in single thread multi-issue instruc- tions issued for typical algorithms . Very frequently, instruc- tions directly following in a sequence of instructions will require results generated in the instruction directly preced- ing or preceding only a few steps before . It is likely that such instructions are issued in a single threading-multi- issue-operation in the same clock cycle and will thus be placed into di f ferent broadcast regions . Accordingly, the problem is way more pronounced for single thread multi-issue operations . Depending on the data bus topology and use , it is not unlikely that the respective processing element are spaced further away from each other on average than would be the case i f the multi-issue regions are di f ferent from the broadcast regions . Above , it has been described that hori zontal bus lines are provided including hori zontal forward bus lines and

[0899] 109

[0900] SUBSTITUTE SHEET (RULE 26) hori zontal feedback bus lines ; it is believed that enhanced use of this hori zontal data interconnection structure becomes possible by issuing multiple instructions in one clock cycle along structures orthogonal thereto .

[0901] Note that even where multiple instructions per clock cycle are issued into di f ferent regions , a situation might still arise where all processing elements from which operand data are needed for instructions currently issued and would thus have to be re-triggered, are placed by chance in the same instruc- tion issue region . Therefore , buses need to be suf ficiently wide . For example , where issuing 4 instructions per clock cy- cle , up to 8 operands would be needed and accordingly, up to 8 processing elements would have to be re-triggered . What is more is that these up to 8 processing elements would each have to receive their broadcast target region ( s ) as well , resulting in rather broad buses for example , for issuing 4 instructions , the up to 8 retrigger signals together with the respective target regions and the addresses of the processing element concerned would be 232 bits wide .

[0902] An example of an instruction issue column-structure for an ar- ray having rows and columns is shown in figures 20 and 21.

[0903] Figure 20 shows an issue into four slots , each of the slots being capable of issuing instruction...

Claims

Claims1 . A data processor comprising at least an instruction issue stage issuing instructions , a number of processing elements to which at least some of the instructions are issued and which receive operand data, generate result data in accordance with the instruc- tions received and transmit their result data to other processing ele- ments for use as new operands ; and a bus system for these transmissions , wherein an instruction issue stage is provided adapted to issue instructions to a group of processing elements to operate them in at least two di f ferent modes , namely an out-of-order mode wherein instructions may be exe- cubed out of order a loop acceleration mode wherein loops can be executed ef ficiently, and the bus system comprises an arbiter operative in the out-of-order mode to arbitrate access of the group of processing elements to at least a part of the bus system and inoperative in the loop acceleration mode .2 . The data processor according to claim 1 , wherein the processing elements have unique identi fiers , the bus is adapted to be used with a protocol allowing a a transmitting processing element to be identi fied by its unique identi fier and274the processing elements have snooping circuitry for snooping the bus system in the out-of-order mode .3 . The data processor according to any of the preceding claims 1 or 2 wherein the group of processing elements is arranged in a multi- dimensional array the bus system has first interconnections usable by the group of processing elements during loop execution in a temporarily fixed manner for transmissions generally along a first direc- tion and second interconnections usable by the group of processing elements during loop execution in an unarbitrated manner for transmissions generally along in particular a direction opposite to the first direction and usable by the group of processing elements dur- ing out of order execution in the arbitrated manner for transmissions between processing elements of the group .4 . The data processor according to the preceding claim where- in, for loop acceleration mode , an instruction issue stage is adapted to issue instructions to the group of processing ele- ments and to issue instructions to fix the bus interconnection for the time of accelerated loop execution .5 . The data processor according to one of the preceding claims 1 -4 wherein, the first interconnections are provided with path balancing registers for use during loop accel- eration mode .6 . A data processing chip comprising a plurality of processing elements and an interconnection arrangement having for the transmission of data in a general data flow direction a plurality of interconnection data lines interconnecting the processing elements of the plurality of processing elements275with other processing elements of the plurality of processing elements wherein between di f ferent pairs of processing elements of the plurality of processing elements di f ferent number of lines are provided such that the number of interconnection lines provided be- tween processing elements of the plurality of processing elements will be higher for pairs of processing elements closer to each other than for pairs of processing elements further apart from each other and wherein at least some of the interconnection data lines each follow a path to a number of processing elements are subsequently arranged at the path such that they can receive the data, and can either allow the data to propagate further downstream its path to the other processing ele- ments following subsequently or can transmit other data along its path to oth- er processing elements following subsequently .7 . A data processing chip according to the previous claim, wherein multiplexers are used in the path to either allow data to propagate further downstream the path to the oth- er processing elements following subsequently or to transmit to other processing elements following subsequently di f ferent data .8 . A data processing chip according to the previous claim, wherein the di f ferent data transmittable to other pro- cessing elements following subsequently is at least one of result data generated in the processing element , con- trol data generated in the processing element , in partic- ular trigger data and / or address data .9 . A data processor chip according to one of the preceding claims wherein the processing elements are arranged in a two dimensional array having rows and columns or are arranged in an array having multiple planes having rows and columns ,276wherein at least some of the processing elements receive inter- connection lines starting at or following paths along the processing element in the row above in the same column, the processing element in the row above in the col- umn left thereof the processing element in the row above in the col- umn two steps left thereof and preferably also from the processing element in the row above in the col- umn right thereof and preferably also from the processing element in the row above in the col- umn two steps right thereof .10 . A data processor chip according to the preceding claim wherein the least some of the processing elements receive interconnection lines for the transmission of data in a general data flow direction only starting at or following a paths coming from the processing element in the row above in the same column, the processing element in the row above in the col- umn left thereof the processing element in the row above in the col- umn two steps left thereof the processing element in the row above in the col- umn right thereof the processing element in the row above in the col- umn two steps right thereof .11 . A data processor chip according to claim 9 or 10 , wherein for at least some processing elements of the plurality of processing elements , the interconnection lines between one of the at least some of the processing element and other processing elements of the plurality of pro- cessing elements placed in the row above the row of the one of the at least some of the processing elements and in the column two or more steps left or right thereof start at the at the row above at the other processing ele- ments .27712 . A data processor chip according to one of the preceding claims wherein the interconnection lines and preferably only the in- terconnection lines between one of the at least some of the processing element and other processing elements of the plurality of processing elements placed in the row above the row of the one of the at least some of the processing elements and in the same column follow a path that has passed or started at processing elements in the same column but at a row at least two rows above .13 . A data processor chip according to one of the preceding claims wherein the number of interconnection lines coming from the processing element in the row above in the same column is at 4 and preferably less than 6 , coming from the processing element in the row above , but in the column left or right from thereof is less than the number of lines coming down from that row from the processing element in the same column and is preferably no more than 3 and is preferably 2 , and coming from the processing element in the row above but in the column two steps left or right from thereof is less than than the number of lines com- ing down from that row from the processing element in the same column and is preferably 1 .14 . A data processor chip according to one of claims 6 - 13 wherein the processing element are capable of executing instructions of a register based instruction set archi- tecture , the data processing chip is adaptable to process data in a dataflow manner and the number of data input lines into one of the plurality of processing elements and usable while processing data in a dataflow manner is lower than the number of regis- ters used in the instruction set .15 . A data processor chip according to the preceding claims wherein each of the plurality of interconnection data lines has a number of parallel wires and is operated us- ing a bus protocol .16 . A data processor chip according to one of one of claims 6- 15 , adapted for use in a dataflow processing mode wherein for the dataflow processing mode , the instructions of a278number of processing elements and the interconnections be- tween the processing elements using interconnection data lines of the plurality of interconnection data lines are fixedly set for a plurality of processing steps .17 . A data processor chip according to one of claims 6- 16 , adapted for use in a dataflow processing mode and having instruction f etch / decode / issue circuitry for fetching, de- coding and issuing instructions to processing elements for execution in the dataflow processing mode , wherein the number of data input lines into one of the plurality of processing elements and usable while processing data in a dataflow manner is lower than the number of registers used in the instruction set , the instructions fetched for execution and dataflow mode refer to more di f ferent registers than the number of data input lines into one of the plurality of processing ele- ments and wherein the instruction f etch / decode / issue circuitry for fetching, decoding and issuing instructions in the da- taflow processing mode is adapted to evaluate at least one designation of one of the registers of an instruction fetched to decide on the use of elements used in the pro- cessing of the instructions , in particular at least one of delays , tightly coupled memory, caches , processing ele- merits , de-arbitrated buses that are operated in an arbi- trated manner in a processing mode di f ferent from the data- flow processing mode .18 . A data processor chip according to the previous claim having circuitry such that for at least some of the in- structions issued to processing elements , the indication of the register evaluated by the instruction f etch / decode / issue circuitry is also evaluated by the processing element the instruction is issued to to select at least one of an input line for operand data or an out- put line for outputting results generated .19 . A data processor chip according to one of claims 14 to 18 wherein the circuitry is adapted to in response to the evaluation of the indication of the register use or not use an element , to selectively bypass an element , in particular a path balancing delay, and / or to change the placement of an instruction in a pro- cessing element , in particular by changing an upcoming or current placement from a standard pattern .27920 . A dataflow processor having a number of coarse-granular cells capable of processing data and interconnection lines for transmitting data between the cells , wherein the coarse-granular cells are adapted to have their functionality of the coarse-granular cells defined by in- structions relating to registers , with the configuration of the interconnection lines being set according to at least one register designation indicat- ed by the instruction .21 . A dataflow processor according to the preceding claim hav- ing circuitry adapted to place a number of ordered instruc- tion in the coarse-granular cells following a predefined placement pattern and are adapted to alter the placement of a current or next instruction to be placed in response to the at least one register designation indicated by the in- struction, in particular by altering the placement relative to the predefined placement pattern .22 . A dataflow processor according to the preceding claim having a placement position pointer and placement position pointer altering circuitry adapted to alter the position the pointer points to in response to the at least one reg- ister designation indicated by the instruction .23 . The data flow processor according to the preceding claim wherein the data flow processor is an FPGA.24 . A data processor according to one of the preceding claims , comprising a signal line to the processing ele- ments for indicating whether the processing elements will be used in a data flow mode , in particular a loop accel- eration mode or in at least one other mode .25 . A data processor chip according to one of the preceding claims 6-24 , wherein the interconnection data lines following a path to a number of processing elements subsequently arranged at the path are provided for a dataflow in one general direction in da- ta flow operation mode ; the chip is also adapted for operation in at least one oth- er mode of operation not having a predominant data flow di- rection and has a bus system for data transmission in the at least one other mode of operation; and wherein data is propagated during data flow mode execu- tion in a data flow counter direction using the buses .26 . A data processor chip according to the preceding claim, wherein access to the buses is arbitrated during operation in the at least one other mode of operation280and access to these busses is fixedly set for data propaga- tion without arbitration during data flow mode execution, in particular bypassing all or some of FI FO registers pro- vided in these buses .27 . A method for operating a chip according to claim 6 or a claim depending therefrom, wherein the plurality of interconnection data lines is used during operation of the device to transmit as data operands for executing instructions having input and output registers and wherein a processing element arranged along the path will receive the data on an interconnection line i f the data corresponds to an operand register it needs , and will ignore incoming data i f it does not corre- spond to a source register needed to execute its in- struction .28 . A method for operating a chip according to claim 27 , wherein the chip is also adapted for operation in at least one other mode of operation not having a predominant data flow direction and has a bus system for data transmission in the at least one other mode of operation, wherein instructions of an instruction set are issued to the processing elements both for the data flow mode and in the other mode of operation and wherein for a given instruction with the same source and target registers , the behavior of the processing ele- ment , in particular the selection of an operand input , in- duced by the instruction in data flow mode is di f ferent from the behavior of the processing .281