Floating-point storage-computation integrated circuit based on large model inference and control method

Abstract

The application relates to the technical field of integrated circuits, and discloses a floating-point storage and calculation integrated circuit based on large model inference and a control method. In the application, an exponent part and a mantissa part of each floating-point operand are written into an exponent search array and a mantissa storage array respectively by a control circuit in a writing stage, and a target exponent sequence is generated with a global maximum exponent as an initial value in a calculation stage; the exponent search array outputs the global maximum exponent in the writing stage, and performs matching search on the exponent part according to a current target exponent and outputs an exponent matching result in the calculation stage; a tail number access control signal with a single effective position corresponding to a current search round is generated by an alignment mode generation circuit; and the mantissa storage array is gated to the corresponding mantissa bit and is accumulated by an accumulation unit round by round. In the application, the mantissa alignment is completed by selecting the storage access position, the mantissa is kept in the original place, and the problem that the mantissa alignment is difficult to balance the operation speed, the circuit area and the energy efficiency is solved.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and in particular to a floating-point in-memory computing circuit and control method based on large model inference. Background Technology

[0002] In-memory computing circuits perform the computation process within the data storage array, reducing data movement between memory and computing units. This makes them suitable for large-scale matrix operations such as artificial intelligence inference. When the data involved in the computation is in floating-point format, each floating-point operand is represented by an exponent and a mantissa. Operands with different exponents have different place values ​​for their mantissas. Therefore, multiple floating-point operands need to be mantissa aligned according to the same place value base before being accumulated.

[0003] In floating-point in-memory computing circuits based on large model inference, the handling of the mantissa alignment stage is related to the operation speed, circuit area and energy efficiency, and related technologies are still struggling to achieve a balance among these three factors. Summary of the Invention

[0004] The purpose of this invention is to provide a floating-point in-memory computing circuit and control method based on large model inference, so as to solve the problem in related technologies that it is difficult to achieve a balance between operation speed, circuit area and energy efficiency in mantissa alignment.

[0005] To address the aforementioned technical problems, this invention provides a floating-point in-memory computing circuit based on large model inference, comprising: a control circuit, used to write the exponent portion of each floating-point operand to an exponent search array and the mantissa portion of each floating-point operand to a mantissa storage array during the write phase; and to generate a target exponent sequence with the global maximum exponent as the initial value during the calculation phase; an exponent search array, connected to the control circuit; used to store the exponent portion of each floating-point operand, outputting the global maximum exponent among all exponent portions during the write phase, and performing a matching search on the exponent portion according to the current target exponent in the target exponent sequence during the calculation phase, and outputting the exponent matching result; an alignment pattern generation circuit, connected to the exponent search array; used to generate a mantissa access control signal according to the exponent matching result; the mantissa access control signal has a single valid position corresponding to the current search round; a mantissa storage array, connected to the control circuit and the alignment pattern generation circuit; used to store the mantissa portion of each floating-point operand, and to select and output the corresponding mantissa bit according to the mantissa access control signal; and an accumulation unit, connected to the mantissa storage array, used to accumulate the mantissa bits output by the mantissa storage array in multiple search rounds.

[0006] This invention also provides a control method for a floating-point in-memory computing circuit based on large model inference, applied to the aforementioned floating-point in-memory computing circuit based on large model inference; the method includes: writing the exponent portion of each floating-point operand to an exponent search array and writing the mantissa portion of each floating-point operand to a mantissa storage array; outputting the global maximum exponent among all exponent portions through the exponent search array; generating a target exponent sequence with the global maximum exponent as the initial value during the calculation phase; performing a matching search on the exponent portion according to the current target exponent in the target exponent sequence through the exponent search array and outputting the exponent matching result; generating a mantissa access control signal according to the exponent matching result through an alignment mode generation circuit, the mantissa access control signal having a single valid position corresponding to the current search round; selecting and outputting the corresponding mantissa bit through the mantissa storage array according to the mantissa access control signal; and accumulating the mantissa bits output by the mantissa storage array in multiple search rounds through an accumulation unit.

[0007] In this invention, by storing the exponent portion of each floating-point operand in an exponent search array, the comparison between each exponent portion and the current target exponent is completed within the array storing the exponents. The target exponent sequence progresses round by round, starting with the global maximum exponent. Each round of search corresponds to a specific bit weight. The alignment mode generation circuit converts the matching result of this round into a mantissa access control signal with only a single valid position. The mantissa storage array selects and outputs the mantissa bit at that position accordingly, ensuring that the mantissa bits read in the same round have the same bit weight. The exponent difference thus reflects the order in which the mantissa bits of each floating-point operand enter the accumulation round. Therefore, mantissa alignment is completed by selecting the storage access position. The mantissa remains stored in its original location during the alignment process. The exponent search and mantissa alignment are connected within the same storage array structure, achieving a balance between computational speed, circuit area, and energy efficiency in the same solution. Attached Figure Description

[0008] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0009] Figure 1 This is a schematic diagram of a serial alignment scheme in the prior art; Figure 2 This is a schematic diagram of a parallel alignment scheme in the prior art; Figure 3 A schematic diagram of the overall architecture of a floating-point in-memory computing circuit based on large model inference provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the writing stage of a floating-point in-memory computing circuit based on large model inference, as provided in an embodiment of the present invention. Figure 5 This is a flowchart illustrating the computation stage of a floating-point in-memory computing circuit based on large model inference, as provided in an embodiment of the present invention. Figure 6 A schematic diagram of the circuit structure of an exponential storage unit provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the alignment pattern generation circuit provided in one embodiment of the present invention; Figure 8 A schematic diagram of the circuit structure of a mantissa storage unit provided in one embodiment of the present invention; Figure 9 This is a timing comparison diagram of the current target index and the access mask positions of each floating-point operand during the calculation stage, provided as an embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0011] In this invention, "electrical connection" refers to the interconnection of two circuit structures via a conductive material, where current or voltage signals are transmitted between the connected structures when the circuit is in operation. "Bit weight" refers to the weight represented by a specific bit in a binary number, and "bit weight order" refers to the arrangement of bits according to their weights from highest to lowest or lowest to highest. "Exponent part" and "mantissa part" refer to the field representing the order of magnitude and the field representing significant digits in floating-point data, respectively. "First logic value" and "second logic value" are used to distinguish between two level states of digital signals and are not used to limit specific voltage values.

[0012] The floating-point in-memory computing circuit and control method based on large model inference provided in this invention are applicable to various arithmetic structures that perform floating-point data alignment and accumulation within a memory array. In an in-memory computing architecture, this circuit can be used as an in-memory computation macro to perform floating-point matrix multiplication and accumulation operations, forming a computation unit together with the interface circuit on the periphery of the array. In a near-memory computing architecture, this circuit can be placed on one side of the memory to perform floating-point data alignment and partial sum accumulation nearby, and then send the accumulation result to the computation core. The computation unit composed of this circuit can be used as a floating-point operation acceleration component in various processors such as graphics processors, neural network processors, tensor processors, and data processors, or as a computation array in an independent acceleration chip or accelerator card for applications requiring large-scale floating-point multiplication and accumulation operations, such as inference and training of artificial intelligence models, scientific computing, and digital signal processing. The above application scenarios are illustrative, and this invention does not limit the application scenarios.

[0013] In the floating-point in-memory computing circuit of this invention, when performing floating-point matrix operations, the exponents of the floating-point operands involved in the accumulation are often different, and the place values ​​represented by each mantissa are also different. Therefore, it is necessary to first adjust the mantissas to the same place value base before entering the accumulation process. See also... Figure 1 In one related technology, mantissa alignment is performed by a serial mantissa aligner on the periphery of the memory array: when the exponent of a floating-point operand is not yet consistent with the global maximum exponent, the exponent increments incrementally, approaching the global maximum exponent step by step, with the corresponding sign bit output step by step, and the mantissa not participating in the effective calculation; after the exponent is consistent with the global maximum exponent, the mantissa is output sequentially as a bit string, with the most significant bit entering the in-memory calculation macro first, and the remaining mantissa bits entering step by step according to their bit weights. See also Figure 2 In another related technology, mantissa alignment is performed by a parallel mantissa aligner on the periphery of the memory array: the peripheral circuit calculates the exponent difference in a single clock cycle, and the mantissa is shifted as a whole to the position corresponding to the global maximum exponent by a barrel shift circuit or an equivalent wide shift circuit according to the exponent difference, and then sent into the in-memory computation macro at once; the parallel mantissa aligner can be set before or after the in-memory computation array.

[0014] While the two alignment methods described above differ in their trade-offs regarding timing and area overhead, they share the same premise: the exponent difference must first be explicitly calculated into a numerical value, and then this value drives a shift of the mantissa data itself. Following this premise, the serial method spreads the exponent difference across the time axis, and the number of frames required for alignment increases with the dispersion of the exponent distribution; the wider the exponent interval, the longer the pre-alignment and transmission delays. The parallel method spreads the exponent difference spatially, and the shift network must cover the maximum possible exponent difference and the entire mantissa bit width; its area and line length increase with the product of these two, and the size of the crossbar switch also increases accordingly.

[0015] Furthermore, in both implementations, alignment is treated as applying a transformation to the data. However, since the mantissa data is already stored in the storage array, any transformation of the data requires reading out the mantissa, sending it to the external storage, transforming it, and then sending it back for calculation. Data movement thus becomes an accompanying overhead of mantissa alignment and simultaneously consumes storage access bandwidth. Since the aforementioned premise that "alignment depends on data movement" is not necessarily required, this invention proposes to transfer the alignment operation to the read access stage of the storage array, replacing data shifting with the selection of the access location.

[0016] See Figure 3 An embodiment of the present invention provides a floating-point in-memory computing circuit based on large model inference, which includes a control circuit, an exponent search array, an alignment mode generation circuit, a mantissa storage array, and an accumulation unit.

[0017] The control circuit is configured to write the exponent portion of each floating-point operand to the exponent search array and the mantissa portion of each floating-point operand to the mantissa storage array during the write phase; and to generate a target exponent sequence with the global maximum exponent as the initial value during the calculation phase.

[0018] The exponent search array is connected to the control circuit and is configured to store the exponent portion of each floating-point operand. During the write phase, it outputs the global maximum exponent among all exponent portions. During the calculation phase, it performs a matching search on the exponent portion based on the current target exponent in the target exponent sequence and outputs the exponent matching result.

[0019] The alignment pattern generation circuit is connected to the exponent search array and is configured to generate a mantissa access control signal based on the exponent matching result. The mantissa access control signal has a single valid position corresponding to the current search round.

[0020] The mantissa storage array is connected to the control circuit and the alignment mode generation circuit, and is configured to store the mantissa portion of each floating-point operand, and to select and output the corresponding mantissa bit according to the mantissa access control signal.

[0021] The accumulation unit is connected to the mantissa storage array and is configured to accumulate the mantissa bits output by the mantissa storage array in multiple search rounds.

[0022] See Figure 3In one example, the periphery of the exponent search array includes a decoder and write word line driver, an interface and write bit line driver, a search line and search mode line controller, a maximum exponent detector, and a match line driver and a sense amplifier. The search line and search mode line controller are configured to apply the mode level corresponding to each exponent bit of the current target exponent and the selected match condition to the array during the calculation phase. The match line driver and sense amplifier are configured to preset the match lines to the start level before the search and read out the level of each match line after the search. The mantissa storage array is arranged in the same area as the alignment mode generation circuit, and its periphery includes a decoder and write word line driver, an interface and write bit line driver, a read bit line driver, and a sense amplifier. A clock generator array is arranged between the match line driver and the mantissa storage array and is configured to convert the sensing results on each match line into the clock signal required by the alignment mode generation circuit. The accumulator unit includes an accumulator and a quantizer electrically connected to the read bit line driver and the sense amplifier.

[0023] Optionally, the floating-point operand is floating-point format data including an exponent part and a mantissa part. The floating-point format data can be, for example, BF16 format data, FP16 format data, or FP32 format data. Other floating-point format data including an exponent part and a mantissa part are also applicable. This invention does not limit the specific type of floating-point format.

[0024] In one example, the floating-point operands use BF16 format data, with an 8-bit exponent and a mantissa consisting of 7 explicit mantissa bits and 1 implicit bit, totaling 8 bits. Each floating-point operand in the mantissa storage array thus corresponds to 8 mantissa bit positions, and each floating-point operand in the alignment mode generation circuit corresponds to 8 latches. When using other floating-point data formats, the number of bits in the exponent and mantissa portions varies with the format, and the number of storage locations occupied by each floating-point operand in the exponent search array and mantissa storage array, as well as the number of latches corresponding to each floating-point operand in the alignment mode generation circuit, are adjusted accordingly.

[0025] Optionally, the exponent storage location in the exponent search array and the mantissa storage location in the mantissa storage array are set according to the same floating-point operand, so that the exponent and mantissa parts of the same floating-point operand have corresponding array positions. This correspondence allows the exponent matching result of a floating-point operand to be passed down along the path corresponding to that floating-point operand and finally applied to the mantissa storage unit of that floating-point operand. There is no need to set up an address mapping circuit to re-address the matching result between the two arrays, which helps to shorten the signal path between the matching result and the mantissa access.

[0026] The operation of the above-described floating-point in-memory computing circuit includes a writing phase and a calculation phase. See also... Figure 4During the write phase, the control circuit writes each floating-point operand involved in the calculation one by one. The exponent portion of each floating-point operand is written to the exponent search array, and the mantissa portion is written to the mantissa storage array. During the write process, the written exponent portions are compared with each other, and the maximum value obtained from the comparison is retained as the global maximum exponent and sent to the control circuit. See also... Figure 5 During the calculation phase, the control circuit generates a target exponent sequence with the global maximum exponent as the initial value. The exponent search array performs a matching search round by round according to this sequence. Within a search round, the exponent search array performs a matching search on each exponent part with the current target exponent as the search word and the matching line indicates whether each floating-point operand has a hit. The hit result is converted into a clock signal for the corresponding floating-point operand by the clock generator array and drives the alignment mode generation circuit to update the mantissa access control signal of the floating-point operand. The mantissa storage array selects the mantissa storage unit at the corresponding bit weight position according to the mantissa access control signal of each floating-point operand and outputs the mantissa bits. The output mantissa bits are weighted according to the absolute bit weight corresponding to this search round and then entered into the accumulation unit for accumulation.

[0027] In some embodiments, the method by which the exponential search array provides the global maximum exponent during the write phase is further described.

[0028] The essential function of the exponential search array during the writing phase is to determine the largest value among the written exponential parts. This determination process can be achieved in various ways, such as comparing the value with the current reserved value during writing, performing a range search with progressively decreasing search words after all writing is completed, or using an independent comparison circuit on the periphery of the array to compare sequentially and provide the result. This embodiment will use the method of comparing the value during writing as an example for explanation.

[0029] The exponent search array is also used to compare the written exponent portions during the write phase and output the maximum value obtained from the comparison as the global maximum exponent to the control circuit.

[0030] In a specific example, see Figure 3 and Figure 4 At the start of the write operation, the value stored in the maximum exponent detector is set to the minimum representable value. For each exponent part written, the exponent search array uses the currently stored value as the search word and performs a matching search according to the greater than or equal to matching condition. If the matching line corresponding to the newly written exponent part gives a hit result, it indicates that the exponent part is not lower than the currently stored value, and the maximum exponent detector then updates the stored value to the exponent part. After all exponent parts have been written, the value stored in the maximum exponent detector is the maximum value among all exponent parts, and is output to the control circuit as the global maximum exponent.

[0031] Therefore, the global maximum exponent is obtained synchronously while each exponent part is written. The acquisition of the exponent extreme value and the data writing are merged into the same process. Compared with the case where all exponent parts are traversed separately before the start of the calculation phase to find the extreme value, it is beneficial to shorten the waiting time between data loading and calculation start.

[0032] Optionally, the global maximum exponent can also be determined by an independent comparison circuit located around the exponent search array, which sequentially compares each exponent portion. The maximum value obtained from the comparison is also retained and output to the control circuit. The above two methods can be implemented individually or in combination.

[0033] In some embodiments, the method of generating the target index sequence and the termination conditions of the matching search are further explained.

[0034] The essential function of the target index sequence in the above circuit is to provide a progressively decreasing comparison benchmark for each search round. This benchmark can be provided in various ways, such as by a decrementing counter providing it round by round with a predetermined step size, by the control circuit providing it round by round after looking up a table based on the global maximum index, or by the target index of the previous search round being provided by a subtraction circuit. This embodiment will use the method of providing it round by round with a decrementing counter as an example for explanation.

[0035] The target index sequence includes multiple target indices, starting with the global maximum index and decreasing in increments of a predetermined step size. The control circuit is also used to end the matching search when the current target index is lower than the preset minimum index or when a predetermined number of search rounds is reached.

[0036] In a specific example, see Figure 5 The predetermined step size is 1. The current target index is decremented by 1 after each search round, so that the absolute weights of two adjacent search rounds differ by one bit, matching the weight relationship of two adjacent mantissa positions in the mantissa storage array. The predetermined step size can also be greater than 1. When the predetermined step size is greater than 1, the access mask of each floating-point operand advances the same number of mantissa positions towards the lower weight direction in each search round. Each search round still reads only one mantissa bit. The mantissa bits located between two adjacent search rounds are not included in the accumulation, and the number of search rounds is reduced accordingly, resulting in a corresponding decrease in output precision.

[0037] In one example, a preset minimum exponent is used to define the lowest absolute bit weight for accumulation. Its value is determined by the required mantissa precision: when the difference between the preset minimum exponent and the global maximum exponent is not less than the number of bits in the mantissa, all mantissa bits of each floating-point operand have a chance to be included in the accumulation; when the preset minimum exponent value is increased, mantissa bits with lower absolute bit weights are truncated, and the number of search rounds is reduced accordingly. The predetermined number of search rounds directly provides an upper bound on the number of rounds, used to limit the duration of the calculation phase when the exponent distribution is unknown.

[0038] Therefore, the duration of the computation phase is determined by the predetermined step size, the preset minimum exponent, and the predetermined number of search rounds, and has a predictable upper bound. Compared to the case where the maximum exponent difference must be calculated first to determine the alignment duration, this is beneficial for decoupling the timing arrangement of the computation phase from the exponent distribution of floating-point operands.

[0039] In a specific example, see Figure 9 The exponents of the three floating-point operands involved in the accumulation are 5, 3 and 5 respectively, the mantissa of each floating-point operand is 8 bits, the global maximum exponent is 5, and the predetermined step size is 1. Figure 9 In this diagram, M7 to M0 represent the positions of each bit weight in the mantissa from the highest to the lowest bit weight. The numbers 1, 2, and 3 within the boxes represent the mantissa bits of the first, second, and third floating-point operands that are accessed and output in this search round, respectively. The dashed lines show the trajectory of the effective positions of each floating-point operand moving towards the lower bit weight in each round. In the first search round, the current target exponent is 5. The two floating-point operands with an exponent of 5 are hit, and their effective positions in the access mask are both located at the highest mantissa bit weight position, each outputting one mantissa bit. The floating-point operand with an exponent of 3 is not hit, and no mantissa bit is output in this round. In the second search round, the current target exponent is 4. The effective positions of the two hit floating-point operands each move one bit towards the lower bit weight. In the third search round, the current target exponent is 3. The floating-point operand with an exponent of 3 is hit for the first time, and its effective position is located at the highest mantissa bit weight position. The effective positions of the two previously hit floating-point operands have moved to the third position from the highest bit weight. At this point, the three have the same absolute place value in the mantissa of the output in this round: the absolute place value of the third mantissa from the highest place value of the floating-point operand with an exponent of 5 is equal to the absolute place value of the highest mantissa of the floating-point operand with an exponent of 3.

[0040] It can be seen that the difference in the exponents of each floating-point operand is converted into the difference in the rounds from which the mantissa begins to accumulate during the above process. Each mantissa read in the same search round has the same absolute bit weight, and the mantissa data itself is always stored in the original mantissa storage unit throughout the entire alignment process.

[0041] In some embodiments, the specific manner in which the exponential search array performs a matching search is further described.

[0042] The essential function of the index search array in the above circuit is to compare each index part with the current target index within the array storing the index parts. This comparison can be achieved in various ways, such as comparing bit by bit within the memory cell using content addressing, directly giving the size relationship using a range comparison structure, or giving a comparison conclusion within the array by judging bit by bit starting from the highest bit weight. This embodiment will use the method of comparing bit by bit within the memory cell and summarizing by the matching line as an example for explanation.

[0043] The index search array includes multiple index storage units and matching lines connected to the multiple index storage units. The matching lines are used to output the index matching result when the current target index and the corresponding index part meet the matching conditions.

[0044] In one example, multiple exponent storage units in the exponent search array are arranged in rows and columns. Each exponent bit of the exponent portion of the same floating-point operand is stored in multiple exponent storage units within the same row. The matching line of that row passes sequentially through each exponent storage unit, starting from the exponent storage unit corresponding to the most significant exponent bit. The comparison results within each exponent storage unit are passed bit-by-bit along the matching line and summarized at the end of the row as the exponent matching result for that row. Each floating-point operand involved in the operation occupies a separate row, and the matching search for each row is performed simultaneously within the same search round.

[0045] Therefore, the comparison results of each exponent part with the current target exponent are given directly in the form of the level on the matching line. A single matching search can cover all the exponent parts written in the array in parallel. Compared with configuring a digital comparator for each floating-point operand on the periphery of the array and comparing them one by one, this helps to keep the search latency basically unchanged as the number of floating-point operands involved in the operation increases.

[0046] In some embodiments, the matching criteria are further described.

[0047] Matching conditions include at least one of the following: equal to, greater than or equal to, and less than or equal to.

[0048] In one example, the computation phase uses a greater than or equal to matching condition. With this condition, once the exponent portion of a floating-point operand is matched in a search round, it remains matched in subsequent rounds because the target exponent continues to decrease. This allows the access mask of the floating-point operand to advance one bit towards the lower weight in each round. With the equal to matching condition, a match only occurs in the round where the target exponent is equal to the matching exponent portion, suitable for situations where only the mantissa corresponding to a specific exponent needs to be read. With the less than or equal to matching condition, a match occurs in rounds where the exponent portion is not higher than the target exponent, suitable for situations where floating-point operands are filtered by the upper bound of the exponent.

[0049] Therefore, the same exponential search array supports three comparison semantics without changing the exponential storage cell structure. The filtering range of the matching search can be selected according to the accumulation requirements, which is beneficial for the same array to undertake both the continuous reading of the mantissa of each floating-point operand and the extreme value determination in the writing stage.

[0050] In some embodiments, the circuit structure of the exponential storage cell is further described.

[0051] The exponent search array also includes search lines connected to multiple exponent storage units. The search lines are used to input the exponent bits corresponding to the current target exponent. The exponent storage unit includes a storage sub-unit for storing the exponent bits and a transmission branch connected between the first and second segments of the matching line. The transmission branch is used to make the first and second segments of the matching line conduct when the exponent bits input to the search line match the exponent bits stored in the storage sub-unit.

[0052] In a specific example, see Figure 6 The storage sub-unit is implemented using a six-transistor static random access memory (SRAM) structure. Its storage node and inverting storage node are connected to the write bit line and inverting write bit line respectively via transmission transistors controlled by the write word line, used to write and hold one exponent bit during the write phase. The search line and inverting search line are connected to one end of two comparator transistors, whose gates are connected to the storage node and inverting storage node respectively. Their common terminal forms the comparator node, whose voltage level reflects whether the exponent bit input on the search line matches the exponent bit held by the storage node. The transmission branch consists of a transmission transistor controlled by the comparator node, connected between the first and second segments of the matching line. When the exponent bits match, the comparator node activates the transmission branch, connecting the first and second segments of the matching line, and the comparison result is passed to the exponent storage unit corresponding to the next exponent bit.

[0053] Therefore, the comparison of the multi-digit index is carried out sequentially along the matching line from the highest digit weight to the lowest digit weight. When all digits are consistent, each segment of the matching line is connected in sequence. The on / off state of the entire matching line is the summary result of the multi-digit comparison in that row. The comparison process does not occupy the logic level outside the row.

[0054] In some embodiments, the handling of exponent storage units when exponent bits are inconsistent is further described.

[0055] The exponent search array also includes a search mode line connected to multiple exponent storage cells. The search mode line is used to provide a mode level corresponding to the matching condition. The exponent storage cell also includes a pull-up / pull-down control branch connected between the first segment of the matching line and the search mode line. The pull-up / pull-down control branch is used to connect the first segment of the matching line to the search mode line when the exponent bit input to the search line is inconsistent with the exponent bit stored in the storage sub-cell.

[0056] In a specific example, see Figure 6 The pull-up / pull-down control branch consists of a pull-up / pull-down transistor controlled by the comparator node, connected between the first segment of the matching line and the search mode line. This pull-up / pull-down transistor has the opposite conductivity type to the transmission transistor in the transmission branch, and their gate terminals are connected to the same comparator node. Therefore, only one of them conducts at the same level of the comparator node: the transmission transistor conducts and the pull-up / pull-down transistor is off when the comparator node is at the level corresponding to agreement; the pull-up / pull-down transistor conducts and the transmission transistor is off when the comparator node is at the level corresponding to disagreement. When the exponent bits of a given bit are disagreed, the comparator node activates the pull-up / pull-down control branch, and the first segment of the matching line is connected to the search mode line. The comparison result for that row is directly given by the mode level of that bit without being passed to the lower-weighted bits. Since the comparison proceeds along the matching line from the highest-weighted bit to the lowest-weighted bit, the comparison result for that row is determined by the first disagreeing exponent bit from the highest-weighted bit, which is consistent with the order of judgment in binary number size comparison.

[0057] In one example, the mode level on the search mode line is given according to the selected matching condition: when using the equal-match condition, the mode level is the level that makes the row a miss, and any inconsistency in any bit results in a miss; when using the greater-than-equal-match condition, the mode level is the inverted value of the exponent bit input on the search line, so that at the first inconsistent exponent bit from the high-weight, if the stored exponent bit is the first logic value and the input exponent bit is the second logic value, the row is still considered a hit; when using the less-than-equal-match condition, the mode level is the exponent bit input on the search line, and the determination relationship is the opposite of the former.

[0058] Therefore, switching between the three matching conditions can be accomplished simply by changing the mode level on the search mode line. The structure of the exponent storage cell and the connection relationship of the array do not change with the matching conditions, which is beneficial for selecting the comparison semantics according to the operation needs on the same array.

[0059] In some embodiments, the specific method by which the alignment mode generation circuit generates the mantissa access control signal is further described.

[0060] The essential function of the alignment mode generation circuit in the above circuit is to convert the exponent matching result into a control signal indicating which mantissa weight position should be read. This conversion can be achieved in various ways, such as using cascaded latches to propagate the valid state and detect state transitions, using a counter in conjunction with a decoder to provide position encoding, or using a ring shift register to directly output a single-hot signal. This embodiment uses the method of cascaded latches in conjunction with state transition detection as an example for explanation.

[0061] The alignment pattern generation circuit includes multiple cascaded latches; the multiple latches correspond to different mantissa bit weight positions in the mantissa storage array, and the multiple latches are used to propagate the valid state according to the exponent matching result.

[0062] In a specific example, see Figure 7 The alignment mode generation circuit is grouped according to floating-point operands. Each floating-point operand corresponds to a cascaded set of latches, a clock generator, and a set of mask generation circuits. The number of latches in this group is the same as the number of mantissa bits in the floating-point operand. Figure 7 Latches #7 to #0 correspond sequentially to the mantissa bit positions from the most significant bit position to the least significant bit position. Mask generation circuits #7 to #0 correspond one-to-one with each latch. The input terminal of the latch corresponding to the most significant bit position is connected to a constant valid state. Each latch is alternately driven by the odd-period clock signal, even-period clock signal, and their inverted signal corresponding to the floating-point operand, causing the valid state to propagate one level towards the lower bit position with each received clock signal. Each latch also has a reset signal connected to it, used to reset the output state of each latch to an invalid state before the calculation phase begins. The output state of each latch constitutes a thermometer code, i.e., a consecutive number of bits from the most significant bit position are valid, and the remaining bits are invalid. The boundary between the valid and invalid states is the mantissa bit position that the floating-point operand should read in the current search round.

[0063] In some embodiments, the method for generating a single valid location is further described.

[0064] The single valid position in the mantissa access control signal can be provided in a variety of ways, such as detecting the boundary between valid and invalid states in the thermometer code, decoding the position code and outputting it, or directly outputting it with a single hot shift register. This embodiment will use the method of detecting changes in the output state of adjacent latches as an example for explanation.

[0065] The alignment mode generation circuit also includes a mask generation circuit, which is used to detect changes in the output state of adjacent latches and generate an access mask with a single valid position based on the changes in the output state; the mantissa access control signal includes the access mask.

[0066] In a specific example, see Figure 7Each mantissa position corresponds to a mask generation circuit. The two inputs of this mask generation circuit are connected to the latch output of the current mantissa position and the latch output of the adjacent lower mantissa position, respectively. The other input of the mask generation circuit corresponding to the least significant mantissa position is connected to a constant invalid state. When the latch output of the current mantissa position is valid and the latch output of the adjacent lower mantissa position is invalid, the mask generation circuit outputs a valid access mask bit; otherwise, it outputs an invalid access mask bit. Since the boundary between valid and invalid states in the thermometer code occurs only once at any given time, each mask generation circuit outputs exactly one valid bit, forming an access mask with a single valid position.

[0067] In one example, see Figure 3 and Figure 7 The clock generator array is also grouped by floating-point operand. Each clock generator for a floating-point operand receives the matching line sensing result of the row containing that operand. When the operand is hit in the current search round, the corresponding clock generator outputs a clock signal, advancing the cascaded latch of that operand by one level, and the valid position of the access mask moves one bit towards the lower bit weight. When the operand is not hit, the corresponding clock generator does not output a clock signal, and the output state of its latch remains unchanged from the access mask. Therefore, starting from the search round in which the exponent part of a floating-point operand is first hit, the valid position of the access mask moves round by round from the highest mantissa bit weight position towards the lower bit weight. The number of bits between the valid position and the highest bit weight position is equal to the difference between the exponent part of the floating-point operand and the current target exponent.

[0068] Therefore, the exponent difference of each floating-point operand is converted into the round difference of the access mask. The positional relationship required for mantissa alignment is handled by the state propagation of the latch chain. There is no need to calculate the exponent difference for each floating-point operand separately, nor is there a need to configure a path for moving the mantissa according to the exponent difference. The mantissa bits output by each floating-point operand in the same search round thus have the same absolute bit weight, which helps to decouple the circuit size required for alignment from the product of the mantissa bit width and the maximum exponent difference.

[0069] In the above implementation, the exponent matching result is directly mapped to the location of the accessed cell in the mantissa storage array: the hit information given by the matching line indicates which floating-point operands were hit in this search round. This information is passed down along the path of each floating-point operand, directly determining which group of mantissa storage cells is accessed; the search round in which the hit occurs indicates the bit weight that should be read now. This information is carried by the state propagation of the latch chain, directly determining which mantissa bit weight position within the group the valid position of the access mask falls on. The above two together determine the selected mantissa storage cell. Therefore, there is no need to set up a decoding circuit or mapping circuit to convert the matching result into a storage address between the exponent matching result and the mantissa access, which helps to shorten the signal path between the result given by the self-matching search and the reading of the mantissa bit.

[0070] In one embodiment of the present invention, the clock generator corresponding to the missed floating-point operand does not generate a clock signal in that search round, and the latch, mask generation circuit, and mantissa read path of that floating-point operand are not activated in that round. This gating method makes the dynamic power consumption vary with the number of floating-point operands actually hit in each round. When the exponent distribution is relatively dispersed and only a small number of floating-point operands are hit in each round, it helps to reduce the overall power consumption of the alignment mode generation circuit and the mantissa read path.

[0071] In some embodiments, the specific method by which the mantissa storage array outputs the mantissa bits according to the mantissa access control signal is further described.

[0072] The essential function of the mantissa storage array in the above circuit is to provide the mantissa bits according to the position indicated by the mantissa access control signal. This process can be implemented in various ways, such as parallel gating by grouping floating-point operands, addressing each bit position by row and column decoding, or providing the mantissa bits by controlling each mantissa storage unit separately with multiple gating lines arranged along the bit weight direction. This embodiment will use the parallel gating by grouping floating-point operands as an example for explanation.

[0073] The mantissa storage array includes multiple mantissa storage units, which are grouped according to each floating-point operand. The mantissa storage units corresponding to each floating-point operand are arranged in order of the bit weight of the mantissa portion. In each search round, the mantissa access control signal selects the mantissa storage unit in the corresponding floating-point operand with the bit weight position corresponding to the current search round in parallel for each floating-point operand indicated by the exponent matching result, and outputs the corresponding mantissa bit.

[0074] In one example, see Figure 8The mantissa storage array employs an orthogonal access structure: write lines and read bit lines are arranged in mutually perpendicular directions. During the write phase, row-selected mantissa bits of the same floating-point operand are written along the write bit lines. During the calculation phase, the mantissa bits are read along the read bit lines. Each mantissa storage cell corresponding to the same floating-point operand is connected to the same read bit line. Each mask bit of the access mask is connected to the mantissa storage cell of the floating-point operand along the bit weight direction by the self-alignment mode generation circuit. Each floating-point operand corresponds to a read bit line, so in the same search round, each floating-point operand indicated by the exponent matching result can simultaneously provide a mantissa bit on its respective read bit line.

[0075] Therefore, the number of mantissa bits read in each search round is the same as the number of floating-point operands hit in that round. The reading of each floating-point operand is parallel and does not affect each other. Compared with the case where the mantissa bits of each floating-point operand are sent into the alignment path in sequence, it is beneficial to ensure that the duration of a single search round does not increase with the increase of the number of floating-point operands involved in the operation.

[0076] Optionally, the mantissa storage array includes independent write paths and read paths. The write path is used to write the mantissa portion during the write phase, and the read path is used to read the selected mantissa bits according to the mantissa access control signal during the calculation phase. When the write path and read path are independent of each other, the bit-by-bit read during the calculation phase does not occupy the write path, and the loading of the mantissa portion and the reading of the mantissa bits can be performed according to their respective timing sequences, which is beneficial to improving the utilization rate of the storage array access bandwidth.

[0077] Optionally, the read path includes a read control terminal, a storage node, and a read bit line. The read bit line is used to generate a discharge state corresponding to the selected mantissa bit when the mantissa access control signal is a first logic value and the storage node is a second logic value. This condition ensures that the read bit line discharges only when the access mask is selected and the stored mantissa bit is a specific logic value. Unselected mantissa storage cells do not generate a current path on the read bit line, which helps reduce dynamic power consumption during bit-by-bit readout.

[0078] In a specific example, see Figure 8The mantissa storage unit consists of a six-transistor static random access memory structure and a three-transistor read logic. The three-transistor read logic includes a read enable transistor controlled by a read enable signal, a mask transistor controlled by the mask bit of the access mask, and a storage node transistor controlled by the storage node. These three transistors are connected in series between the read bit line and ground. The read enable signal is the inverted signal of the precharge control signal. Before the search round begins, the read bit line is preset to a high level by the read bit line driver and the sense amplifier. At this time, the read enable transistor is turned off, and the read path is disconnected from the read bit line. After entering the judgment stage, the read enable transistor is turned on. When the mantissa storage unit is selected by the mask bit and its storage node is at a high level, the mask transistor and the storage node transistor are turned on simultaneously. The read bit line discharges through this series path, and the read bit line driver and the sense amplifier determine that the mantissa bit is at a high level. In other cases, the read bit line remains at the preset level, and the judgment result is a low level. The read enable signal and the read bit line are both arranged along the bit weight direction as column lines that run through each mantissa storage cell, and each mantissa storage cell in the same column shares these two lines.

[0079] In some embodiments, the specific implementation of the accumulation unit is further described.

[0080] In one example, see Figure 3 The accumulation unit includes an accumulator and a quantizer: the accumulator is configured to apply corresponding weights to each mantissa bit read in the current search round according to its absolute bit weight, and accumulate the weighted values ​​of each search round into a partial sum; the quantizer is configured to convert the accumulated result and the global maximum exponent into the target floating-point format output. Since the absolute bit weight corresponding to the first search round is the highest, and the number of bits decreases by the same predetermined step size in each subsequent round, the weight applied by the accumulator to the mantissa bits read in each search round also decreases in the same order, so the mantissa bits read in each round are correctly included in the partial sum according to their absolute bit weight. This weighting occurs in the accumulation path after reading, and only determines the weight of each mantissa bit when included in the partial sum, regardless of the storage position of the mantissa part in the mantissa storage array; the mantissa alignment itself is still completed by the mantissa bit weight position selected in each search round.

[0081] Therefore, the last digits read in multiple search rounds are merged into a fixed-point partial sum corresponding to the global maximum exponent, and then converted into floating-point format output by the quantizer in one go. Compared with the case of restoring to floating-point values ​​in each round and then adding them, this helps to reduce the number of format conversions in the accumulation path.

[0082] In one example, the accumulator is also configured to perform a multiplication operation on the read mantissa bits before weighting and accumulation: another multiplier involved in the multiplication-accumulation operation is provided from the input of the accumulator, each mantissa bit read in the current search round is multiplied by this multiplier, and the resulting product is then weighted according to the absolute bit weight corresponding to the current search round and fed into the accumulator for accumulation round by round. The accumulator thus completes the multiplication-accumulation operation within the same path. The mantissa storage array stores the mantissa part of each floating-point operand itself, not the product. The multiplication operation is completed within the accumulator, and the positional relationship on which the mantissa alignment is based does not change with the introduction of the multiplication operation.

[0083] In one embodiment of the invention, the control circuit can also dynamically determine the timing of ending the matching search based on the output accuracy of the quantizer: when the absolute bit weight corresponding to the remaining search rounds is lower than the lowest bit weight that the quantizer output accuracy can distinguish, the control circuit ends the matching search early. This method enables the number of search rounds to adaptively converge with the target output accuracy, which helps to reduce the matching search and read access overhead caused by invalid rounds in inference scenarios with low target accuracy.

[0084] Optionally, the sign bit of the floating-point operand can be stored together with the mantissa portion in the mantissa storage array and provided through the independent read path corresponding to the floating-point operand. This is used to determine the polarity of the mantissa bit read in each search round when it enters the accumulator. Alternatively, the sign bit can be stored separately by the sign processing circuit outside the array and applied during accumulation. The above two methods can be implemented individually or in combination.

[0085] Optionally, a floating-point operand with a value of zero is stored with the exponent part set to the minimum representable value and all bits of the mantissa part set to invalid values. This floating-point operand is not considered a hit in the early search rounds of the calculation phase. Even if it is considered a hit and the mantissa is read in the round where the current target exponent drops to the minimum value, the read mantissa does not change the accumulated result. Denormalized numbers can also participate in the matching search according to the actual value of their exponent part, and their implicit bits can be set to invalid according to the format.

[0086] Optionally, the rounding and overflow of the accumulated result are handled by the quantizer: when the accumulated result exceeds the range that the target floating-point format can represent, the quantizer limits the output result to the maximum value that the format can represent; when the matching search terminates at a preset minimum exponent, the quantizer provides the output result by rounding to the nearest value or truncating the value. This invention does not limit the specific form of the rounding method.

[0087] Optionally, the index search array can be implemented using other forms of addressable storage structures or range comparison structures, which can perform parallel search and matching judgment between each index part and the current target index within the storage array.

[0088] Optionally, the alignment pattern generation circuit can use other timing control methods or encoding methods to generate the mantissa access control signal, which can provide a control signal with a single valid position based on the exponent matching result.

[0089] Optionally, the quadrature access structure of the mantissa storage array can be implemented through different combinations of word lines, bit lines, or access paths, and can complete the bit-by-bit selection and output of the mantissa bits under the action of the mantissa access control signal.

[0090] This invention also provides a control method for a floating-point in-memory computing circuit based on large model inference. The floating-point in-memory computing circuit described in this invention corresponds to the control method; each component in the circuit corresponds one-to-one with each step in the control method. The preceding descriptions of the specific implementation, optional extensions, and technical effects of each component also apply to the corresponding steps of this control method, and will not be repeated here.

[0091] The control method includes the following steps.

[0092] During the write phase, the exponent portion of each floating-point operand is written to the exponent search array, and the mantissa portion of each floating-point operand is written to the mantissa storage array. The global maximum exponent is output from all exponent portions through the exponent search array.

[0093] During the calculation phase, a target exponent sequence is generated using the global maximum exponent as the initial value. An exponent search array performs a matching search on the exponent portion based on the current target exponent in the target exponent sequence, outputting the exponent matching result. An alignment pattern generation circuit generates a mantissa access control signal based on the exponent matching result; this signal has a single valid position corresponding to the current search round. A mantissa storage array selects and outputs the corresponding mantissa bit based on the mantissa access control signal. An accumulation unit accumulates the mantissa bits output by the mantissa storage array across multiple search rounds.

[0094] like Figure 5 As shown, in the calculation phase, the steps of index matching search are repeated round by round according to the target index sequence. In each round, the current target index decreases by a predetermined step size until the current target index is lower than the preset minimum index or reaches the predetermined number of search rounds. The accumulation operation of the output mantissa by the accumulation unit is performed continuously between rounds. The mantissa read in each round is accumulated after being weighted according to its corresponding absolute position weight.

[0095] The floating-point in-memory computing circuit and control method based on large model inference provided in the above embodiments can extend the mantissa alignment from shifting the mantissa data according to the exponent difference to selecting the mantissa reading position according to the search round. The array storing the exponent part is used to compare each exponent part with the current target exponent. Thus, the exponent search and mantissa alignment are connected within the same storage structure. This decouples the circuit size required for alignment from the product of the mantissa bit width and the maximum exponent difference, giving the calculation stage a predictable upper bound. The mantissa part is kept in the original storage unit during the alignment process, which is beneficial to achieving a balance between operation speed, circuit area and energy efficiency.

[0096] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A floating-point in-memory computing circuit based on large model inference, characterized in that, include: The control circuit is used to write the exponent part of each floating-point operand to the exponent search array and the mantissa part of each floating-point operand to the mantissa storage array during the writing phase; and to generate the target exponent sequence with the global maximum exponent as the initial value during the calculation phase. An exponent search array, connected to the control circuit, is used to store the exponent portions of each floating-point operand. During the writing phase, the global maximum exponent among all the exponent portions is output. During the calculation phase, the exponent portions are matched and searched according to the current target exponent in the target exponent sequence, and the exponent matching result is output. An alignment pattern generation circuit is connected to the exponential search array; Used to generate a mantissa access control signal based on the exponent matching result; The tail number access control signal has a single valid position corresponding to the current search round; A mantissa storage array is connected to the control circuit and the alignment mode generation circuit; Used to store the mantissa portion of each floating-point operand, and to select and output the corresponding mantissa bit according to the mantissa access control signal; An accumulation unit, connected to the mantissa storage array, is used to accumulate the mantissa bits output by the mantissa storage array in multiple search rounds.

2. The floating-point in-memory computing circuit based on large model inference according to claim 1, characterized in that, The index search array includes multiple index storage units and matching lines connected to the multiple index storage units. The matching lines are used to output the index matching result when the current target index and the corresponding index part meet the matching conditions.

3. The floating-point in-memory computing circuit based on large model inference according to claim 2, characterized in that, The exponent search array is also used to compare each of the written exponent portions during the writing phase, and output the maximum value obtained from the comparison as the global maximum exponent to the control circuit.

4. The floating-point in-memory computing circuit based on large model inference according to claim 2, characterized in that, The matching conditions include at least one of the following: equal to matching conditions, greater than or equal to matching conditions, and less than or equal to matching conditions.

5. The floating-point in-memory computing circuit based on large model inference according to claim 2, characterized in that, The exponent search array further includes a search line connected to the plurality of exponent storage units, the search line being used to input the exponent bits of the corresponding bits in the current target exponent; The exponent storage unit includes a storage sub-unit for storing exponent bits and a transmission branch connected between the first and second segments of the matching line. The transmission branch is used to make the first segment of the matching line and the second segment of the matching line conduct when the exponent bit input to the search line matches the exponent bit stored in the storage sub-unit.

6. The floating-point in-memory computing circuit based on large model inference according to claim 5, characterized in that, The exponential search array further includes a search mode line connected to the plurality of exponential storage units, the search mode line being used to provide a mode level corresponding to the matching condition; The index storage unit also includes a pull-up / pull-down control branch connected between the first segment of the matching line and the search mode line; The pull-up / pull-down control branch is used to connect the first segment of the matching line to the search mode line when the exponent bit input to the search line is inconsistent with the exponent bit stored in the storage sub-unit.

7. The floating-point in-memory computing circuit based on large model inference according to claim 1, characterized in that, The target index sequence includes multiple target indices, starting with the global maximum index and decreasing in successive rounds according to a predetermined step size. The control circuit is also used to end the matching search when the current target index is lower than the preset minimum index or when a predetermined number of search rounds is reached.

8. The floating-point in-memory computing circuit based on large model inference according to claim 1, characterized in that, The alignment pattern generation circuit includes multiple cascaded latches; The plurality of latches correspond to different mantissa bit weight positions in the mantissa storage array, and the plurality of latches are used to propagate the valid state based on the exponent matching result.

9. The floating-point in-memory computing circuit based on large model inference according to claim 8, characterized in that, The alignment pattern generation circuit also includes a mask generation circuit. The mask generation circuit is used to detect changes in the output state of adjacent latches and generate an access mask with the single valid position based on the changes in the output state. The mantissa access control signal includes the access mask.

10. A control method for a floating-point in-memory computing circuit based on large model inference, characterized in that, The method is applied to a floating-point in-memory computing circuit based on large-model inference as described in any one of claims 1 to 9; the method includes: During the write phase, the exponent portion of each floating-point operand is written to the exponent search array, and the mantissa portion of each floating-point operand is written to the mantissa storage array; during the write phase, the global maximum exponent among all the exponent portions is output through the exponent search array. During the calculation phase, a target index sequence is generated using the global maximum index as the initial value. During this phase, the index search array performs a matching search on the index portion based on the current target index in the target index sequence, outputting the index matching result. The alignment pattern generation circuit generates a mantissa access control signal based on the index matching result, the mantissa access control signal having a single valid position corresponding to the current search round. The mantissa storage array selects and outputs the corresponding mantissa bit based on the mantissa access control signal. The accumulation unit accumulates the mantissa bits output by the mantissa storage array in multiple search rounds.

Images