Storage circuit, storage device, and electronic apparatus

Abstract

The utility model provides a storage circuit, a storage and calculation device and electronic equipment, and relates to the technical field of semiconductors. Comprising a first storage area and a second storage area. The first storage area includes a first storage packet including a plurality of layers of first storage arrays stacked in a first direction. The first storage area comprises a first storage group, the second storage area comprises a second storage group, the second storage group comprises a plurality of layers of second storage arrays stacked along the first direction, the first storage group and the second storage group are arranged along a second direction, the first storage area is configured to store model parameter data, and the second storage area is configured to store model parameter data. The model parameter data is used for linear calculation of the model, and the second storage area is configured to store intermediate result data obtained through linear calculation. According to the scheme, the storage resources can be more effectively utilized in the AI system, and the operation efficiency of the AI system is improved.

Description

Technical Field

[0001] This application relates to the field of semiconductor technology, and more specifically, to a storage circuit, a memory computing device, and an electronic device. Background Technology

[0002] With the rapid development and application of artificial intelligence (AI) technology, the demand for storage resources in AI systems is increasing daily. In order to obtain more storage resources within a limited chip area, three-dimensional storage circuit structures have been developed, and storage density has been improved. However, the improvement in storage density is still insufficient to meet the hardware resource requirements of AI systems. How to utilize storage resources more effectively has become an urgent problem to be solved. Utility Model Content

[0003] This application provides a storage circuit, a storage computing device, and an electronic device that enable storage resources to be used more effectively in AI systems and improve the operating efficiency of AI systems.

[0004] In a first aspect, a storage circuit is provided, including a first storage region and a second storage region. The first storage region includes a first storage group, which comprises a multi-layer first storage array stacked along a first direction. The second storage region includes a second storage group, which comprises a multi-layer second storage array stacked along the first direction. The first and second storage groups are arranged along a second direction, and the first storage region is configured to store model parameter data used for linear calculations of a model. The second storage region is configured to store intermediate result data obtained from the linear calculations.

[0005] By storing model parameter data and intermediate result data in the same storage circuit through the technical solution of this application embodiment, it is beneficial to improve the inference efficiency or training efficiency of the model, thereby improving the utilization rate of hardware resources and reducing hardware costs.

[0006] In some implementations of the first aspect, the aforementioned model parameter data includes a first type of parameter and a second type of parameter. The first type of parameter is used for matrix multiplication calculations, and the second type of parameter is used to adjust the results of the matrix multiplication calculations. Thus, by storing the second type of parameter in the in-memory computing device, the latency required to call the second type of parameter can be shortened, which is beneficial to improving the computational efficiency of the in-memory computing device.

[0007] In some implementations of the first aspect, the first storage region includes a first sub-region configured to store the first type of parameters; and a second sub-region configured to store the second type of parameters. The first and second sub-regions are arranged along the first direction, and the first sub-region includes a first storage array with a first layer, while the second sub-region includes a first storage array with a second layer, where the first layer is greater than the second layer. This allocation of storage resources for a storage group to the first and second sub-regions reduces the number of storage group switching operations during model execution, thereby reducing call latency. Furthermore, setting the first layer to be greater than the second layer allows for higher storage resource utilization in conjunction with the storage of model parameter data.

[0008] In some implementations of the first aspect, the number of the first layer and the number of the second layer satisfy a set ratio; or, the proportion of the first layer or the second layer in the first storage group satisfies a set proportion; or, the first layer or the second layer has a set value in the first storage group. Thus, the setting of the first layer and the second layer is flexible, ensuring full utilization of the storage resources for the first type of parameters.

[0009] In some implementations of the first aspect, the aforementioned storage circuit further includes a third storage region configured to store configuration information of the storage circuit. This prevents the configuration information of the first storage circuit from interfering with the model data, thereby improving the storage reliability of the first storage circuit.

[0010] Secondly, a storage computing device is provided, comprising: a storage circuit of any one of the first aspects; and a control circuit connected to the storage circuit and configured to control the storage of model parameter data and intermediate result data in the storage circuit. In this way, by using the control circuit to coordinate the storage of model parameter data and intermediate result data in the same storage circuit, it is possible to store both model parameter data and intermediate result data in the same storage circuit. This facilitates the retrieval of intermediate results during model operation, improving the model's inference or training efficiency, thereby increasing the utilization rate of hardware resources and reducing hardware costs.

[0011] In some implementations of the second aspect, the aforementioned model parameter data includes a first type of parameter and a second type of parameter. The first type of parameter is used for multiplication calculations in linear calculations of the model, and the second type of parameter is used to adjust the result of the multiplication calculations. The control circuit is configured to store the first type of parameter in a first sub-region of a first storage region of the storage circuit and to store the second type of parameter in a second sub-region of the first storage region.

[0012] In some implementations of the second aspect, the control circuit is configured to store the first type of parameters in the first sub-region in storage units. This allows the first type of parameters to be stored simultaneously with the data for one scheduling operation, reducing the need to switch storage groups during a single scheduling operation, lowering the latency caused by switching storage groups, and further improving scheduling efficiency.

[0013] In some implementations of the second aspect, a first buffer region is further included for buffering the aforementioned second type of parameters, wherein the storage unit is determined based on the size of the first buffer region. Thus, when scheduling the first type of parameters and the second type of parameters, the control circuit can prevent the expansion of storage groups for scheduling the first type of parameters in a single scheduling operation, thereby satisfying the required amount of data for both types of parameters through as few scheduling operations as possible.

[0014] In some implementations of the second aspect, a second buffer region is further included for buffering the intermediate result data. The control circuit is configured to store the intermediate result data buffered in the second buffer region into the second storage region of the storage circuit when the amount of intermediate result data buffered in the second buffer region reaches a set amount. This reduces the number of programming iterations, improves the model's operating efficiency, and extends the lifespan of the storage circuit.

[0015] In some implementations of the second aspect, the method further includes: a first cache for caching first intermediate result data, which is read by the control circuit from a first space in the second storage area of ​​the storage circuit based on first read address information, wherein the first read address information indicates the first space; and a second cache for caching second intermediate result data, which is prefetched by the control circuit from a second space in the second storage area, wherein the second space is determined based on the first read address information and is adjacent to the first space. Thus, when the control circuit performs calculations using the second intermediate result data, it can directly call the prefetched intermediate result data from the cache area, which is faster than calling data from the storage circuit, thereby shortening the delay caused by waiting for the second intermediate result data and improving the model's operating efficiency.

[0016] In some implementations of the second aspect, the control circuit is further configured to control the calculation of the model, which includes multiple calculations, each including linear and nonlinear calculations. The model parameter data is used for the linear calculations, and the intermediate result data includes multiple intermediate result data from the linear calculations. The in-memory computing device further includes at least one processing circuit configured to perform the nonlinear calculations. In this way, the storage circuit is configured to perform the linear calculations, or the at least one processing circuit is configured to perform the linear calculations. Thus, multiplication calculations can be implemented using in-memory computing technology, further utilizing storage resources to improve computing power or efficiency.

[0017] Thirdly, an electronic device is provided, including the storage device of any one of the second aspects. Attached Figure Description

[0018] Figure 1 A schematic diagram of a memory computing device according to an exemplary embodiment of this application is shown.

[0019] Figure 2 A schematic diagram of yet another memory computing device according to an exemplary embodiment of this application is shown.

[0020] Figure 3 A schematic diagram of a storage circuit according to an exemplary embodiment of this application is shown.

[0021] Figure 4 A schematic cross-sectional view of a storage circuit according to an exemplary embodiment of this application is shown.

[0022] Figure 5 A cross-sectional schematic diagram of another storage circuit according to an exemplary embodiment of this application is shown.

[0023] Figure 6 A cross-sectional schematic diagram of another storage circuit according to an exemplary embodiment of this application is shown.

[0024] Figure 7 A schematic diagram illustrating the programming process of a memory circuit according to an exemplary embodiment of this application is shown.

[0025] Figure 8 A schematic diagram of another memory computing device according to an exemplary embodiment of this application is shown.

[0026] Figure 9 A schematic diagram of an electronic device according to an exemplary embodiment of this application is shown. Detailed Implementation

[0027] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0028] To keep the drawings concise, the figures in this application only schematically show the parts related to the corresponding embodiments, and they do not represent the actual structure of the product. In addition, to make the drawings concise and easy to understand, some figures only schematically show some structures or components, and there may actually be more or fewer identical or similar structures or components.

[0029] The business scenarios described in the embodiments of this application are for illustrative purposes only and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0030] In this application, unless otherwise expressly specified and limited, "connection" includes direct or indirect connection between objects: connected objects may be directly connected through a medium (e.g., wires, traces, etc.), or indirectly connected through other components, or may be an internal connection. "Coupling" includes signal connection between objects, which may be achieved directly through a medium (e.g., wires, traces, etc.), or through other components. "Grounding" includes direct grounding or indirect grounding, with indirect grounding including, for example, grounding through other components.

[0031] In this application, unless otherwise expressly specified and limited, ordinal numbers, such as "first," "second," etc., are used only to distinguish the objects being described and should not be construed as indicating or implying the relative importance or order between the objects being described. Furthermore, ordinal numbers do not represent the quantity of the objects being described. "Multiple" includes two or more, and other quantifiers are similar. "Or," "and / or," etc., are used to describe the relationship between objects, indicating a non-exclusive inclusion. For example, "A and / or B," "A or B" can include: "A alone," "B alone," or "A and B." Similarly, "A, B, and / or C," "A, B, or C" can include: "A alone," "B alone," "C alone," "A and B," "A and C," "B and C," or "A, B, and C." Additionally, the " / " in this application is used to indicate an "or" relationship between preceding and following objects. The meaning of "one or more of A and B" or "at least one of A and B" in this application is the same as the meaning of "A and / or B" or "A or B" above. "One or more of A, B and C" or "at least one of A, B and C" has the same meaning as "A, B and / or C" or "A, B or C" above.

[0032] This application provides a storage computing device that utilizes a control circuit to coordinate the storage of model parameter data and intermediate result data of an AI model (hereinafter referred to as the model) in the same storage circuit. Storing model parameter data and intermediate result data in the same storage circuit improves the model's inference or training efficiency, increases hardware resource utilization, and reduces hardware costs. Furthermore, the control circuit can control the model's operation, such as its computation. This facilitates the coordinated use of storage and computing resources during model operation, comprehensively considering both to improve the adaptability of the model and hardware resources.

[0033] Figure 1 A schematic diagram of a memory computing device according to an exemplary embodiment of this application is shown. Figure 1 As shown, the in-memory computing device 100 includes a storage circuit 110 and a control circuit 120. The storage circuit 110 is configured to store model data, and the control circuit 120 is connected to the storage circuit 110 and configured to control the storage of the model data. The model data includes model parameter data and intermediate result data.

[0034] Based on the above scheme, the model parameter data and intermediate result data of the model data are coordinated by the control circuit to be stored in the same storage circuit. The model parameter data and intermediate result data of the model data can be stored in the same storage circuit, which is beneficial for calling intermediate results during the model operation, improving the inference efficiency or training efficiency of the model, thereby improving the utilization of hardware resources and reducing hardware costs.

[0035] The above control circuits can also control the operation of the model, such as controlling the model's calculations. This helps to coordinate the use of storage and computing resources during model operation, and comprehensively consider storage and computing resources to improve the adaptability of the model and hardware resources.

[0036] Please continue to refer to this. Figure 1 In some embodiments of this application, the storage circuit 110 includes a first storage area 111 and a second storage area 112. The first storage area 111 is used to store model parameter data, and the second storage area 112 is used to store intermediate result data of the model. This regional storage of model parameter data and intermediate result data facilitates partitioned management of different data and supports centralized storage of both model parameter data and intermediate result data. Centralized storage of model parameter data facilitates faster reading of model parameter data or in-memory computation of model parameter data, improving the computational efficiency of the model. Centralized storage of intermediate result data facilitates the cumulative storage of intermediate result data generated during model operation, reducing write control complexity and improving write efficiency. Furthermore, it facilitates subsequent retrieval of historically accumulated intermediate result data, improving the efficiency of retrieving intermediate result data during model operation.

[0037] Intermediate result data is continuously generated during model execution, requiring multiple writes. Storing this data in an independent area helps reduce interference from intermediate result data writes to model parameter data. Model parameter data has a higher reuse rate than intermediate result data; for example, different users' conversations can share model parameter data, while different users have independent intermediate result data. Independent area storage improves storage resource utilization and enhances the reliability of model parameter data storage.

[0038] The first storage area 111 can store model parameter data, which is used for linear calculations of the model. That is, the first storage area 111 can support linear calculations of the model and can be called the model calculation support area (or storage area). The second storage area 112 can store intermediate result data of the model. That is, the second storage area 112 can support the storage of intermediate result data generated during model execution and can be called the model storage support area. The centralized storage of intermediate result data in the second storage area 112 facilitates the retrieval of historically accumulated intermediate result data, improves retrieval efficiency, and thus improves the computational efficiency of the model using intermediate result data for calculations.

[0039] The operation of the model can include various computations, including at least one linear computation and at least one nonlinear computation. Linear computations can include, for example, vector (or matrix) multiplication. Multiplication can be implemented using in-memory computing technology to utilize storage resources and improve computing power or efficiency. For example, in some embodiments of this application, the above multiplication can be implemented through in-memory computation or near-memory computation. For example, in in-memory computation, the storage circuit can perform analog domain multiplication and accumulation calculations based on stored weight data and received input data within the storage circuit to achieve vector multiplication. Similarly, in near-memory computation, the weight data stored in the storage circuit can be read out, and a processing circuit outside the storage circuit performs digital domain multiplication and accumulation calculations to achieve vector multiplication.

[0040] In some embodiments of this application, the result of the multiplication calculation can be further adjusted. For example, the result of the multiplication calculation can be adjusted by a weighting coefficient (G) and a bias coefficient (bias); the weighting coefficient can also be called a scaling coefficient. For example, y = (x·w + b) / g, where y represents the result of the linear calculation, b represents the bias coefficient, g represents the weighting coefficient, w represents the weight vector of the vector (or matrix) multiplication calculation, which can be represented by weight data stored in a storage circuit, and x represents the input vector of the vector (or matrix) multiplication calculation, which can be represented by input data from a storage circuit or input data from a processing circuit. Optionally, the above y = (x·w + b) / g can be replaced by g'·(x·w + b), where g' represents the weighting coefficient, and g' = 1 / g.

[0041] In the above embodiments, when adjusting the result of the multiplication calculation, the bias coefficient can be adjusted first, and then the weight coefficient can be adjusted. In other embodiments, the weight coefficient can be adjusted first, and then the bias coefficient can be adjusted. This application does not limit the order of result adjustment, nor the magnitude of the weight coefficient and the bias coefficient.

[0042] In some embodiments of this application, model parameter data is used for linear calculation of the model, and intermediate result data is obtained through this linear calculation. For example, it can be obtained by multiplication, or by adjusting the result of the multiplication calculation.

[0043] In in-memory computing technology, storage and computing are physically integrated. This physical integration includes, for example, integrating storage and computing components close together through processes such as packaging; integrating processing circuits with processing capabilities within the memory to achieve in-memory processing integration; or implementing computing through storage devices or storing data in computing devices to achieve tight integration of storage and computing. For example, in Figure 1 In the illustrated embodiment, the storage circuit 110 is used to store data; the control circuit 120 is used to control the operating state of the storage circuit 110, such as controlling one or more operations like writing (or programming), reading, erasing, calculating, or sensing the calculation result. For example, in near-memory calculation, the control circuit 120 can control the reading of the weight data stored in the first storage area 111 of the storage circuit 110 and provide the weight data to the integrated processing circuit for calculation. As another example, in in-memory calculation, the control circuit 120 can provide input data to the first storage area 111 of the storage circuit 110 and control the first storage area 111 to perform in-memory calculation based on the stored weight data and the input data; and the control circuit 120 can be used to read or sense the result of the in-memory calculation, and can also provide the result to subsequent circuits, such as the processing circuit, for further processing.

[0044] This application does not limit the type of storage circuit, which may include, but is not limited to, non-volatile memory (NVM) or volatile memory (VM). Volatile memory may include, but is not limited to, static random access memory (SRAM) or dynamic random access memory (DRAM); non-volatile memory may include, but is not limited to, flash memory, resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric memory (FeRAM), or phase change memory (PCM), etc.

[0045] The storage circuit 110 can store weight data in units of storage cells, which can also be called storage units or storage structures. For example, the storage circuit 110 includes a storage cell array (referred to as a storage array), which includes multiple storage cells arranged in an array.

[0046] The storage unit may include a semiconductor device and utilize the conductivity of the semiconductor device, such as electrical conductance or transconductance, to store weight data. For example, the storage unit may include a resistive storage device or a transistor storage device. For example, weight data can be stored by controlling the conductivity of the resistive storage device, or by controlling the transconductance of the transistor storage device. Alternatively, the storage unit may utilize the energy stored in an energy storage element to store weight data, such as the charge stored in a capacitor; this energy storage element may be connected to the semiconductor device, and the stored energy may act on the semiconductor device, causing the semiconductor device to generate a corresponding conductivity.

[0047] In in-memory computation, the storage circuit 110 can perform calculations in groups. For example, the storage array includes multiple computation groups, and each computation group can include multiple storage cells, which output along a common line. The multiple storage cells in a computation group can store multiple weight data, which can be equivalent to a first data vector (or a first data matrix). In programming mode, the weight data is written into the multiple storage cells, equivalent to writing the first data vector (or first data matrix) into the computation group. In computation mode, the storage circuit 110 receives an input signal, which is converted from input data and provided to the multiple storage cells of the computation group, equivalent to a second data vector (or second data matrix). The weight data stored in the storage cells enables the storage cells to have corresponding conduction capabilities. This conduction capability can change the input signal to obtain an output signal. The multiple storage cells in the computation group are connected by a common output line, enabling cumulative output of the output signal and achieving equivalent multiplication of the first and second data vectors. The storage cells in the computation group can be located in the same row or column of the storage array, or in multiple rows or columns.

[0048] As an example, Figure 2 A schematic diagram of yet another memory computing device according to an exemplary embodiment of this application is shown. Figure 2 As shown, the in-memory computing device 200 includes a storage circuit 210 and a control circuit 220. The storage circuit 210 may include multiple storage arrays, such as storage arrays A1, A2, etc. Each storage array may include multiple storage cells. For example, storage array A1 is described as an example; other storage arrays are similar. Figure 2 As shown, the storage array A1 includes multiple storage cells S 11 -S mn Where m is the number of rows in storage array A1, and n is the number of columns in storage array A1. Storage cell S ij It can store weight data W ij Where i∈[1,m], j∈[1,n]. Control circuit 220 can control the memory array A1 to be in a programming state, so as to send signals to memory cells S. ij Write weight data W ij The control circuit 220 can control the storage array A1 to be in a read state, so as to put the storage cell S... ij Stored weight data W ij Read out or read out the result calculated from multiple storage units. Storage unit S ij Its conduction capability can be based on the stored weight data W ij Optionally, the control circuit 220 can control the storage array A1 to be in a computing state. The control circuit 220 can send signals to the storage cell S through the input terminal IN of the storage array A1. ij Provide input signal V iThe input signal V i Acting on storage unit S ij Storage unit S ij The stored weight data W ij This makes the storage unit S ij It has a corresponding conduction capability, under which current flows to or into the output terminal OUT. Multiple memory cells (e.g., S...) 1j -S mj The output terminals of the memory can be collinear. According to Kirchhoff's laws, the current generated by multiple memory cells accumulates to obtain the output signal I. j Satisfy the following formula:

[0049]

[0050] As can be seen, a storage array can be used to perform multiplication and accumulation calculations, and the result can be output at the output terminal. The sensing output of this result can be understood as reading the calculation result. The multiple storage units with collinear outputs can be called a calculation group.

[0051] The memory computing device 200 may further include a readout circuit 230 for reading data stored in the memory circuit 210 and / or for sensing and outputting the results of the above calculations. The readout circuit 230 may include a first conversion circuit 231 and a second conversion circuit 232. The first conversion circuit 231 performs a first conversion on the output signal of the memory circuit 210 (e.g., readout current), for example, converting the current signal into a voltage signal; the second conversion circuit 232 performs a second conversion on the output of the first conversion, for example, converting the analog signal into a digital signal. The first conversion circuit 231 may also be called a sampling circuit; for example, the sampling circuit may be used to sample the current signal output by the memory circuit as a voltage signal. The second conversion circuit 232 may also be called a decision circuit; for example, the decision circuit may be used to convert the waveform parameters of the analog signal (e.g., voltage signal) into digital signals, for example, converting the amplitude, pulse width, or area of ​​the analog signal into digital signals through decision-making. This application does not limit the accuracy of the conversion or decision; for example, it may include 1-bit or multi-bit conversion accuracy. The control circuit 220 may control the operation of the readout circuit 230.

[0052] Furthermore, different memory arrays can reuse the first conversion circuit 231 and / or the second conversion circuit 232; or, in the same memory array, different output terminals can reuse the sub-circuits in the first conversion circuit 231 and / or the second conversion circuit 232 to reduce the hardware cost of the readout circuit and reduce its footprint on the chip.

[0053] Figure 2 This is just an example illustrating how storage cells are connected in a storage array, except... Figure 2 Besides the connection method shown, other connection methods can also be used. For example, the input terminals of the memory cells can be connected in columns along a common line, and the output terminals of the memory cells can be connected in rows along a common line. Furthermore, the input terminals of the memory cells can include the gate of a transistor memory device, or the input terminals of the memory cells can include the source or drain of a transistor memory device; this application does not limit the specific type of memory cell. This application also does not limit the type of memory cell; for example, the memory cell can include, but is not limited to, transistors, memristors, magnetic tunnel junctions (MTJs), or phase-change structures. This application also does not limit the type of transistor, including, for example, metal-oxide-semiconductor field-effect transistors (MOSFETs), floating-gate transistors (FGTs), ferroelectric field-effect transistors (FeFETs), and thin-film transistors. Optionally, the memory cell may include multiple transistors; for example, the memory cell may include a first transistor and a second transistor, wherein the gate of the first transistor (which may be referred to as a "read transistor" or "read tube") and the source or drain of the second transistor (which may be referred to as a "write transistor" or "write tube") are connected, and the charge stored at the gate of the first transistor can be used to characterize weight data. Optionally, the gate of the first transistor may also be connected to a capacitor to increase the stability and duration of the stored charge.

[0054] Storage arrays can be stacked to realize three-dimensional storage circuitry. For example, in some embodiments of this application, three-dimensional storage circuitry is used to store model parameter data and intermediate result data in separate regions. Figure 3 A schematic diagram of a storage circuit according to an exemplary embodiment of this application is shown. Figure 3 As shown, the three-dimensional spatial directions, including the X, Y, and Z directions, are used as examples. To facilitate the management of storage resources, the storage circuit 300 can be divided into multiple storage sets, for example, storage sets P1 to P2. s Where s represents the number of storage sets and is a positive integer greater than 1; for example, storage sets P1 to P2... s It can be arranged along the X direction. Storage set P u It can be further divided, for example, into multiple storage groups G. u1 -G ug Where u∈[1,s], g represents the number of storage groups in a storage set and is a positive integer greater than 1; for example, storage groups G u1 -Gug It can be arranged along the Y direction. Store group G. uv It can be further divided, for example, including multiple storage blocks B. uv1 -B uvt Where v∈[1,g], t represents the number of storage blocks in a storage group and is a positive integer greater than 1; for example, storage block B uv1 -B uvt It can be arranged along the Y direction. Store group G. uv This includes multi-layered storage arrays Auv1-Auvw stacked along the Z direction, where w represents the number of layers in the storage array and is a positive integer greater than 1. Storage array Auvr is, for example, a two-dimensional storage array where r∈[1,w]. uvr It can include storage cells arranged in an array along the X and Y directions.

[0055] In the above embodiments, the X, Y, and Z directions are merely examples. In other embodiments, other coordinate systems may be used to identify the three-dimensional storage circuit. In some embodiments of this application, the stacking direction may be referred to as the first direction, which is, for example, a direction perpendicular to the chip substrate on which the storage circuit is located. For example, it may be the Z direction in the above embodiments. The second and third directions are used, for example, to identify a two-dimensional plane perpendicular to the stacking direction, such as a plane parallel to the chip substrate on which the storage circuit is located. For example, the second direction may include the Y direction in the above embodiments, and the third direction may include the X direction in the above embodiments; or, the second direction may include the X direction in the above embodiments, and the third direction may include the Y direction in the above embodiments.

[0056] In some embodiments of this application, the storage circuitry includes, but is not limited to, NAND flash memory; the storage set is referred to as a Plane; the storage group is referred to as a Bank; and the storage block is referred to as a Block. This application is not limited thereto, and storage circuits with different three-dimensional structures may have different naming, diversity, or grouping methods.

[0057] In some embodiments of this application, memory cells at the same horizontal position (which may be referred to as a string of memory cells) within different layers of memory arrays can share input lines or input terminals. This simplifies the routing of the memory circuit. For example, the string of memory cells can be connected in series and coupled to the input signal via a transistor at the top or bottom. This transistor may be referred to as a switching transistor, and optionally, the gate of the switching transistor may be coupled to the input signal. In some embodiments of this application, a single layer of memory array includes multiple computing groups, and computing groups at the same position in a second or third direction of different layers of memory arrays can share output lines or output terminals. The selection of memory cells can be implemented in the stacking direction via control lines (e.g., word lines).

[0058] In some embodiments of this application, the first storage region 111 and the second storage region 112 are statically partitioned or dynamically determined within the storage circuit 110. For example, the positions of the first storage region 111 and the second storage region 112 within the storage circuit 110 are statically partitioned or dynamically determined; and / or, the sizes of the first storage region 111 and the second storage region 112 within the storage circuit 110 are statically partitioned or dynamically determined. In the static partitioning scheme, the size and / or position of the first and second storage regions are fixed, and the control logic of the control circuit 120 on the storage circuit 110 is simple. In the dynamic determination scheme, the size and / or position of the first and second storage regions can change dynamically, which is beneficial for the control circuit 120 to flexibly utilize storage resources, making the utilization of storage resources more flexible and further improving the utilization rate of storage resources when adapting to different business scenarios.

[0059] In some embodiments of this application, the control circuit 120 can set the size and / or position of the first storage area 111 and the second storage area 112 of the storage circuit 110 based on the model. Different models may have different storage requirements for model parameter data and intermediate result data. When the model applied by the in-memory computing device is switched, the control circuit 120 can adjust the position and / or size of the first and second storage areas based on the model's storage requirements, so that the in-memory computing device can adapt to different models with higher storage resource utilization. For example, by statistically analyzing the data of different models, it is found that the proportion of model parameter data and intermediate result data may be different in different models. By setting the size and / or position of the first and second storage areas based on the model applied by the in-memory computing device, when the model applied by the in-memory computing device changes, the size and / or position of the first and second storage areas can be reset based on the updated model, so that the in-memory computing device can adapt to different models with higher storage resource utilization.

[0060] In some embodiments of this application, the control circuit 120 can dynamically adjust the size and / or position of the first storage region and the second storage region in the storage circuit based on the usage of storage resources. For example, in the initial configuration of the storage circuit, an initial first storage region is configured. During model operation, unused storage resources in the first storage region can be dynamically allocated to the second storage region to store intermediate result data. As another example, at least two storage resources of the storage circuit can be used for model data storage. Each storage resource may include a first storage region and a second storage region. Taking two storage resources as an example, the first storage region in the second storage resource is used to redundantly back up the model parameter data in the first storage region of the first storage resource to improve the reliability of the model parameter data. During use, the second storage region in the second storage resource and the second storage region in the first storage resource can be combined to store intermediate result data. Figure 3 As an example, suppose the first storage area of ​​a portion of the storage sets P1 to P4 in the storage circuit 300 is used to store model parameter data. The model parameter data of storage sets P1 to P4 is backed up using the first storage areas of other storage sets P5 to P8. The second storage areas of storage sets P1 to P4 and P5 to P8 can be merged to store intermediate result data. In other embodiments, the model parameter data can be backed up multiple times, and the second storage areas of these multiple storage resources can be merged to store intermediate result data. Optionally, the first storage area of ​​the storage resources used for backup can be used to store intermediate result data. In this way, the size and location of the second storage area can be dynamically adjusted according to the resource requirements of the intermediate result data, balancing the reliability of the model parameter data and the resource requirements of the intermediate result data, as well as balancing the utilization rate of storage resources and the requirements of computational reliability.

[0061] In some embodiments of this application, the first storage region can be larger than, smaller than, or equal to the second storage region; that is, the capacity of the first storage region can be greater than, smaller than, or equal to the capacity of the second storage region. The capacity of a storage region can also be referred to as the size of the storage region, storage upper limit, storage scale, storage range, or storage quota, etc. For example, the number of storage groups included in the first storage region is greater than, smaller than, or equal to the number of storage groups included in the second storage region. Another example is that the number of storage blocks included in the first storage region is greater than, smaller than, or equal to the number of storage blocks included in the second storage region. Yet another example is that the number of storage arrays included in the first storage region is greater than, smaller than, or equal to the number of storage arrays included in the second storage region. For a given model, the range of model parameter data stored by the storage circuit is relatively fixed. The amount of intermediate result data can gradually accumulate. For example, the amount of intermediate result data is related to the number of users; as the number of users increases, the amount of intermediate result data can increase. Setting the first storage region to be larger than the second storage region can better adapt to the resource requirements of model parameter data and intermediate result data when the number of users is small, such as when applied to edge devices, thus making more efficient use of storage resources. Setting the first storage area to be smaller than the second storage area can meet the needs of more users when the number of users is large, such as when applied to servers or cloud devices, thereby increasing the user capacity of the storage circuit; for example, when applied to end-side devices, it can meet user needs when the end-side user demand is large (e.g., a large number of sessions).

[0062] In some embodiments of this application, the amount of intermediate result data can be related to the number of users. As the number of users increases, the amount of intermediate result data can increase. Therefore, when dynamically or statically allocating the second storage area, redundant space can be reserved to facilitate the expansion of the second storage area.

[0063] In some embodiments of this application, the model parameter data includes a first type of parameter and a second type of parameter. The first type of parameter is used for multiplication calculations in linear computation, and the second type of parameter is used to adjust the result of the multiplication calculation. For example, the first type of parameter may include the weight data described in the above embodiments, and the second type of parameter may include the weight coefficient (G) and bias coefficient (bias) described in the above embodiments.

[0064] In in-memory computation, a first type of parameter is stored in the first storage area. Multiplication in the analog domain is performed in the first storage area based on the first type of parameter to obtain the result of the multiplication. This result and the second type of parameter can be read out to adjust the result using the second type of parameter. In near-memory computation, the first type of parameter and the second type of parameter in the first storage area are read out. Multiplication in the digital domain is performed using the first type of parameter to obtain the result of the multiplication. The result is then adjusted using the second type of parameter.

[0065] In the above embodiments, storing the first type of parameters and the second type of parameters in the same storage area is beneficial for comprehensively considering the storage and computing resources of linear computation, improving resource utilization, reducing the switching latency between different storage areas, reducing scheduling time, and improving computational efficiency. For example, in in-memory computation, multiplication can be performed directly using storage resources based on the first type of parameters, and the result of the multiplication and the second type of parameters can be read synchronously to adjust the result of the multiplication and obtain the result of the linear computation, reducing scheduling time and improving computational efficiency. As another example, in near-memory computation, the first type of parameters and the second type of parameters can be scheduled synchronously for linear computation, reducing scheduling time and improving computational efficiency.

[0066] In some embodiments of this application, the first storage area may include a first sub-region and a second sub-region, which are used to store first type of parameters and second type of parameters, respectively.

[0067] by Figure 3Taking the storage circuit shown as an example, the first storage area can include multiple storage groups. Each storage group can include multiple layers of storage arrays. Within each storage group, a portion of the storage array layers can be allocated to the first sub-region, and a portion to the second sub-region. The number of layers of storage arrays allocated to the first sub-region is greater than the number of layers allocated to the second sub-region. Statistical analysis of various model data reveals that the amount of data for the first type of parameters is significantly greater than that for the second type of parameters. For example, the amount of data for the first type of parameters can differ from that for the second type of parameters by three orders of magnitude. Allocating the storage resources of the larger storage array layers within a storage group to the first type of parameters and the storage resources of the smaller storage array layers to the second type of parameters can achieve a higher storage resource utilization rate in conjunction with the storage of the model parameter data. Furthermore, in linear computation, the first and second types of parameters of the model parameter data can be called from the same storage group. The first type of parameters can be used to perform multiplication calculations to obtain the result of the multiplication calculation, and the second type of parameters can be used to adjust the result, thereby reducing the call latency caused by storage group switching in linear computation. Alternatively, in linear computation, multiplication calculations can be performed based on the stored first type of parameters using the storage group, the result of the multiplication calculation can be read, and the second type of parameters stored in the same storage group can be called to adjust the result, thereby reducing the call latency caused by storage group switching.

[0068] For example, Figure 4 A schematic cross-sectional view of a memory circuit according to an exemplary embodiment of this application is shown. Figure 4 As shown, the storage circuit 400 includes a first storage area 410 and a second storage area 420; the first storage area 410 is used to store model parameter data, and the second storage area 420 is used to store intermediate result data. The first storage area 410 includes storage blocks G. 11 ~G 1N Storage group G 1i (For clarity, this can be referred to as the first storage group) includes a multi-layered storage array A1 (for clarity, this can be referred to as the first storage array) stacked along a first direction (e.g., the Z direction), where i ∈ [1, N], and N represents the number of first storage groups in the first storage region 410, and is a positive integer greater than 1. The second storage region 420 includes storage groups G. 21 ~G 2M Storage group G 2j (For clarity, this can be referred to as the second storage group) includes a multi-layered storage array A2 (for clarity, this can be referred to as the second storage array) stacked along the first direction, where j∈[1,M], M represents the number of second storage groups in the second storage region 420, and is a positive integer greater than or equal to 1. Storage group G 1i and storage group G 2jArranged along a second direction (e.g., the Y direction).

[0069] Figure 4 The diagram shows only a cross-sectional view along the Z and Y directions. The memory circuit can extend along the X direction, meaning that memory arrays A1 and A2 can be arranged in arrays along the Y and X directions and stacked along the Z direction to form a multi-layer stacked memory array.

[0070] In some embodiments of this application, the first storage region and the second storage region may belong to the same storage die or different storage dies. For example, the storage circuit 110 may include multiple storage dies (bare dies), and the first storage region 111 and the second storage region 112 may be located on the same storage die or different storage dies. For instance, when the storage circuit needs to store a large amount of user data, such as on the server side, the first storage region 111 and the second storage region 112 can be located on different storage dies to achieve efficient utilization of more storage resources through the control circuit 120, thereby serving the needs of more users.

[0071] In some embodiments of this application, the first storage region 410 and the second storage region 420 may belong to the same storage set, for example, storage set P. u Optionally, the multiple memory sets of the storage circuit 400 can be configured in the same or similar ways. The size and location of the first and second memory regions included in different memory sets can be the same or different. Within the same memory set, the first memory region 410 and the second memory region 420 can be arranged along a second direction (e.g., the Y direction).

[0072] In some embodiments of this application, the first storage region and the second storage region may belong to different storage sets; for example, the first storage region and the second storage region may be arranged along a third direction (e.g., the X direction). Figure 5 A cross-sectional schematic diagram of another storage circuit according to an exemplary embodiment of this application is shown. Figure 5 As shown, the storage circuit 500 includes a first storage area 510 and a second storage area 520; the first storage area 510 is used to store model parameter data, and the second storage area 520 is used to store intermediate result data. The first storage area 510 includes storage blocks G. 11 ~G 1N Storage group G 1i The storage array A1 comprises multiple layers stacked along a first direction (e.g., the Z direction), where i ∈ [1, N], and N represents the number of storage packets in the first storage region 510 and is a positive integer greater than 1. The second storage region 520 comprises storage packets G. 21 ~G 2M Storage group G 2jThis includes a multi-layered storage array A2 stacked along the first direction, where j∈[1,M], and M represents the number of storage packets in the second storage region 520, and is a positive integer greater than or equal to 1. Storage packet G 1i and storage group G 2j Arranged along a third direction (e.g., the X direction). For example, storage group G... 1i Located in storage set P1, storage group G 2j Located in storage set P2, storage sets P1 and P2 are arranged in the X direction. Figure 5 In this example, storage sets P1 and P2 are for illustrative purposes only, and storage group G... 1i and storage group G 2j They can be located in adjacent or non-adjacent storage sets.

[0073] In some embodiments of this application, a first storage region includes multiple storage groups, and each storage group includes a multi-layer storage array stacked along a first direction. A first sub-region includes a first number of storage arrays of the multiple storage groups, and a second sub-region includes a second number of storage arrays of the multiple storage groups, wherein the number of first layers is greater than the number of second layers.

[0074] For example, please continue to refer to Figure 4 The first storage region 410 includes a multi-layer storage array A1 stacked along the Z direction, and includes a first sub-region 411 and a second sub-region 412. The first sub-region 411 may include the multi-layer storage array A1, and the second sub-region 412 may include the multi-layer storage array A1. Within the same storage group, the number of layers of storage array A1 included in the first sub-region 411 is greater than the number of layers of storage array A1 included in the second sub-region 412. For further details, please refer to... Figure 5 The first storage region 510 includes a multi-layer storage array A1 stacked along the Z direction, and includes a first sub-region 511 and a second sub-region 512. The first sub-region 511 may include the multi-layer storage array A1, and the second sub-region 512 may include the multi-layer storage array A1. In the same storage group, the number of layers of storage array A1 included in the first sub-region 511 is greater than the number of layers of storage array A1 included in the second sub-region 512.

[0075] Optionally, the number of storage array layers included in the first sub-region of different storage groups can be the same, and the number of storage array layers included in the second sub-region can be the same, which helps to simplify the management of storage resources.

[0076] In the above embodiments of this application, dividing the storage resources of a storage group into a first sub-region and a second sub-region can reduce the number of storage group switching during model operation, thereby reducing the call latency caused by it.

[0077] In some other embodiments, the first sub-region and the second sub-region may be divided into different storage groups. For example, Figure 6 A cross-sectional schematic diagram of another storage circuit according to an exemplary embodiment of this application is shown. Figure 6 As shown, the storage circuit 600 includes a first storage area 610 and a second storage area 620; the first storage area 610 is used to store model parameter data, and the second storage area 620 is used to store intermediate result data. The first storage area 610 includes a first sub-area 611 and a second sub-area 612, and the first sub-area 611 and the second sub-area 612 can be located in different storage groups. Figure 6 Taking the second sub-region 612 as an example, which includes one storage group, other embodiments may include more than one storage group. Similar to the above. Figure 4 In the illustrated embodiment, the first storage region 610 and the second storage region 620 may belong to the same storage set, for example, storage set P. u Multiple memory sets of the memory circuit 600 can be configured in the same or similar ways. The size and location of the first memory region and the second memory region included in different memory sets can be the same or different. Within the same memory set, the first memory region 610 and the second memory region 620 can be arranged along a second direction (e.g., the Y direction), and the first sub-region 611 and the second sub-region 612 can be arranged along a second direction (e.g., the Y direction).

[0078] Among the above-mentioned various layout methods of storage areas, Figure 4 The illustrated embodiment can provide greater bandwidth for calling model parameter data and intermediate result data, and has lower scheduling latency. Figure 4 In this paper, the position of the storage array of the second sub-region in the Z direction in the storage group is only an example. It can be set at any position, such as the top, bottom, or middle, or it can be set non-contiguously in the storage group. This application does not impose any restrictions.

[0079] In some embodiments of this application, the size of the first sub-region and the size of the second sub-region satisfy a predetermined ratio. Alternatively, within the first storage group, the number of first layers of the storage array used for the first sub-region and the number of second layers of the storage array used for the second sub-region satisfy a predetermined ratio. Alternatively, the proportion of the first or second layers in the first storage group satisfies a predetermined proportion. Alternatively, the number of first or second layers has a predetermined value in the first storage group.

[0080] The ratio can be dynamically adjusted based on the model or set statically. In some embodiments, a statically set ratio can meet the storage resource requirements of the first and second types of parameters in different models. Statistics show that the data volume of the second type of parameters is much smaller than that of the first type of parameters in different models. Therefore, a ratio can be set that meets the needs of different models, thus reducing the complexity of resource reallocation in the first storage area. In some embodiments, dynamically adjusting the ratio based on the model can further improve resource utilization. For example, the above-mentioned ratio can be greater than or equal to 50:1, which can adapt to the needs of most models and balance the utilization of storage resources for the first type of parameters. For example, in some embodiments of this application, redundant resources can be set for the second type of parameters; this ratio can be less than 50:1 and greater than or equal to 20:1.

[0081] Similarly, the above-mentioned percentages or values ​​can be dynamically adjusted based on the model or set statically. In some embodiments of this application, a smaller number of storage array layers in the first storage group can be allocated to the second sub-region. For example, a storage array with a set number of layers in the first storage group can be allocated to the second sub-region. This number of layers can be less than or equal to 15 layers, or less than or equal to 10 layers, or less than or equal to 5 layers, etc. This simplifies the configuration of storage resources.

[0082] In some embodiments of this application, the control circuit 120 can dynamically adjust or statically preset the above-mentioned set ratio, set percentage or set value based on the model.

[0083] During the reading of the first type of parameters and / or the second type of parameters, the control circuit 120 may use the same or different reading methods for the first and second types of parameters. In some possible embodiments, the control circuit 120 may use the same reading method to read the first and second types of parameters, thereby simplifying the control logic. In other possible embodiments, the control circuit 120 may use different reading methods to read the first and second types of parameters. For example, the control circuit 120 may use a first reading method to read the first type of parameters and a second reading method to read the second type of parameters, wherein the reading rate of the first reading method is greater than that of the second reading method, and the reliability of the second reading method is higher than that of the first reading method. In the model parameter data, the amount of data for the first type of parameters is larger than that for the second type of parameters. On the one hand, linear calculations have a higher tolerance for errors in the first type of parameters; on the other hand, the reading rate of the first type of parameters has a greater impact on the efficiency of the calculation. Thus, the control circuit can use the first reading method to read the first type of parameters and the second reading method to read the second type of parameters, which can improve the calculation efficiency while still meeting the reliability requirements of the model. For example, in near-memory computation, the control circuit 120 can use a first reading method with a faster reading rate to read the first type of parameters, and a second reading method with higher accuracy to read the second type of parameters, thereby taking into account both the computational efficiency and reliability requirements of the model.

[0084] In some embodiments of this application, the first type of parameters can be stored according to a preset size, which can be referred to as a storage unit. For example, the control circuit 120 stores the first type of parameters in a first sub-region according to the storage unit. The setting of the storage unit can be related to the data size scheduled in a linear calculation by the control circuit 120. By setting the storage unit, the first type of parameters can store the data of a linear calculation, reducing the need to switch storage groups in a linear calculation, reducing the latency caused by switching storage groups, and further improving the running efficiency of the model.

[0085] In some embodiments of this application, when the portion of the current storage group used for the first sub-region is insufficient to store one storage unit of the first type of parameter, or when the remaining space of the current storage group used for the first sub-region is insufficient to store one storage unit, the first type of parameter to be stored is stored in the next storage group. For example, the first storage region includes multiple storage groups, including a first storage group and a second storage group. The control circuit 120 is configured to store the first portion of the first type of parameter in the first storage group on a storage unit basis. When the remaining space of the first storage group used for the first sub-region is insufficient to store one storage unit, the control circuit 120 is also configured to store the second portion of the first type of parameter in the space of the second storage group used for the first sub-region on a storage unit basis. The first type of parameter may also include more portions and be stored in more storage groups; this application does not limit this.

[0086] The storage circuit calls the first type of parameters at the granularity of storage units or integer multiples of storage units. In a single call, it can reduce the scheduling across storage groups, thereby reducing the latency of cross-group data reading and further improving the running efficiency of the model.

[0087] For example, Figure 7 A schematic diagram illustrating a programming process for a memory circuit according to an exemplary embodiment of this application is shown. This programming process is used to write a first type of parameter. For example... Figure 7 As shown, storage group G1 and storage group G2 are used as examples; programming for more storage groups is similar. Additionally, the example of a storage unit containing the data volume of three layers of storage arrays is merely an example and not intended to limit the size of the storage unit; a storage unit can have more or fewer layers of storage arrays. If the remaining space in storage group G1 for the first sub-region is insufficient for three layers of storage arrays, the control circuit 120 can skip the remaining space in storage group G1 for the first sub-region (e.g., ...). Figure 7 The blank box in the middle represents the storage array, which stores the first type of parameter to be stored in the storage space of storage group G2 for the first sub-region.

[0088] In some possible embodiments, the size of the storage unit may be pre-configured.

[0089] In some possible embodiments, the size of the storage unit can be determined based on the size of the buffer region within the memory device used to buffer (or cache) the second type of parameters. For example, Figure 8 A schematic diagram of another memory computing device according to an exemplary embodiment of this application is shown. Figure 8As shown, the in-memory computing device 800 includes a storage circuit 810 and a control circuit 820. The descriptions of the storage circuit 810 and the control circuit 820 can be found in any of the above embodiments and will not be repeated here. The in-memory computing device 800 may also include a first buffer region for buffering second-type parameters. The storage unit is determined based on the size of the first buffer region. The size of the first buffer region can be used to determine the amount of second-type parameter data that can be obtained in one scheduling operation. In the linear calculation of the model, the amount of first-type and second-type parameters participating in one linear calculation can have a certain proportional range. The size of the storage unit is determined based on the size of the first buffer region, which can associate storage with scheduling, ensuring that the amount of first-type parameter data stored within a storage group is always an integer multiple of the storage unit. The storage unit is determined according to the size of the first buffer region, ensuring that the amount of first-type parameter data included in the storage unit and the amount of second-type parameter data in the first buffer region satisfy the above proportional range. Thus, when the control circuit 820 schedules the first-type and second-type parameters, it can prevent the scheduling of first-type parameters across storage groups in one scheduling operation, thereby satisfying the amount of first-type and second-type parameter data required for linear calculation with as few scheduling operations as possible. For example, in linear calculations, the control circuit can read the first type of parameters based on storage units, and the second type of parameters that participate in the calculation together with the read first type of parameters can be buffered in the first buffer area, so that the read first type of parameters and the buffered second type of parameters in the first buffer area meet the above ratio range. In this way, the buffering capacity of the first buffer area can be fully utilized, and it can help reduce the number of times the first type of parameters are scheduled across storage groups, reduce the latency of cross-group reading in the calculation, and improve the running efficiency of the model.

[0090] Figure 4 or Figure 5 The allocation method of the first and second sub-regions shown in the embodiment, combined with the above programming and scheduling methods, can better utilize the surplus layer in the first direction. This surplus layer can be used to store the second type of parameters, greatly improving the utilization rate of storage resources. For example, when storing the first type of parameters by storage unit does not fill the storage group, the remaining storage space (i.e., the surplus layer) can be used to store the second type of parameters.

[0091] Please continue to refer to this. Figure 8In some embodiments of this application, the in-memory computing device 800 may further include a second buffer area for buffering intermediate result data. When the intermediate result data buffered in the second buffer area reaches a set amount, the control circuit 820 stores the intermediate result data buffered in the second buffer area into the second storage area. During computation (e.g., during autoregressive inference), intermediate result data is generated iteratively. If the generated intermediate result data is immediately stored in the storage circuit after each iteration, the storage circuit will continuously switch between programming and reading or computation states. On the one hand, state switching reduces the model's operating efficiency; on the other hand, the number of programming cycles for the storage circuit is limited, so frequent switching to the programming state will affect the lifespan of the storage circuit. By buffering the intermediate result data in the second buffer area and storing the buffered intermediate result data in the second storage area after the buffered data reaches a set amount, the control circuit can reduce the number of programming cycles, improve the model's operating efficiency, and extend the lifespan of the storage circuit.

[0092] In some embodiments of this application, the setting amount may include data from one or more programming units. A programming unit may include a subarray of a storage array within a storage group, comprising multiple storage cells arranged in a third direction (e.g., the X direction). Thus, programming is performed only when the intermediate result data to be buffered reaches the data of one or more programming units, improving programming efficiency. This setting amount can be reflected by the size of a second buffer area, thereby simplifying the control logic.

[0093] In some embodiments of this application, the first buffer region and the second buffer region may be located in the same storage circuit or in different storage circuits. For example, the in-memory computing device 800 may include a storage circuit 840, which includes a first buffer region and a second buffer region. This storage circuit 840 is independent of the storage circuit 810 and may employ a memory that is different from or the same as the storage circuit 810. For example, the storage circuit 840 may include SRAM.

[0094] In some embodiments of this application, intermediate result data can be prefetched to further improve the model's running efficiency. For example, the control circuit 820 is also configured to read first intermediate result data from a first space in the second storage area based on first read address information, where the first read address information indicates the first space, and to prefetch second intermediate result data from a second space in the second storage area, where the second space is determined based on the first read address information and is adjacent to the first space. During model operation, multiple calculations can be performed iteratively. The intermediate result data generated by each calculation can be stored sequentially in the second storage area in a preset order. Thus, each time the control circuit calls a set of intermediate result data, it knows the location of the next set of intermediate result data to be called. By utilizing the adjacency relationship of the spatial locations, the prefetching of intermediate result data can improve the model's running efficiency and reduce the waiting time for intermediate result data during operation. For example, when the control circuit 820 reads the first intermediate result data from the first space based on the first read address information and caches it in the first cache, it can prefetch the second intermediate result data stored in the second space adjacent to the first space and cache the prefetched second intermediate result data in the second cache. In this way, when the control circuit uses the second intermediate result data for calculation, it can directly call the prefetched intermediate result data from the buffer area, which is faster than calling the data from the storage circuit 810. This shortens the delay caused by waiting for the second intermediate result data and improves the running efficiency of the model.

[0095] In some possible embodiments, the second space can be determined based on the first read address information and the size of a set of intermediate result data, the size of which can be related to the size of the buffer area that buffers the intermediate result data.

[0096] In some possible embodiments, the in-memory computing device can be configured with multiple third buffer areas for time-sharing storage of multiple copies of intermediate result data. Taking two third buffer areas as an example, intermediate result data read at different times can be alternately buffered in different third buffer areas, thereby alternating the reading of intermediate result data, improving the reading efficiency of intermediate result data, and thus improving the running efficiency of the model.

[0097] According to some embodiments, the third buffer area may include a first cache and / or a second cache.

[0098] In some embodiments of this application, please refer to [the relevant documentation]. Figure 1The first storage circuit may further include a third storage area 113, which is used to store configuration information of the first storage circuit. For example, the configuration information of the first storage circuit may include one or more types of data such as a storage management table, an address translation table, and a status table. This application does not limit the position of the third storage area relative to the first and second storage areas. For example, the third storage area may be set in an independent storage set or storage group, or the third storage area may share a storage set or storage group with the first or second storage area. In some other embodiments of this application, the in-memory computing device may also include an independent storage circuit for storing the configuration information of the first storage circuit.

[0099] In some embodiments of this application, the control circuit can also control the calculation of the model. The calculation of the model can include multiple calculations, and a single calculation can include at least one linear calculation and at least one nonlinear calculation. The model parameter data is used for one or more linear calculations, and the intermediate result data can include intermediate results of one or more linear calculations. That is, as the number of calculations increases, the intermediate result data can include multiple intermediate result data accumulated from multiple linear calculations.

[0100] The in-memory computing device can support both in-memory computation and near-memory computation. The linear computation described above can be performed in the storage circuit 810 via in-memory computation, or via near-memory computation in the processing circuit 830.

[0101] In some embodiments of this application, the control circuit 820 is configured to control a first calculation involving multiple calculations to obtain a first calculation result. The first calculation includes a first linear calculation and a first nonlinear calculation. The first linear calculation is used to obtain first intermediate result data based on model parameter data. The control circuit 820 stores the first intermediate result data in a second storage area. For example, when the first calculation is the first calculation in multiple calculations, the first calculation may include a prefilling process in the model calculation. Optionally, the control circuit 820 may control the storage circuit 810 to perform the linear calculation in the first calculation and control the processing circuit 830 to perform the nonlinear calculation in the first calculation; alternatively, the control circuit 820 may control the processing circuit 830 to perform both the linear and nonlinear calculations in the first calculation to obtain the first calculation result and the first intermediate result data. The control circuit 820 may also control the storage circuit 840 to cache the first intermediate result data and the first calculation result, and store the first intermediate result data in the second storage area of ​​the storage circuit 810. Storing the first intermediate result data in the storage circuit 810 allows for reuse in subsequent calculations, reducing the number of calculations and improving the model's operating efficiency.

[0102] In some embodiments of this application, the control circuit 820 can control a second calculation involving multiple calculations based on the first intermediate result data and the first calculation result to obtain a second calculation result. The second calculation includes a second linear calculation and a second nonlinear calculation. The second linear calculation is used to obtain the second intermediate result data based on the model parameter data. The second intermediate result data is stored in a second storage area. The control circuit 820 can control the iterative execution of the following calculations to implement model operation: based on the accumulated intermediate result data and the (k-1)th calculation result, control the kth calculation involving multiple calculations to obtain the kth calculation result. The accumulated intermediate result data includes the first intermediate result data to the (k-1)th intermediate result data, where k is a positive integer greater than or equal to 3, and the kth calculation includes the kth linear calculation and the kth nonlinear calculation. The kth linear calculation obtains the kth intermediate result data based on the model parameter data. The kth intermediate result data is stored in a second storage area.

[0103] In some embodiments of this application, the linear calculation of any of the above calculations can be implemented through near-memory calculation. For example, the control circuit 820 reads model parameter data from the storage circuit 810, and based on the model parameter data, controls the multiplication calculation of the linear calculation in the k0th calculation to be executed in the processing circuit, where k0 is a positive integer greater than or equal to 1. This multiplication calculation is based on the first type of parameters in the model parameter data. The control circuit 820 can cache the model parameter data and input data in the storage circuit 840, instruct the processing circuit 840 to call it, and perform the multiplication calculation of the first type of parameters and the input data, and adjust the result of the multiplication calculation using the second type of parameters.

[0104] In some embodiments of this application, the linear calculation of any of the above calculations can be implemented through in-memory computation. For example, the control circuit 820 controls the multiplication calculation of the linear calculation in the k0th calculation to be executed in the first storage area, where k0 is a positive integer greater than or equal to 1. This multiplication calculation is based on the first type of parameters in the model parameter data. The control circuit 820 can convert the input data into an input signal and provide it to the storage circuit 810. The first storage area of ​​the storage circuit 810 performs the multiplication calculation in the analog domain based on the input signal and the stored weight data, and outputs the result of the multiplication calculation. The control circuit 820 senses the result of the multiplication calculation and adjusts the result using the second type of parameters stored in the second storage area.

[0105] After obtaining the result of the multiplication calculation, the control circuit 820 can adjust the result itself based on the second type of parameters, or control other circuits, such as the post-processing circuit or the processing circuit 830, to adjust the result based on the second type of parameters.

[0106] In some embodiments of this application, the control circuit 820 is configured to read the result of the multiplication calculation of the k0th linear calculation and the second type of parameters in the model parameter data from the storage circuit 810, and adjust the result of the multiplication calculation based on the second type of parameters. The adjusted result of the multiplication calculation includes the k0th intermediate result data. In other embodiments of this application, the control circuit 820 is configured to read the second type of parameters in the model parameter data from the storage circuit, and adjust the result of the multiplication calculation of the k0th linear calculation performed in the processing circuit based on the second type of parameters, wherein k0 is a positive integer greater than or equal to 1, and the adjusted result of the multiplication calculation includes the k0th intermediate result data.

[0107] In some embodiments of this application, the calculation result may include tokens, and the intermediate result data may include key vectors and value vectors.

[0108] The following description uses the Transformer model as an example, which is merely illustrative and not intended to limit this application. The Transformer model architecture can include an encoder and a decoder. The encoder processes the input sequence, and the decoder generates the output sequence. Both the encoder and decoder can include the same structure, such as a self-attention network and a feedforward network. A single computation can be performed using both the self-attention and feedforward networks. In the encoder, the model embeds each word in the input sequence into a multi-dimensional vector representation. This embedding process allows the model to capture semantic similarities between words. The encoder can combine the input embeddings with positional information; positional encoding enables the model to distinguish the relative positions of elements in the sequence. Then, the model computes three vectors for each word in the input sequence: a query vector (Q), a key vector (K), and a value vector (V). The query vector represents the word's query, i.e., what the model is looking for in the sequence. The key vector represents the word's key, i.e., what other words in the sequence should pay attention to. The value vector represents the word's value, i.e., the information the word contributes to the output. After obtaining the query vector for each word, the model can calculate an attention score for each pair of words in the input sequence, for example, by taking the dot product of the query vector and the key vector to evaluate the similarity between words. Then, the attention scores can be normalized to obtain attention weights. These attention weights are then used to calculate a weighted sum of the value vectors to obtain the self-attention mechanism output for the words in the sequence. The Transformer model can simultaneously focus on different parts of the input sequence based on a multi-head attention mechanism. By dividing the query vector, key vector, and value vector into multiple heads and performing independent self-attention calculations, it obtains more diverse contextual information. The feedforward network can be implemented using multiple linear transformations and non-linear activation functions. For example, the first linear transformation projects the input representation into a higher-dimensional space using a first learnable weight matrix; the output of the first linear transformation can be passed through a non-linear activation function, enabling it to capture complex patterns and relationships in the data; the output of the activation function can be projected back into the original dimensional space using a second learnable weight matrix. During inference, the query vector can be reused. Therefore, in subsequent iterations, the intermediate structure data can include both key and value vectors, saving the repetitive computation of the query vector.

[0109] In some embodiments of this application, a single computation may include computations of a self-attention network and a feedforward network. The computation of the self-attention network may include linear computations that determine the query vector, key vector, and value vector, or linear computations that determine the key vector and value vector. The computation of the feedforward network may include various linear and nonlinear computations. Furthermore, a single computation may also include normalization computations and computations of other functions.

[0110] The control circuit 820, processing circuit 830, and storage circuit 840 in the above embodiments can be integrated together, for example, in the form of a system-on-chip (SOC). The processing circuit 830 is merely an example and may include one or more processors; when multiple processors are included, the types of these processors may differ. A processor is a circuit with signal processing capabilities; for example, a processor may be a circuit with instruction read and execute capabilities. In other possible embodiments, the processor can implement its functions through the logical relationships of hardware circuits, which may be fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an Application Specific Integrated Circuit (ASIC) or a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document to configure the hardware circuit can be understood as the process of the processor loading instructions to achieve its functions. This application does not limit the type of processor, such as a central processing unit (CPU), microcontroller unit (MCU), graphics processing unit (GPU), or digital signal processor. Alternatively, the processor may include hardware circuitry designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), or deep learning processing unit (DPU).

[0111] In the above embodiments, the descriptions of different embodiments each have their own emphasis. Parts not described in detail or recorded in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the different embodiments described above can be freely combined as needed. Moreover, as technology evolves, the elements described in this application can be replaced by equivalent elements appearing after this application.

[0112] This application also provides an electronic device, which can be found in [reference 1]. Figure 9 . Figure 9 A schematic diagram of an electronic device according to an exemplary embodiment of this application is shown. Figure 9 As shown, the electronic device 900 may include any of the above-mentioned in-memory computing devices 910 for processing data of the electronic device. The electronic device may also include an input / output device 920 for receiving user input or outputting processing results. This application does not limit the input type and output type; for example, input may include voice input, text input, image input, or video input. Output may include text output, voice output, image output, or video output. The electronic device may also include a processor 930, which can process data provided to the in-memory computing device 910 or process the output data of the in-memory computing device 910. The output of the input / output device 920 may be based on the output of the processor 930 or the output of the in-memory computing device 910.

[0113] This application does not limit the type of electronic device. For example, according to some embodiments, the electronic device may include wearable devices. Wearable devices include, but are not limited to: head-mounted devices (e.g., helmets or hats), devices worn on the ears (e.g., headphones), devices worn on the wrist (e.g., watches), and devices worn on other parts of the body (e.g., electronic necklaces, medical monitoring devices, or glasses). According to some embodiments, the electronic device may include portable terminals. For example, the electronic device may include, but is not limited to, mobile phones, general-purpose computing devices (e.g., laptops or tablets), personal digital assistants, etc. According to some embodiments, the electronic device may include other types of edge devices, such as personal computers, in-vehicle computers or in-vehicle computing platforms, or smart home electronic products. According to some embodiments, the electronic device may also include devices such as servers.

[0114] In the above embodiments, the descriptions of different embodiments each have their own emphasis. Parts not described in detail or recorded in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the different embodiments described above can be freely combined as needed. Moreover, as technology evolves, the elements described in this application can be replaced by equivalent elements appearing after this application.

Claims

1. A storage circuit, characterized in that, include: A first storage region includes a first storage group, the first storage group including a multi-layer first storage array stacked along a first direction; The second storage region includes a second storage group, the second storage group including a multi-layer second storage array stacked along the first direction, wherein the first storage group and the second storage group are arranged along the second direction, and the first storage region is configured to store model parameter data, the model parameter data being used for linear calculation of the model, and the second storage region is configured to store intermediate result data obtained from the linear calculation.

2. The storage circuit according to claim 1, characterized in that, The model parameter data includes a first type of parameter and a second type of parameter. The first type of parameter is used for the multiplication calculation of the linear calculation, and the second type of parameter is used to adjust the result of the multiplication calculation.

3. The storage circuit according to claim 2, characterized in that, The first storage area includes: The first sub-region is configured to store the first type of parameters; The second sub-region is configured to store the second type of parameters, wherein the first sub-region and the second sub-region are arranged along the first direction, and the first sub-region includes a first storage array with a first number of layers, and the second sub-region includes a first storage array with a second number of layers, wherein the first number of layers is greater than the second number of layers.

4. The storage circuit according to claim 3, characterized in that, The number of the first layer and the number of the second layer satisfy a set ratio; or, The proportion of the first layer or the second layer in the first storage group satisfies a set proportion; or, The first layer number or the second layer number has a set value in the first storage group.

5. The storage circuit according to any one of claims 1 to 4, characterized in that, The storage circuit also includes: A third storage area is configured to store configuration information of the storage circuit.

6. A storage computing device, characterized in that, include: The storage circuit as described in any one of claims 1 to 5; A control circuit, connected to the storage circuit, is configured to control the storage of model parameter data and intermediate result data in the storage circuit.

7. The storage and computing device according to claim 6, characterized in that, The model parameter data includes a first type of parameter and a second type of parameter. The first type of parameter is used for the multiplication calculation of the linear calculation of the model, and the second type of parameter is used to adjust the result of the multiplication calculation. The control circuit is configured to store the first type of parameter in a first sub-region of a first storage area of ​​the storage circuit, and to store the second type of parameter in a second sub-region of the first storage area.

8. The storage device according to claim 7, characterized in that, The control circuit is configured to store the first type of parameters in the first sub-region in storage units.

9. The storage device according to claim 8, characterized in that, Also includes: A first buffer region is used to buffer the second type of parameters, wherein the storage unit is determined based on the size of the first buffer region.

10. The storage device according to claim 6, characterized in that, Also includes: The second buffer area is used to buffer the intermediate result data; The control circuit is configured to store the intermediate result data buffered in the second buffer region into the second storage region of the storage circuit when the intermediate result data buffered in the second buffer region reaches a set amount.

11. The storage device according to claim 6, characterized in that, Also includes: A first cache is used to cache first intermediate result data. The first intermediate result data is read by the control circuit from the first space of the second storage area of ​​the storage circuit based on the first read address information. The first read address information is used to indicate the first space. The second cache is used to cache the second intermediate result data, which is prefetched by the control circuit from the second space of the second storage area. The second space is determined based on the first read address information and is adjacent to the first space.

12. The storage device according to any one of claims 6 to 11, characterized in that, The control circuit is further configured to control the calculation of a model, the calculation including multiple calculations, each calculation including linear and nonlinear calculations, the model parameter data being used for the linear calculations, and the intermediate result data including multiple intermediate result data from the multiple linear calculations; the in-memory computing device further includes: At least one processing circuit is configured to perform the nonlinear calculation; The storage circuit is configured to perform the linear calculation, or the at least one processing circuit is configured to perform the linear calculation.

13. An electronic device, characterized in that, It includes the storage device as described in any one of claims 6 to 12.

Images