Memory system

Abstract

The invention discloses a memory system, and belongs to the technical field of memories. The memory system comprises a memory controller and a memory, the memory controller is used for managing the memory; the memory has N levels of memory sets, each memory set comprises a plurality of memory blocks, the (i + 1) th level of memory set in the N levels of memory sets comprises a plurality of ith level of memory sets, and each Nth level of memory set in the N levels of memory sets is configured with a computing unit; a computing unit is configured in a part of levels or a part of all levels of the memory sets from the first level to the (N-1)-1 level, the computing unit is used for executing computing operation based on data stored in the memory sets, and N and i are positive integers. The computing performance of the memory can be further improved.

Description

Technical Field

[0001] The present application relates to the field of memory technology, and in particular to a memory system. Background Art

[0002] Near memory processing (NMP) technology is a technology that has emerged in recent years. By integrating near memory processing elements (NMP PE) in memory, near memory computing technology can directly perform computing operations on data obtained from memory in memory without reading data from memory to the central processing unit (CPU) for processing, greatly reducing the data transmission overhead between CPU and memory.

[0003] The memory has N levels of memory sets. At present, a near memory computing unit is usually configured for each highest level memory set, and the near memory computing unit of the memory set is responsible for computing the data stored in the memory set and all the memory sets it includes. Multiple highest level memory sets are configured with near memory computing units to perform computing operations in parallel.

[0004] However, since the number of the highest level memory collections in the memory is limited, this method of configuring near-memory computing units has very limited effect on improving the computing performance of the memory. Summary of the invention

[0005] This application provides a memory system. This application helps to further improve the computing performance of memory. The technical solutions provided by this application are as follows:

[0006] In a first aspect, the present application provides a memory system. The memory system includes: a memory controller and a memory; the memory controller is used to manage the memory; the memory has N levels of memory sets, each memory set includes multiple memory blocks, the i+1th level memory set in the N-level memory set includes multiple i-th level memory sets, in the N-level memory set, each N-level memory set is configured with a computing unit, some of the levels or some of the levels in the first to N-1 level memory sets are configured with computing units, and the computing units are used to perform computing operations based on data stored in the memory set, and N and i are positive integers.

[0007] In this way, in this memory system, not only is each Nth level memory set configured with a computing unit, but some memory sets in some levels or all levels of the first to N-1 level memory sets are also configured with the computing units, thereby increasing the total number of computing units configured in the memory and helping to further improve the computing performance of the memory.

[0008] Among them, some first-level memory sets are configured with computing units. The parallelism of multiple memory subsets can guarantee the computational parallelism of the computing units configured for the multiple memory subsets. However, considering the characteristics of the data required for the calculation, such as the different access frequencies of different data, if computing units are configured for all first-level memory sets, problems such as uneven load balancing and low computing resource utilization will occur, and a large chip area and power consumption will be incurred. Therefore, by configuring computing units for some first-level memory sets, the parallelism of computing, the load balancing of computing, the utilization of computing resources, the chip area and the power consumption can be taken into account.

[0009] In one implementation, in the second to N-1 level memory sets, at least some memory sets included in any level memory set configured with computing units are not configured with computing units. In this way, the computing units configured in the second level memory set can be used to process the computing operations of the data stored in the first level memory set that is not configured with computing units, while the computing operations of the data stored in the first level memory set that is configured with computing units are executed by the computing units of the first level memory set, thereby ensuring that the computing operations of the data stored in the memory set can all be executed by the computing units configured in the memory, and avoiding the problem of large chip area and power consumption overhead caused by configuring computing units for each memory set.

[0010] In the memory system of the present application, the memory controller is specifically used to provide computing instructions to the computing unit; the computing unit is specifically used to obtain data required for the calculation from the memory set to which the computing unit belongs based on the computing instructions, and perform calculations based on the data.

[0011] In one implementation, the data stored in the i-th level memory set configured with computing units has a higher requirement for computing parallelism than the data stored in the i-th level memory set not configured with computing units, and the computing parallelism is obtained based on one or more of the following: the access frequency of the data, the size of the data set to which the data belongs, the rated amount of data processed by each computing batch of the computing unit, and the rated capacity and rated bandwidth of the N-level memory set.

[0012] Optionally, the computing units of the i+1th level memory set are specifically used to: provide computing instructions to the computing units of the i+1th level memory set included in the i+1th level memory set when the computing task indicated by the computing instruction does not belong to the computing units of the i+1th level memory set; and execute the computing task indicated by the computing instruction when the computing task indicated by the computing instruction belongs to the computing units of the i+1th level memory set.

[0013] It should be noted that, in some computing logics, when the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set, the computing unit of the i+1th level memory set is specifically used for: when the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set, receiving the computing results provided by the computing units of the i+1th level memory set included in the i+1th level memory set, processing the computing results and outputting the processed computing results.

[0014] Since the Nth level memory set is the highest level memory set in the memory, the output of the memory at the same time is limited. Therefore, when the memory has multiple Nth level memory sets, the memory also includes: an output processing unit, which is used to process the calculation results of the calculation units of multiple Nth level memory sets and output the processed calculation results.

[0015] At this time, the memory controller is specifically used to send the computing instruction to the output processing unit; the output processing unit is specifically used to send the computing instruction to the Nth level memory set to which the computing unit used to process the computing instruction belongs.

[0016] In one implementation, a memory set includes multiple memory subsets, multiple local buffers, multiple control subunits, and a global buffer. Multiple memory subsets correspond to multiple local buffers, multiple memory subsets correspond to multiple control subunits, and the memory subsets include multiple memory blocks arranged in an array. The control subunit corresponding to any memory subset is used to select a target first memory unit from multiple first memory units included in any memory subset based on the address of the data required for calculation, determine the target first memory unit's permission to use the global buffer, and select a target second memory unit from multiple second memory units included in the target first memory unit, one of the first memory unit and the second memory unit is a row and the other is a column. The local buffer corresponding to any memory subset is used to cache data stored in a memory block in the target first memory unit. The global buffer is used to cache target data stored in a memory block in a target second memory unit with permission to use the global buffer. The computing unit includes a computing subunit, which is used for target data and performs computing operations based on the target data.

[0017] Among them, when multiple control sub-units in the memory set determine the target first memory unit's access to the global buffer, the multiple control sub-units need to ensure that at the same time there is a target first memory unit that has access to the global buffer to avoid data transmission conflicts and ensure the correct transmission of data stored in the global buffer.

[0018] The implementation method of the control subunit provided in the present application enables multiple memory subsets in a memory set to simultaneously activate a target first memory unit, and cache the selected target first memory unit in a local buffer corresponding to the memory subset, cache it to the global buffer when the target first memory unit has the right to use the global buffer, and provide the target second memory unit stored in the global buffer to the computing subunit, so that the selected target first memory unit is not necessarily transmitted to the computing subunit, thereby realizing the separation of data activation and data transmission.

[0019] In a second aspect, the present application provides a computing device including a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the memory system provided in the first aspect of the present application and any possible implementation thereof.

[0020] In a third aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and which includes program instructions. When the program instructions are executed on a computing device, the computing device executes the memory system provided in the first aspect of the present application and any possible implementation thereof.

[0021] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the memory system provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of a memory system provided in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0024] Figure 3 It is a structural diagram of a recommendation model based on deep learning provided in an embodiment of the present application;

[0025] Figure 4 is a schematic diagram of another memory system provided by an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of another memory system provided in an embodiment of the present application;

[0027] Figure 6 is a structural diagram of an output processing unit provided in an embodiment of the present application;

[0028] Figure 7 is a structural diagram of another output processing unit provided in an embodiment of the present application;

[0029] Figure 8 It is a structural diagram of a computing unit configured with an N-th level memory set provided in an embodiment of the present application;

[0030] Fig. 9 It is a structural diagram of a computing unit of another N-level memory set configuration provided in an embodiment of the present application;

[0031] Fig.10 It is a structural schematic diagram of a computing unit configured in a j-1th level memory set in a second to N-1th level memory set provided by an embodiment of the present application;

[0032] Fig.11 It is a schematic diagram of the structure of a computing unit configured in a j-1th level memory set in another second to N-1th level memory set provided by an embodiment of the present application;

[0033] Fig.12 It is a structural diagram of a computing unit configured with a first-level memory set provided in an embodiment of the present application;

[0034] Fig.13 It is a structural diagram of a computing unit of another first-level memory set configuration provided in an embodiment of the present application;

[0035] Fig.14 It is a structural diagram of a memory set provided in an embodiment of the present application;

[0036] Fig.15 is a structural schematic diagram of a control subunit provided in an embodiment of the present application;

[0037] Fig.16 is a schematic diagram of the structure of a computing subunit provided in an embodiment of the present application;

[0038] Fig.17 It is a structural diagram of a memory calculation instruction provided in an embodiment of the present application;

[0039] Fig.18 It is a schematic diagram comparing the acceleration ratios of a related technology provided in an embodiment of the present application and the memory system of the present application at different embedded vector lengths;

[0040] Fig.19 It is a schematic diagram comparing the acceleration ratios of a related technology provided in an embodiment of the present application and the memory system of the present application at different embedding operation batch sizes;

[0041] Fig. 20 It is a schematic diagram comparing the energy consumption of a related technology provided in an embodiment of the present application and the memory system of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0043] To facilitate understanding, the technology and background involved in the embodiments of the present application are first explained below.

[0044] Near memory processing (NMP) technology is a technology that has emerged in recent years. By integrating near memory processing elements (NMP PE) in memory, near memory computing technology can directly perform computing operations on data obtained from memory in memory without reading data from memory to the central processing unit (CPU) for processing, greatly reducing the data transmission overhead between CPU and memory.

[0045] The memory has N levels of memory collection. Figure 1 As shown in the figure, the host is connected to the memory through a memory controller, and each memory controller controls a memory channel that can be accessed independently. Each memory channel is connected to multiple memory ranks that share an instruction and data bus. These ranks are physically placed in dual inline memory modules (DIMMs). A DIMM can be loaded with multiple ranks to maximize memory parallelism and meet the requirements of large memory capacity. Due to the dynamic random access memory (DRAM) chip (such as Figure 1 The input / output (I / O) data width of the chip is limited (usually 4, 8 or 16 bits), so multiple DRAM chips are usually combined into one rank to achieve a 64-bit data width. Each DRAM chip consists of multiple memory bank groups, and each bank group includes multiple memory banks. Figure 1 The dotted box in the figure indicates a bank group. A bank is a physical storage unit of the memory, which is a structure that actually stores data in the memory. Both bank group and rank are logical concepts for managing banks. The data stored in a bank group and rank are actually stored in the banks they include. Figure 1 This is a schematic diagram of the structure of a DDR-based memory system, but most current memories have similar structures. Figure 1 To understand.

[0046] Logically, a bank can be viewed as a memory cell array with a row decoder, a column decoder, a row buffer, and multiple sub-arrays. However, implementing a large memory cell array often results in long bit lines and word lines, which poses a great challenge to signal reliability and delay. Therefore, today's banks are usually composed of multiple small memory cell arrays, each of which is usually 512x512 in size. Figure 1 As shown, all arrays in the horizontal direction share the same row decoder and global word line. Such a row of arrays is called a subarray. In DRAM operation, only one row in one of the subarrays in a bank can be activated at a time for read or write operations, so that continuous access to the same bank must be serial operations, which leaves ample opportunity to utilize higher levels of memory parallelism.

[0047] At present, a near memory computing unit is usually configured for each highest level memory set, and the near memory computing unit of the memory set is responsible for computing the data stored in the memory set and all the memory sets it includes. Multiple near memory computing units configured for the highest level memory sets can perform computing operations in parallel. For example, a near memory computing unit is configured for each rank, and the near memory computing unit is responsible for computing the data stored in the rank and the bank group and bank it includes, and the near memory computing units configured for multiple ranks perform computing operations in parallel.

[0048] However, since the number of the highest level memory collections in the memory is limited, this method of configuring near-memory computing units has very limited effect on improving the computing performance of the memory.

[0049] To this end, an embodiment of the present application provides a memory system. The memory system includes: a memory controller and a memory. The memory controller is used to manage the memory. The memory has N levels of memory sets. Each memory set includes multiple memory blocks. The i+1-th level memory set in the N-level memory set includes multiple i-th level memory sets. In the N-level memory set, each N-th level memory set is configured with a computing unit, and some memory sets in some or all levels of the first to N-1 level memory sets are configured with the computing unit. The computing unit is used to perform computing operations based on data stored in the memory set. N and i are positive integers.

[0050] In this way, in this memory system, not only is each Nth level memory set configured with a computing unit, but some memory sets in some levels or all levels of the first to N-1 level memory sets are also configured with the computing units, thereby increasing the total number of computing units configured in the memory and helping to further improve the computing performance of the memory.

[0051] The implementation environment of the memory system provided in the embodiment of the present application includes: a computing device. The computing device is configured with the memory system provided in the embodiment of the present application. The computing device may be a server or a terminal, etc. Figure 2 Schematic diagram of a computing device provided in an embodiment of the present application. Figure 2 As shown, the computing device 20 includes: a processor 201, a communication interface 202, an external memory 203, a memory (also called physical memory) 204, a basic input output system 205 (BIOS) and a bus 206. The memory 204 is configured with a computing unit 1041. The processor 201, the communication interface 202, the external memory 203, the memory 204 and the basic input output system 105 are connected to each other through the bus 206.

[0052] Processor 201 may be a dedicated hardware chip. A dedicated hardware chip is a high-performance processing hardware module. Dedicated hardware chips include digital signal processors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs) or a combination thereof. The above-mentioned PLDs may be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs) or any combination thereof. Processor 201 may also be a general-purpose processor, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), a network processor (NP) or a combination of a CPU and a NP. The CPU may be, for example, a single-core processor (single-CPU) or a multi-core processor (multi-CPU).

[0053] The communication interface 202 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. For example, the communication interface 202 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0054] The external memory 203 is used to store executable code or data generated when executing an application program. The executable code stored in the external memory 203 can be read into the memory and run by the processor 201 to implement the function corresponding to the executable code. Similarly, the data stored in the external memory 203 can be read into the memory and accessed by the processor 201 to perform corresponding processing according to the data. Alternatively, the executable code stored in the external memory 203 can be read into the memory and run by the computing unit 1041 to implement the computing function corresponding to the executable code. Similarly, the data stored in the external memory 203 can be read into the memory and accessed by the computing unit 1041 to perform corresponding computing operations according to the data.

[0055] The external memory 203 is, for example, a read-only memory or other type of static storage device that can store static information and instructions, or a random access memory or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a blue-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired executable code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the external memory 203 is used to store an outbound port queue, etc. The external memory 203 is, for example, independent and connected to the processor 201 via the bus 206. Or the external memory 203 and the processor 201 are integrated together. For example, the external memory 203 may include a volatile memory, such as a random-access memory (RAM); the external memory 203 may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or, the external memory 203 may also include a combination of the above types of memory.

[0056] The memory 204 is used to temporarily store the data required by the processor 201 when it is running, such as reading the operating system from the external memory 203 into the memory 204 when the computing device is started, and the data generated when the application is executed. Since the memory 204 has a faster reading and writing speed, the memory 204 is usually used as a cache for the external memory 203 and the processor 201, that is, during the operation of the computing device, the data that the processor 201 needs to use can be read from the external memory 203 into the memory 204 in advance, so that the processor 201 reads the required data from the memory 204 to ensure the computing speed of the processor 201. In the embodiment of the present application, the memory 204 is configured with a computing unit 1041, which can read the data stored in the memory 204 and perform corresponding computing operations based on the data.

[0057] The basic input and output system 205 is used to perform self-check and initialization on each component in the computing device, and serves as a transmission medium between software programs and hardware to transmit information between software programs and hardware.

[0058] The bus 206 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., the external memory 203, the processor 201, the communication interface 202). For example, the bus 206 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The embodiment of the present application takes the above-mentioned devices in the computing device 20 as an example of interconnecting through the bus 206. Optionally, the above-mentioned devices in the computing device 20 can also be connected to each other in communication with each other using other connection methods other than the bus 206. For example, the above-mentioned devices in the computing device 20 are interconnected through an internal logical interface.

[0059] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0060] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing a program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, all or part of the functions of the memory method provided in the embodiments of the present application are implemented in whole or in part.

[0061] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0062] In one implementation, the memory system provided in the embodiment of the present application can be used to accelerate computing operations. Acceleration here refers to the execution of computing operations based on data in the memory by a computing unit configured by the memory. Since there is no need to read the data from the memory to the CPU, the overall time consumption of the calculation is shortened, thereby achieving the effect of accelerating the calculation. For example, the memory system can be used to accelerate the computing operations of the embedded layer in the recommendation system, that is, the computing units in the memory system are used to execute the computing operations of the embedded layer. In this way, there is no need to read the data required for the embedded layer calculation from the memory to the CPU for processing, which shortens the time consumption of the embedded layer to perform the calculation, thereby achieving the purpose of accelerating the calculation.

[0063] Currently, due to its industrial importance, recommender systems have become the infrastructure of the Internet. Recommender systems have been widely deployed in various applications, including e-commerce, search engines, media services, and social networks. In order to provide fast, accurate, and personalized recommendation services, deep learning has been applied to recommender systems and has attracted increasing attention. Figure 3 A recommendation model based on deep learning is presented. Figure 3 As shown in the figure, the recommendation model includes two fully connected layers (fully connected layer 1 and fully connected layer 2) and an embedding layer with multiple embedding tables. Fully connected layer 1 is responsible for processing dense inputs containing personal information. Personal information includes, for example, user age, gender, and other preferences. The embedding layer is responsible for processing sparse inputs containing behavioral information collected from past user interactions. Behavioral information includes, for example, information about the products purchased and browsed by the user and the interaction information generated during the period. The outputs of fully connected layer 1 and the embedding layer are combined as the input of fully connected layer 2 to obtain the user's preference for the current scenario.

[0064] The embedding layer needs to perform computational operations based on different numbers and sizes of embedding tables (depending on the recommendation model). Each embedding table is a two-dimensional matrix containing millions of rows. For example, each row of the embedding table is an embedding vector containing 32 to 256 dimensions. The embedding layer looks up the embedding table based on a set of indices, and then performs aggregation operations (e.g., direct addition, averaging, weighted sum, etc.) based on the embedding vectors in the embedding table (usually 20 to 80).

[0065] Compared with the fully connected layer, the embedding layer exhibits the following characteristics: (1) Large data volume. Compared with the fully connected layer that occupies a relatively small storage space, for the recommendation model in the production environment, the embedding layer can occupy hundreds of gigabytes (GB) to several terabytes (TB) of storage space. Therefore, such a large embedding table is usually stored in a large-capacity memory; (2) Low computational density. Compared with the computationally intensive fully connected layer, the embedding layer often performs simple operations. Its computational density is 0.25 FLOPs / Byte, which is much lower than the 5.5 FLOPs / Byte of the recurrent neural network (RNN), the 18 FLOPs / Byte of the fully connected (FC) network structure, and the 18 FLOPs / Byte of the convolutional neural network (CNN). (1) The embedding layer has a performance bottleneck of 141 FLOPs / Byte, which is even worse as the amount of data increases. Therefore, the present application uses the computing units in the memory system to perform the computing operations of the embedding layer, which greatly reduces the data transmission between the CPU and the memory, and can use the parallelism of the computing units in the memory system to ensure the computing performance.

[0066] It should be understood that the above content is an illustrative description of the application scenarios of the memory method provided in the embodiments of the present application, and does not constitute a limitation on the application scenarios of the memory method. A person of ordinary skill in the art will know that as business needs change, its application scenarios can be adjusted according to application needs, and the embodiments of the present application do not list them one by one.

[0067] The following is an example of the memory system provided in the embodiment of the present application. Figure 4 is a schematic diagram of a memory system provided by an embodiment of the present application. Figure 4 As shown, the memory system 40 includes a memory controller 401 and a memory 402 .

[0068] The memory controller 401 is used to manage the memory 402. For example, the memory controller 401 is used to receive a memory computing request, generate a memory computing instruction based on the memory computing request, and send the memory computing instruction to the memory 402, so that the memory 402 performs a corresponding computing operation based on the memory computing instruction using the computing unit 4022. For another example, the memory controller 401 is used to receive a memory computing instruction, and distribute the memory computing instruction to the memory 402, so that the computing unit 4022 of the memory 402 performs a corresponding computing operation based on the memory computing instruction using the computing unit 4022. The memory controller 401 is also used to receive a computing result provided by the computing unit 4022 in the memory 402, and output the computing result.

[0069] The memory 402 has N levels of memory sets 4021, each memory set 4021 includes multiple memory blocks ( Figure 4 The i+1th level memory set 4021 in the N-level memory set 4021 includes a plurality of i-th level memory sets 4021. In the N-level memory sets 4021, each N-level memory set 4021 is configured with a computing unit 4022, and some of the memory sets 4021 in some or all of the first to N-1 level memory sets 4021 are configured with computing units 4022, and N and i are positive integers.

[0070] In one implementation, some of the levels or some of the memory sets 4021 in all the levels of the first to N-1 level memory sets 4021 are configured with computing units 4022, including at least the following situations: In the first situation, when viewed from the perspective of the level to which the memory sets 4021 belong, each level of the memory sets 4021 in the first to N-1 levels of memory sets 4021 is configured with computing units 4022, but some levels of memory sets 4021 in the first to N-1 levels where one or more levels of memory sets 4021 exist are not configured with computing units 4022; In the second situation, when viewed from the perspective of the level to which the memory sets 4021 belong, some levels of memory sets 4021 in the first to N-1 levels of memory sets 4021 are not configured with computing units 4022, and some memory sets 4021 in the remaining levels of memory sets 4021 are configured with computing units 4022.

[0071] The memory set 4021 includes a plurality of memory subsets, each memory subset includes a plurality of memory blocks arranged in an array, and the memory blocks are used to store data. The computing unit 4022 configured in the memory set 4021 is used to perform computing operations based on the data stored in the memory set 4021. For example, the data stored in the memory blocks of the memory set 4021 are data required for computing. The computing unit 4022 configured in the memory set 4021 is used to read the data required for computing from the memory blocks of the memory set 4021, and then perform computing operations based on the data.

[0072] In this way, in the memory system 40, not only is each Nth level memory set 4021 configured with a computing unit 4022, but some memory sets 4021 in some levels or all levels in the first to N-1 level memory sets 4021 are also configured with the computing unit 4022, thereby increasing the total number of computing units 4022 configured in the memory 402, which helps to further improve the computing performance of the memory 402.

[0073] In one implementation, part of the first-level memory set 4021 is configured with a computing unit 4022. The first-level memory set 4021 includes multiple memory subsets. Each memory subset includes multiple memory blocks arranged in an array. Since the multiple memory subsets have parallelism, when an operation on a memory subset is not completed, it is possible to perform an operation on another memory subset without waiting for the operation on the one memory subset to be completed. Therefore, the parallelism of the multiple memory subsets can guarantee the computational parallelism of the computing units 4022 configured for the multiple memory subsets. However, considering the characteristics of the data required for the calculation, such as different access frequencies of different data, if all the first-level memory sets 4021 are configured with computing units 4022, there will be problems of uneven load balancing and low computing resource utilization, and it will bring about a large chip area and power consumption overhead. Therefore, by configuring computing units 4022 for part of the first-level memory sets 4021, it is possible to take into account the parallelism of the calculation, the load balancing of the calculation, the computing resource utilization, the chip area and the power consumption overhead.

[0074] Further, in the second to N-1 level memory sets 4021, at least some memory sets 4021 included in any level memory set 4021 configured with a computing unit 4022 are not configured with a computing unit 4022. For example, when the memory 402 has three levels of memory sets 4021, each third level memory set 4021 is configured with a computing unit 4022, some second level memory sets 4021 are configured with a computing unit 4022, and some first level memory sets 4021 are configured with a computing unit 4022, then some first level memory sets 4021 included in the second level memory set 4021 configured with a computing unit 4022 are not configured with a computing unit 4022. In this way, the computing unit 4022 configured in the second-level memory set 4021 can be used to process the computing operations of the data stored in the first-level memory set 4021 which is not configured with the computing unit 4022, while the computing operations of the data stored in the first-level memory set 4021 configured with the computing unit 4022 are executed by the computing unit 4022 of the first-level memory set 4021, thereby ensuring that the computing operations of the data stored in the memory set 4021 can all be executed by the computing unit 4022 configured in the memory 402, and avoiding the problem of large chip area and power consumption overhead caused by configuring a computing unit 4022 for each memory set 4021.

[0075] Below Figure 1 Taking the memory system 40 shown as an example, the configuration of the computing unit 4022 in the memory system 40 provided in the embodiment of the present application is illustrated. According to the previous description, the memory system 40 includes multiple ranks, each rank includes multiple bank groups, and each bank group includes multiple banks. That is, the memory system 40 includes three levels of memory sets 4021, the third level memory set 4021 is a rank, the second level memory set 4021 is a bank group, and the third level memory set 4021 is a bank. In the embodiment of the present application, each rank is configured with a computing unit 4022, some bank groups and some banks are configured with computing units 4022, and the bank group configured with computing units 4022 includes banks that are not configured with computing units 4022.

[0076] In one implementation, the memory controller 401 is specifically used to provide a computing instruction to the computing unit 4022. Optionally, the computing instruction can be generated by the memory controller 401, or received by the memory controller 401 and passed to the computing unit 4022. The computing unit 4022 is specifically used to obtain the data required for the calculation from the memory set 4021 to which the computing unit 4022 belongs based on the computing instruction, and perform the calculation based on the acquired data. For example, the memory controller 401 is used to receive a memory computing request, and generate a memory computing instruction based on the memory computing request, and send the memory computing instruction to the memory 402, so that the memory 402 performs the corresponding computing operation based on the memory computing instruction using the computing unit 4022. Alternatively, the memory computing instruction can be generated by a driver layer or a programming library of a computing device, and the memory controller 401 is used to receive the memory computing instruction and distribute the memory computing instruction to the memory 402, so that the computing unit 4022 of the memory 402 performs the corresponding computing operation based on the memory computing instruction using the computing unit 4022.

[0077] In the embodiment of the present application, when customizing the memory system for a fixed usage scenario, the configuration of the computing unit 4022 in the memory 402, that is, which memory sets 4021 need to be configured with the computing unit 4022 and which memory sets 4021 do not need to be configured with the computing unit 4022, can be determined according to the usage scenario. The usage scenario determines the data characteristics of the data involved in the calculation and the computing requirements of the usage scenario. The storage method of the data involved in the calculation in the memory 402 and the configuration of the computing unit 4022 in the memory 402 can be determined according to the data characteristics and computing requirements.

[0078] In one implementation, the requirement for computational parallelism of the data stored in the i-th level memory set 4021 configured with the computing unit 4022 may be higher than the requirement for computational parallelism of the data stored in the i-th level memory set 4021 not configured with the computing unit 4022. Optionally, the computational parallelism is obtained based on one or more of the following features: the access frequency of the data, the size of the data set to which the data belongs, the rated amount of data processed by each computational batch of the computing unit 4022, and the rated capacity and rated bandwidth of the N-level memory set 4021. The rated amount of data processed by each computational batch of the computing unit 4022 refers to the maximum amount of data that can be processed by each computational batch when the data involved in the computation is distributed in multiple computational batches for computation. For example, when the memory 402 stores an embedded table and the computing unit 4022 performs a computational operation based on the embedded table, its computational parallelism is obtained based on one or more of the following features: the access frequency of the embedded vector in the embedded table, the size of the embedded table, the processing batch size when computing the embedded table, and the capacity and bandwidth of the memory sets 4021 at different levels in the memory system 40.

[0079] After obtaining the above features, the data set required for calculation can be divided based on the above features, and then the data in the data set is stored in the memory 402 according to the division scheme, and then the degree of necessity of using the computing unit 4022 for the data stored in each level of the memory set 4021 is determined according to the storage method of the data in the memory 402, and then it is determined whether to configure the computing unit 4022 for the corresponding memory set 4021 according to the degree of necessity. In one implementation, the degree of necessity can be reflected by the requirement of the data for the computing parallelism. When the requirement of the data for the computing parallelism is higher, it is necessary to try to configure the computing unit 4022 for the memory set 4021 of the lower level in the memory set 4021 to which the memory block storing the data belongs. For example, taking the memory system 40 including the three-level memory set 4021 as an example, when the data has a high requirement for computational parallelism, the computing unit 4022 is configured for the first-level memory set 4021 to which the memory block storing the data belongs, when the data has a generally high requirement for computational parallelism, the computing unit 4022 is configured for the second-level memory set 4021 to which the memory block storing the data belongs, and when the data has a low requirement for computational parallelism, the computing unit 4022 is configured for the third-level memory set 4021 to which the memory block storing the data belongs. By way of example, in the embodiment of the present application, the data capacity ratio of the memory blocks in the rank, bank group, and bank-level memory set 4021 is 4:3:1.

[0080] Alternatively, in a non-customized memory, the configuration of the computing unit in the memory and the characteristics of the data to be stored can be obtained first, and then the data to be stored in different memories can be determined according to the configuration and the characteristics of the data. The configuration of the computing unit and the characteristics of the data can be referred to in the previous description, and will not be repeated here.

[0081] It should be noted that after the data is stored in the memory 402, a data mapping table can also be generated for the data. The data mapping table is used to indicate the correspondence between the storage location of the data in the memory 402 and the original storage location of the data in the external memory, so that other components of the computing device can access the data based on the data mapping table.

[0082] In an embodiment of the present application, the computing unit 4022 of the i+1-level memory set 4021 is specifically used to: after obtaining a computing instruction, determine whether the computing task indicated by the computing instruction belongs to the computing unit 4022 of the i+1-level memory set 4021; when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-level memory set 4021, provide a computing instruction to the computing unit 4022 of the i+1-level memory set 4021 included in the i+1-level memory set 4021; when the computing task indicated by the computing instruction belongs to the computing unit 4022 of the i+1-level memory set 4021, execute the computing task indicated by the computing instruction.

[0083] It should be noted that, in some computing logics, when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-th level memory set 4021, the computing unit 4022 of the i+1-th level memory set 4021 is specifically further used for: when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-th level memory set 4021, receiving the calculation result of the computing task provided by the computing unit 4022 of the i-th level memory set 4021 included in the i+1-th level memory set 4021, processing the calculation result and outputting the processed calculation result. For example, when the computing unit 4022 performs a computing operation of the embedded layer, according to the computing logic of the embedded layer, the computing unit 4022 of the higher-level memory set 4021 needs to accumulate the computing results of the computing units 4022 of the memory set 4021 it includes. In this case, when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-level memory set 4021, the computing unit 4022 of the i+1-level memory set 4021 is also used to receive the computing result of the computing task provided by the computing unit 4022 of the i-level memory set 4021 included in the i+1-level memory set 4021, accumulate the computing results and output the accumulated results.

[0084] Since the Nth level memory set 4021 is the highest level memory set 4021 in the memory 402, the output of the memory 402 at the same time is limited. Therefore, when the memory 402 has multiple Nth level memory sets 4021, such as Figure 5 As shown, the memory 402 further includes: an output processing unit 4023. The output processing unit 4023 is used to process the calculation results of the calculation units 4022 of the plurality of N-th level memory sets 4021, and output the processed calculation results. Figure 5 It is a schematic diagram of a DIMM including two N-level memory sets 4021, in which the output processing unit 4023 is used to process the calculation results of the computing units 4022 of the two N-level memory sets 4021. For example, when the memory 402 has multiple ranks, each rank is configured with a computing unit 4022, and the memory 402 is also configured with an output processing unit 4023, and the output processing unit 4023 is used to process the calculation results received from the computing units 4022 of multiple ranks, so that the memory 402 outputs a result at each moment. Among them, when the computing unit 4022 of a rank performs the calculation operation of the embedded layer, the processing of the calculation result by the output processing unit 4023 can be to accumulate the calculation results of the computing units 4022 of multiple ranks. Alternatively, when the computing unit 4022 of a rank performs other operation logic, the processing performed by the output processing unit 4023 on the calculation result of the computing unit 4022 can be the processing indicated by the operation logic. Alternatively, when the output processing unit 4023 does not need to execute operation logic on the calculation results of the calculation units 4022 of multiple ranks, its processing can be to queue the calculation results of multiple ranks.

[0085] Optionally, when the memory 402 includes an output processing unit 4023, the memory controller 401 is specifically used to send computing instructions to the output processing unit 4023, and the output processing unit 4023 is specifically used to send computing instructions to the Nth level memory set 4021 to which the computing unit 4022 used to process the computing instructions belongs.

[0086] The following takes the example that the memory 402 includes three-level memory sets 4021, and the three-level memory sets 4021 include ranks, bank groups and banks to illustrate the implementation method of the output processing unit 4023 and the computing units 4022 configured in each memory set 4021.

[0087] Figure 6 4 is a schematic diagram of the structure of an output processing unit 4023 provided in an embodiment of the present application. Figure 6As shown, the output processing unit 4023 includes: a physical layer (physical layer, PHY) chip, a multiplexer (multiplexer, MUX) and a processing subunit. The physical layer chip is used to interact with the memory controller 401. For example, the physical layer chip is used to receive instructions (C / A) and data (DQ) such as memory computing instructions sent by the memory controller 401, and output calculation results to the memory controller 401. The multiplexer is used to distribute memory computing instructions to the computing unit 4022 of the Nth level memory set 4021. For example, the memory computing instruction carries an identifier of the memory set 4021 used to process the memory computing instruction (such as Figure 6 The multiplexer is used to send a memory calculation instruction to the memory set 4021 indicated by the identifier. The processing unit is used to process the calculation results of the calculation units 4022 of multiple N-th level memory sets 4021 and output the processed calculation results.

[0088] Alternatively, if Figure 7 As shown, the output processing unit 4023 also includes: an instruction queue, multiple first caches and a second cache. The instruction queue is used to queue the memory calculation instructions received by the physical layer chip. The multiple first caches are respectively used to cache the calculation results sent by the calculation units 4022 of the multiple N-th level memory sets 4021. The second cache is used to cache the calculation results output by the processing sub-unit.

[0089] In one implementation, when the computing unit 4022 in the memory 402 is used to perform the computing operation of the embedded layer, the processing subunit is used to accumulate the computing results of the multiple N-th level memory sets 4021, such as Figure 7 As shown, the processing subunit includes multiple accumulators. After the processing subunit accumulates the calculation results of the memory set 4021, it can output the accumulated results through the physical layer chip and the memory 402 data bus. Figure 7 It is a schematic diagram of the processing sub-unit accumulating the calculation results of the two calculation units 4022.

[0090] Figure 8 4 is a schematic diagram of the structure of a computing unit 4022 configured with an N-th level memory set 4021 provided in an embodiment of the present application. Figure 8 As shown, the computing unit 4022 includes: a first multiplexer, an instruction decoder, a computing subunit, and a second multiplexer. The first multiplexer is used to determine, based on the memory computing instruction sent by the output processing unit 4023, to send the memory computing instruction to the computing unit 4022 of the N-1th level memory set 4021 included therein (e.g. Figure 8For example, the memory computing instruction carries an identifier of the memory set 4021 used to process the memory computing instruction (such as Figure 8 In BGTag), the first multiplexer is used to send the memory computing instruction to the instruction decoder of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the first multiplexer belongs, and to send the memory computing instruction to the N-1th level memory set 4021 included in the memory set 4021 to which the first multiplexer belongs. The instruction decoder is used to decode the received memory computing instruction, obtain the algorithm used to execute the memory computing instruction and the address of the required computing data, and provide the address to the Nth level memory set 4021 (such as Figure 8 DDR signal in the memory) so that the Nth level memory set 4021 provides the computing subunit with the computing data stored at the address, and provides the computing subunit with the algorithm so that the computing subunit executes the memory computing instruction based on the algorithm. For example, when the computing unit 4022 in the memory 402 is used to perform the computing operation of the embedded layer, the algorithm provided by the instruction decoder to the computing subunit may be the weight required to perform the computing operation of the embedded layer. The computing subunit is used to perform computing operations, such as executing the computing operations of the embedded layer, based on the computing data provided by the memory set 4021 to which it belongs and the algorithm provided by the instruction decoder. The second multiplexer is used to process the memory computing instruction based on the identifier of the memory set 4021 (such as Figure 8 BGTag in the memory set 4021), determine to send the calculation result to the calculation subunit providing the N-1 level memory set 4021, or directly output the calculation result sent by the calculation unit 4022 of the N-1 level memory set 4021. For example, the memory calculation instruction carries the identifier of the memory set 4021 used to process the memory calculation instruction (such as Figure 8 In BGTag), the second multiplexer is used to send the calculation result of the calculation subunit of the N-1th level memory set 4021 to the calculation subunit of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the second multiplexer belongs, and to send the calculation result (such as) to the calculation unit 4022 of the N-1th level memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 included in the memory set 4021 to which the first multiplexer belongs. Figure 8 The BG result in the calculation is directly outputted, such as outputting the calculation result to the output processing unit 4023 via the memory 402 data bus.

[0091] Alternatively, if Fig. 9As shown, the computing unit 4022 configured in the N-th level memory set 4021 also includes: an instruction queue and a result register. The instruction queue is used to queue the received memory computing instructions. The result register is used to temporarily store the calculation results, including temporarily storing the calculation results sent by the computing subunit, and temporarily storing the calculation results sent by the second multiplexer that need to be directly output. When data is distributed in multiple computing batches for calculation, the result register is also used to determine the timing of outputting the temporarily stored data according to which batch the data calculated by the current memory computing instruction indicated by the batch tag belongs to.

[0092] Fig.10 4 is a schematic diagram of the structure of a computing unit 4022 configured in a j-1 level memory set 4021 in a second to N-1 level memory set 4021 provided in an embodiment of the present application. Fig.10 As shown, the computing unit 4022 includes: a first multiplexer, an instruction decoder, a computing subunit, and a second multiplexer. The first multiplexer is used to determine, based on the memory computing instruction sent by the computing unit 4022 of the j-th level memory set 4021, to send the memory computing instruction to the computing unit 4022 of the j-2nd level memory set 4021 included therein (e.g. Fig.10 For example, the memory computing instruction carries an identifier of the memory set 4021 used to process the memory computing instruction (such as Fig.10In the embodiment of the present invention, the first multiplexer is used to send the memory computing instruction to the instruction decoder of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the first multiplexer belongs, and to send the memory computing instruction to the j-2nd level memory set 4021 included in the memory set 4021 to which the first multiplexer belongs. The instruction decoder is used to decode the received memory computing instruction, obtain the algorithm used to execute the memory computing instruction and the address of the required computing data, and provide the address to the j-1st level memory set 4021, so that the j-1st level memory set 4021 provides the computing data stored at the address to the computing subunit, and provides the algorithm to the computing subunit, so that the computing subunit executes the memory computing instruction based on the algorithm. For example, when the computing unit 4022 in the memory 402 is used to execute the computing operation of the embedded layer, the algorithm provided by the instruction decoder to the computing subunit can be the weight required for executing the computing operation of the embedded layer. The computing subunit is used to perform computing operations, such as computing operations of the embedded layer, based on the computing data provided by the memory set 4021 to which it belongs and the algorithm provided by the instruction decoder. The second multiplexer is used to determine, based on the identifier of the memory set 4021 that processes the memory computing instruction, whether to send the computing result to the computing subunit of the j-2 level memory set 4021, or to directly output the computing result sent by the computing unit 4022 of the j-2 level memory set 4021. For example, a memory computing instruction carries the identifier of the memory set 4021 used to process the memory computing instruction (such as Fig.10 bankTag in the bankTag), the second multiplexer is used to send the calculation result of the calculation subunit of the j-2 level memory set 4021 to the calculation subunit of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the second multiplexer belongs, and to send the calculation result (such as the calculation result) of the calculation unit 4022 of the j-2 level memory set 4021 to the calculation subunit of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 included in the memory set 4021 to which the first multiplexer belongs. Fig.10 The calculation result is directly outputted (Bank result in the j-th level memory set 4021), such as outputting the calculation result to the calculation unit 4022 through the memory 402 data bus.

[0093] Alternatively, if Fig.11As shown, the computing unit 4022 configured in the j-1 level memory set 4021 also includes: an instruction queue and a result register. The instruction queue is used to queue the received memory computing instructions. The result register is used to temporarily store the calculation results, including temporarily storing the calculation results sent by the computing subunit, and temporarily storing the calculation results sent by the second multiplexer that need to be directly output. When data is distributed in multiple computing batches for calculation, the result register is also used to determine the timing of outputting the temporarily stored data according to which batch the data calculated by the current memory computing instruction indicated by the batch tag belongs to.

[0094] Fig.12 4 is a schematic diagram of the structure of a computing unit 4022 configured with a first-level memory set 4021 provided in an embodiment of the present application. Fig.12 As shown, the computing unit 4022 includes: an instruction decoder and a computing subunit. The instruction decoder is used to decode the received memory computing instruction, obtain the algorithm used to execute the memory computing instruction and the address of the required computing data, and provide the address to the first-level memory set 4021, so that the first-level memory set 4021 provides the computing subunit with the computing data stored at the address, and provides the algorithm to the computing subunit, so that the computing subunit executes the memory computing instruction based on the algorithm. For example, when the computing unit 4022 in the memory 402 is used to perform the computing operation of the embedded layer, the algorithm provided by the instruction decoder to the computing subunit can be the weight required to perform the computing operation of the embedded layer. The computing subunit is used to perform computing operations, such as executing the computing operation of the embedded layer, based on the computing data provided by the memory set 4021 to which it belongs and the algorithm provided by the instruction decoder.

[0095] Alternatively, if Fig.13 As shown, the computing unit 4022 configured in the first-level memory set 4021 also includes: an instruction queue and a result register. The instruction queue is used to queue the received memory computing instructions. The result register is used to temporarily store the calculation results sent by the computing subunit, and output the results when the output conditions are met, such as outputting the calculation results to the computing unit 4022 of the second-level memory set 4021 through the memory 402 data bus.

[0096] The following describes an implementation method in which the memory set 4021 provides the computing subunit with the data required for calculation based on the address provided by the instruction decoder.

[0097] The memory set 4021 includes: multiple memory subsets, multiple local buffers, multiple control subunits and a global buffer. The multiple memory subsets correspond to the multiple local buffers. For example, the multiple memory subsets correspond to the multiple local buffers one by one. The multiple memory subsets correspond to the multiple control subunits. For example, the multiple memory subsets correspond to the multiple control subunits one by one. The memory subset includes multiple memory blocks arranged in an array.

[0098] The control subunit corresponding to any memory subset is used to select a target first memory unit from multiple first memory units included in any memory subset based on the address of the data required for calculation, determine the target first memory unit's right to use the global buffer, and select a target second memory unit from multiple second memory units included in the target first memory unit, one of the first memory unit and the second memory unit is a row and the other is a column. The local buffer corresponding to any memory subset is used to cache the data stored in the memory block in the target first memory unit. The global buffer is used to cache the target data stored in the memory block of the target second memory unit with the right to use the global buffer. The calculation unit 4022 includes a calculation subunit, which is used to obtain the target data and perform calculation operations based on the target data. Among them, when multiple control subunits in the memory set 4021 determine the right to use the global buffer for the target first memory unit, the multiple control subunits need to ensure that at the same time there is a target first memory unit with the right to use the global buffer to avoid data transmission conflicts and ensure the correct transmission of data stored in the global buffer.

[0099] The implementation method of the control subunit provided in the embodiment of the present application enables multiple memory subsets in the memory set 4021 to simultaneously activate a target first memory unit, and cache the selected target first memory unit in a local buffer corresponding to the memory subset, cache it to the global buffer when the target first memory unit has the right to use the global buffer, and provide the target second memory unit stored in the global buffer to the computing subunit, so that the selected target first memory unit is not necessarily transmitted to the computing subunit, thereby realizing the separation of data activation and data transmission.

[0100] Fig.14 4021 is a schematic diagram of the structure of a memory set 4021 provided in an embodiment of the present application. Fig.14 As shown, the memory set 4021 includes: a global buffer, multiple memory subsets, multiple local buffers corresponding to the multiple memory subsets, and multiple control subunits corresponding to the multiple memory subsets. Fig.14 In the example, the first memory unit is a row, the second memory unit is a column, and the memory subset is a memory block subarray including multiple rows and columns of memory blocks. Then the global buffer is also called a global row buffer, and the local buffer is also called a row buffer.

[0101] like Fig.14 As shown, the control subunit corresponding to any memory subarray is used to select a target row from the multiple rows included in the memory subarray based on the address of the data required for calculation, determine the target row's right to use the global row buffer, and select a target column from the multiple columns included in the target row. After selecting the target row in the memory subarray, the data stored in the memory block in the target row can be transferred to the row buffer corresponding to the memory subarray, and the data stored in the memory block in the target row can be cached in the row buffer corresponding to the memory subarray. After determining that the target row has the right to use the global row buffer, the target column can be selected from the multiple columns included in the target row, and then the data stored in the memory block in the target column can be transferred to the global row buffer, and the data stored in the memory block in the target column can be cached in the global row buffer. The target data stored in the memory block in the target column can be read from the global row buffer, and the target data can be provided to the calculation unit 4022. The calculation unit 4022 includes a calculation subunit, which is used to obtain the target data and perform calculation operations based on the target data.

[0102] It should be noted that, in order to facilitate the memory block to identify the address of the data, each memory block subarray is also configured with a row decoder, and each level of memory set 4021 is also configured with a column decoder. Fig.14 As shown, the row decoder is used to decode the signal sent by the control subunit and provide the decoded signal to the memory subset. The column decoder is used to decode the signal sent by the instruction decoder and provide the decoded signal to the global buffer.

[0103] In one implementation, the control logic of the control subunit is as follows: Fig.15 As shown, the control subunit has four input signals: row address signal, memory subarray identifier, subarray selection signal and column selection signal. When the row address signal is input, it will be first stored in the temporary register. When the input memory subarray identifier is consistent with the identifier of the memory subarray corresponding to the control subunit, the row address signal will be transmitted to the row decoder, and the row indicated by the row address signal will be activated, that is, the row will be selected as the target row, and the data stored in the memory block in the target row will be transmitted to the row buffer corresponding to the memory subarray. The subarray selection signal is used to indicate whether the memory subarray corresponding to the control subunit has the right to use the global buffer to avoid data transmission conflicts. The column selection signal is used to select the target column from the multiple columns included in the target row. Among them, the row address signal, the memory subarray identifier, the subarray selection signal and the column selection signal are obtained based on the address provided by the instruction decoder.

[0104] In the embodiment of the present application, the implementation of the computing subunit can be determined according to the computing requirements. For example, when the computing subunit is used to perform computing operations of the embedded layer, such as Fig.16As shown, the computing subunit may include multiple circular accumulators. The input of the computing subunit is the embedding vector and its corresponding weight scalar, and the circular accumulator is used to multiply the weight and the embedding vector, and then accumulate the product with the previous result. Optionally, the computing subunit also includes a register, which is used to temporarily store the calculation result. It should be noted that the Fig.16 The implementation of the computing subunit shown is an example, and its implementation can be changed according to the computing requirements. For example, when the computing subunit is used to obtain the mean, weight sum, maximum value and minimum value of the embedded vector, the implementation of the computing subunit can be changed accordingly.

[0105] The embodiment of the present application further provides a memory calculation instruction, which compresses all instructions and address information into one instruction and transmits it to the calculation unit 4022 of the memory 402 . Fig.17 Schematic diagram of a memory calculation instruction provided in an embodiment of the present application. Fig.17As shown, the memory calculation instruction includes the following fields: a bit operation code, a memory operation code, a memory address and data size of the data required for calculation, an algorithm indicator code, a batch indicator code, an identification code of the memory set 4021 to which the calculation unit 4022 belongs, and a last batch indicator code. The bit operation code is used to indicate the type of calculation operation performed by the calculation unit 4022, for example, indicating whether the type of embedded operation performed is addition or weighted sum. The memory operation code is used to indicate the type of operation required for the memory 402 operation, such as indicating that the mode of the memory 402 operation is to activate a row (ACT), read (RD) and precharge (PRE). The data required for calculation can be an embedding table, whose memory address is the memory address of the embedded vector, and whose data size can be the length of the embedded vector. The algorithm indicator code is used to indicate the algorithm used for the calculation performed by the calculation unit 4022. For example, when the calculation unit 4022 performs an embedding operation, the algorithm indicator code is used to indicate the weight value used for the embedding operation. The batch indicator code (batch Tag) is used to indicate which batch the data calculated by the current memory calculation instruction belongs to when the data is distributed in multiple calculation batches for calculation. The identification code of the memory set 4021 to which the computing unit 4022 belongs is used to indicate the memory set 4021 to which the computing unit 4022 that executes the computing task indicated by the memory computing instruction belongs. Optionally, when the memory 402 includes N-level memory sets 4021, the memory computing instruction may include the identification codes of the memory sets 4021 to which N-1 computing units 4022 corresponding to the N-level memory sets 4021 belong, and the identification code corresponding to any memory set 4021 is used to indicate whether the memory set 4021 to which the computing unit 4022 that executes the computing task indicated by the memory computing instruction belongs is the memory set 4021. For example, when the memory 402 includes a rank, a bank group, and a bank memory set 4021, the memory computing instruction includes two identification codes of the memory set 4021 to which the computing units 4022 belong, which are BGTag and bankTag, respectively. BGTag is used to indicate whether the memory set 4021 to which the computing unit 4022 that executes the computing task indicated by the memory computing instruction belongs is a rank or a bank group, and bankTag is used to indicate whether the memory set 4021 to which the computing unit 4022 that executes the computing task indicated by the memory computing instruction belongs is a bank group or a bank. The last batch indicator code (lastTag) is used to indicate whether the data calculated by the current memory computing instruction belongs to the last batch when the data is distributed in multiple computing batches for calculation. For example, when the last batch indicator code indicates that the data belongs to the last batch, it means that the current embedding operation has ended. The size of each field in the memory computing instruction can be determined according to application requirements.For example, the bit operation code can be 3 bits, the memory operation code can be 3 bits, the memory address of the data required for calculation can be 34 bits, the data size of the data required for calculation can be 3 bits, the algorithm indicator code can be 32 bits, the batch indicator code can be 4 bits, the identification code of the memory collection 4021 to which each computing unit 4022 belongs can be 1 bit, and the last batch indicator code can be 1 bit.

[0106] Fig.18 It is a schematic diagram comparing the acceleration ratios of the related art and the memory system 40 of the present application at different embedding vector lengths. Fig.19 4 is a schematic diagram comparing the speedup ratios of the related art and the memory system 40 of the present application at different embedding operation batch sizes. Fig. 20 1 is a schematic diagram comparing the energy consumption of the memory system 40 of the related art and the present application. Table 1 is a comparison of the additional area overhead of the memory system 40 of the related art and the present application. Fig.18 , Fig.19 and Fig. 20 In the present application, the memory system 40 has three levels of memory sets 4021, namely, rank, bank group and bank. The data capacity ratio of memory blocks in the rank, bank group and bank-level memory sets 4021 is 4:3:1, and each rank is configured with a computing unit 4022, and some bank groups and some banks are configured with computing units 4022. Fig.18 and Fig.19 In the figure, the rectangle filled with dots represents the CPU baseline solution, in which no computing unit 4022 is provided in the memory 402. The rectangle filled with horizontal lines represents the TensorDIMM solution, in which a computing unit 4022 is provided in each rank. The rectangle filled with vertical lines represents the RecNMP solution, in which a computing unit 4022 is provided in each rank and an additional cache is provided. The rectangle filled with slashes represents the NMP TRiM-G solution, in which a computing unit 4022 is provided in each bank group. The unfilled rectangle represents the NMP TRiM-B solution, in which a computing unit 4022 is provided in each bank group and each bank. The rectangle filled with black represents the solution of the present application (referred to as ReCross).

[0107] like Fig.18 and Fig.19As shown, the present application scheme has 9.3 times and 7.9 times performance improvement compared to TensorDIMM and RecNMP with computing units 4022 configured at the Rank level, respectively, and has about 2.5 times and 1.8 times performance improvement compared to NMP TRiM-G with computing units 4022 configured at the bank group level and NMP TRiM-B with computing units 4022 configured at the bank level, respectively. The related art mainly relies on the number of computing units 4022 configured in the memory 402 to improve the parallelism and internal bandwidth of the memory 402, while the present application scheme provides different levels of memory 402 parallelism for different data types, making fuller use of bandwidth. In particular, for frequently accessed data, the sub-array level parallelism is also utilized, greatly reducing data access latency.

[0108] exist Fig. 20 In the figure, "1" represents the energy consumption of the computing unit, "2" represents the static power consumption of the computing unit, "3" represents the energy consumption of the input / output of off-chip data, and "4" represents the energy consumption of the on-chip memory access. The black-filled part of the energy consumption of the present application solution represents the energy consumption of the sub-array parallelism. Fig. 20 As shown, compared with CPU baseline, TensorDIMM, RecNMP, TRiM-G and TRiM-B, the solution of the present application reduces energy consumption by 58.5%, 57.2%, 51.9%, 28.5% and 23.7% respectively. This is mainly because the sub-array level parallelism in the present application reduces data access latency, and most of the embedding operations are completed within the bank, reducing off-chip I / O energy consumption. In addition, the shorter execution time also reduces the static energy consumption of the memory 402 subsystem.

[0109] As shown in Table 1, since TensorDIMM and RecNMP are only configured with computing units 4022 in the memory set 4021 at the rank level, while TRiM-B, TRiM-G and ReCross are configured with computing units 4022 in the memory set 4021 at the bank group and bank levels, the additional area overhead of ReCross on DIMM is 0.34 square millimeters per DIMM and 2.35 square millimeters per DRAM chip, which is slightly different from TRiM-G because they have the same number of computing units 4022, but much smaller than TRiM-B which is equipped with more computing units 4022.

[0110] Table 1

[0111]

[0112]

[0113] As can be seen from the above, in the memory system provided in the embodiment of the present application, since each N-th level memory set is configured with a computing unit, some of the levels or some of the memory sets in all the levels of the first to N-1 level memory sets are also configured with the computing unit, which increases the total number of computing units configured in the memory, which helps to further improve the computing performance of the memory. In addition, since multiple memory subsets in the memory set have parallelism, by configuring computing units for some of the first-level memory sets, it is possible to take into account the parallelism of computing, the load balancing of computing, the utilization of computing resources, the chip area and the power consumption overhead, and can effectively improve the computing performance of the memory system. For example, compared with the related technology, it can achieve a performance improvement of 1.8 to 2.5 times, configure fewer computing units, and save memory chip area and energy consumption overhead.

[0114] It should be noted that the components in the memory system provided in the embodiment of the present application can be increased or decreased accordingly. Any technician familiar with the technical field can easily think of a change method within the technical scope disclosed in this application, which should be included in the protection scope of this application, so it will not be repeated.

[0115] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements the memory method provided in the embodiment of the present application.

[0116] The embodiment of the present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer implements the memory method provided by the embodiment of the present application.

[0117] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0118] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image information and executable code involved in this application are all obtained with full authorization.

[0119] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means a plurality, unless otherwise expressly defined.

[0120] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0121] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0122] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0123] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A memory system, characterized in that: The memory system comprises: a memory controller and a memory; The memory controller is used to manage the memory; The memory has N levels of memory sets, each memory set includes multiple memory blocks, the i+1-th level memory set in the N-level memory sets includes multiple i-th level memory sets, and in the N-level memory sets, each N-th level memory set is configured with a computing unit, and some memory sets in some levels or all levels of the first to N-1 level memory sets are configured with the computing units, and the computing units are used to perform computing operations based on data stored in the memory sets, and N and i are positive integers.

2. The system according to claim 1, characterized in that Part of the first level memory set is configured with the computing unit.

3. The system according to claim 1 or 2, characterized in that: In the second to N-1 level memory sets, at least some memory sets included in any level memory set configured with the computing unit are not configured with the computing unit.

4. The system according to any one of claims 1 to 3, characterized in that: The memory controller is specifically used to provide computing instructions to the computing unit; The computing unit is specifically used to obtain data required for the calculation from the memory set to which the computing unit belongs based on the computing instruction, and perform the calculation based on the data.

5. The system according to claim 4, characterized in that The requirement for computing parallelism of the data stored in the i-th level memory set configured with the computing unit is higher than the requirement for computing parallelism of the data stored in the i-th level memory set not configured with the computing unit, and the computing parallelism is obtained based on one or more of the following: the access frequency of the data, the size of the data set to which the data belongs, the rated amount of data processed by each computing batch of the computing unit, and the rated capacity and rated bandwidth of the N-level memory set.

6. The system according to any one of claims 1 to 5, characterized in that: The computing unit of the i+1th level memory set is specifically used for: When the computing task indicated by the computing instruction does not belong to the computing unit of the (i+1)th level memory set, providing the computing instruction to the computing unit of the i-th level memory set included in the (i+1)th level memory set; When the computing task indicated by the computing instruction belongs to the computing unit of the (i+1)th level memory set, the computing task indicated by the computing instruction is executed.

7. The system according to claim 6, characterized in that The computing unit of the i+1th level memory set is further specifically used for: When the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set, the computing unit of the i-th level memory set included in the i+1th level memory set receives the computing result provided by the computing unit, processes the computing result and outputs the processed computing result.

8. The system according to claim 7, characterized in that When the memory has multiple N-th level memory sets, the memory further includes: an output processing unit, which is used to process the calculation results of the calculation units of the multiple N-th level memory sets and output the processed calculation results.

9. The system according to claim 8, characterized in that The memory controller is specifically used to send the calculation instruction to the output processing unit; The output processing unit is specifically used to send the computing instruction to the Nth level memory set to which the computing unit used to process the computing instruction belongs.

10. The system according to any one of claims 1 to 9, characterized in that: The memory set includes a plurality of memory subsets, a plurality of local buffers, a plurality of control subunits and a global buffer, the plurality of memory subsets correspond to the plurality of local buffers, the plurality of memory subsets correspond to the plurality of control subunits, and the memory subsets include a plurality of memory blocks arranged in an array; The control subunit corresponding to any memory subset is used to select a target first memory unit from a plurality of first memory units included in any memory subset based on the address of the data required for calculation, determine the use authority of the target first memory unit to the global buffer, and select a target second memory unit from a plurality of second memory units included in the target first memory unit, wherein one of the first memory unit and the second memory unit is a row and the other is a column; The local buffer corresponding to any one of the memory subsets is used to cache the data stored in the memory block in the target first memory unit; The global buffer is used to cache target data stored in a memory block in a target second memory unit having the right to use the global buffer; The computing unit includes a computing subunit, which is used for the target data and performs the computing operation based on the target data.

11. A computing device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program instructions, and the processor executes the program instructions to implement the memory system according to any one of claims 1 to 10.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device implements the memory system according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device implements the memory system according to any one of claims 1 to 10.

Images