Memory system

Abstract

The present application relates to the technical field of memories, and discloses a memory system. The memory system comprises a memory controller and a memory. The memory controller is used for managing the memory. The memory has N levels of memory sets; each memory set comprises a plurality of memory blocks; an (i+1)th-level memory set among the N levels of memory sets comprises a plurality of ith-level memory sets; among the N levels of memory sets, each Nth-level memory set is configured with a processing element; some memory sets among some or all of the first to (N-1)th levels of memory sets are configured with processing elements; and the processing elements are used for executing a computation operation on the basis of data stored in the memory sets, N and i being positive integers. The present application helps to further improve the computing performance of the memory.

Description

Memory system

[0001] This application claims priority to Chinese patent application No. 202310723108.2, filed on June 16, 2023, with the invention name “Memory System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of memory technology, and in particular to a memory system. Background Art

[0003] Near memory processing (NMP) technology has emerged in recent years. By integrating a near memory processing element (NMP PE) within memory, NMP technology can perform computations directly on data retrieved from memory, eliminating the need to fetch the data from memory to the central processing unit (CPU) for processing. This significantly reduces the data transfer overhead between the CPU and memory.

[0004] Memory has N levels of memory collections. Currently, a near-memory computing unit is typically configured for each top-level memory collection. This near-memory computing unit is responsible for computing data stored in that memory collection and all included memory collections. Having near-memory computing units configured for multiple top-level memory collections enables parallel execution of computing operations.

[0005] However, due to the limited number of memory sets at the highest level in memory, this method of configuring near-memory computing units has very limited effect on improving memory computing performance.

[0006] Summary of the Invention

[0007] This application provides a memory system. This application helps further improve the computing performance of memory. The technical solutions provided by this application are as follows:

[0008] In a first aspect, the present application provides a memory system. The memory system includes: a memory controller and memory; the memory controller is configured to manage the memory; the memory has N levels of memory sets, each memory set including multiple memory blocks, the i+1th level memory set in the N levels of memory sets including multiple ith level memory sets, each level N memory set in the N levels of memory sets is configured with a computing unit, and some or all levels of the first to N-1 levels of memory sets are configured with computing units, the computing units being configured to perform computing operations based on data stored in the memory sets, where N and i are positive integers.

[0009] In this way, in this memory system, not only is each Nth level memory set configured with a computing unit, but some levels or some memory sets in all levels of the first to N-1 level memory sets are also configured with the computing units, which increases the total number of computing units configured in the memory and helps to further improve the computing performance of the memory.

[0010] Among them, some first-level memory sets are configured with computing units. The parallelism of multiple memory subsets can ensure the computational parallelism of the computing units configured for these multiple memory subsets. However, considering the characteristics of the data required for calculation, such as the different access frequencies of different data, if computing units are configured for all first-level memory sets, problems such as uneven load balancing and low computing resource utilization will occur, and this will result in larger chip area and power consumption overhead. Therefore, by configuring computing units for some first-level memory sets, it is possible to balance computational parallelism, computational load balancing, computing resource utilization, chip area, and power consumption overhead.

[0011] In one implementation, among the second to N-1 level memory sets, any level memory set configured with computing units includes at least some memory sets that are not configured with computing units. This allows computing units configured in the second level memory set to process computing operations on data stored in the first level memory sets that are not configured with computing units, while computing operations on data stored in the first level memory sets that are configured with computing units are performed by computing units in the first level memory sets. This ensures that computing operations on data stored in each memory set can be performed using the computing units configured in the memory sets, and avoids the issues of increased chip area and power consumption caused by configuring computing units for each memory set.

[0012] In the memory system of the present application, the memory controller is specifically used to provide computing instructions to the computing unit; the computing unit is specifically used to obtain the data required for the calculation from the memory set to which the computing unit belongs based on the computing instructions, and perform calculations based on the data.

[0013] In one implementation, the data stored in the i-th level memory set configured with computing units has a higher requirement for computing parallelism than the data stored in the i-th level memory set not configured with computing units, and the computing parallelism is obtained based on one or more of the following: the access frequency of the data, the size of the data set to which the data belongs, the rated amount of data processed by each computing batch of the computing unit, the rated capacity and rated bandwidth of the N-level memory set.

[0014] Optionally, the computing unit of the i+1th level memory set is specifically used to: provide computing instructions to the computing units of the i-th level memory set included in the i+1th level memory set when the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set; and execute the computing task indicated by the computing instruction when the computing task indicated by the computing instruction belongs to the computing unit of the i+1th level memory set.

[0015] It should be noted that, in some computing logics, when the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set, the computing unit of the i+1th level memory set is specifically further used to: when the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set, receive the computing results provided by the computing units of the i-th level memory set included in the i+1th level memory set, process the computing results and output the processed computing results.

[0016] Since the N-th level memory set is the highest level memory set in the memory, the output of the memory at the same time is limited. Therefore, when the memory has multiple N-th level memory sets, the memory also includes: an output processing unit, which is used to process the calculation results of the calculation units of multiple N-th level memory sets and output the processed calculation results.

[0017] At this time, the memory controller is specifically used to send the computing instruction to the output processing unit; the output processing unit is specifically used to send the computing instruction to the Nth level memory set to which the computing unit for processing the computing instruction belongs.

[0018] In one implementation, a memory set includes multiple memory subsets, multiple local buffers, multiple control subunits, and a global buffer. The multiple memory subsets correspond to the multiple local buffers, and the multiple memory subsets correspond to the multiple control subunits. The memory subsets include multiple memory blocks arranged in an array. The control subunit corresponding to any memory subset is used to select a target first memory unit from the multiple first memory units included in any memory subset based on the address of the data required for calculation, determine the target first memory unit's permission to use the global buffer, and select a target second memory unit from the multiple second memory units included in the target first memory unit, where one of the first memory unit and the second memory unit is a row and the other is a column. The local buffer corresponding to any memory subset is used to cache data stored in the memory block of the target first memory unit. The global buffer is used to cache target data stored in the memory block of the target second memory unit that has permission to use the global buffer. The computing unit includes a computing subunit, which is used to perform computing operations based on the target data.

[0019] Among them, when multiple control sub-units in the memory set determine the target first memory unit's access to the global buffer, the multiple control sub-units need to ensure that at the same time there is a target first memory unit that has access to the global buffer to avoid data transmission conflicts and ensure the correct transmission of data stored in the global buffer.

[0020] The implementation method of the control subunit provided in the present application enables multiple memory subsets in a memory set to simultaneously activate a target first memory unit, and cache the selected target first memory unit in a local buffer corresponding to the memory subset, cache it to the global buffer when the target first memory unit has the right to use the global buffer, and provide the target second memory unit stored in the global buffer to the computing subunit, so that the selected target first memory unit is not necessarily transmitted to the computing subunit, thereby realizing the separation of data activation and data transmission.

[0021] In a second aspect, the present application provides a computing device comprising a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the memory system provided in the first aspect of the present application and any possible implementation thereof.

[0022] In a third aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device executes the memory system provided in the first aspect of the present application and any possible implementation thereof.

[0023] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when run on a computer, enables the computer to execute the memory system provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a schematic structural diagram of a memory system provided in an embodiment of the present application;

[0025] FIG2 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0026] FIG3 is a schematic diagram of the structure of a recommendation model based on deep learning provided in an embodiment of the present application;

[0027] FIG4 is a schematic diagram of another memory system provided in an embodiment of the present application;

[0028] FIG5 is a schematic diagram of another memory system provided by an embodiment of the present application;

[0029] FIG6 is a schematic structural diagram of an output processing unit provided in an embodiment of the present application;

[0030] FIG7 is a schematic structural diagram of another output processing unit provided in an embodiment of the present application;

[0031] FIG8 is a schematic diagram of the structure of a computing unit configured with an N-th level memory set according to an embodiment of the present application;

[0032] FIG9 is a schematic diagram of the structure of a computing unit configured with another N-level memory set according to an embodiment of the present application;

[0033] FIG10 is a schematic diagram of the structure of a computing unit configured in a j-1th level memory set in a second to N-1 level memory set according to an embodiment of the present application;

[0034] 11 is a schematic diagram of the structure of a computing unit configured in a j-1th level memory set in another second to N-1 level memory sets provided by an embodiment of the present application;

[0035] FIG12 is a schematic diagram of the structure of a computing unit configured with a first-level memory set according to an embodiment of the present application;

[0036] FIG13 is a schematic diagram of the structure of a computing unit configured with another first-level memory set according to an embodiment of the present application;

[0037] FIG14 is a schematic diagram of the structure of a memory set provided in an embodiment of the present application;

[0038] FIG15 is a schematic structural diagram of a control subunit provided in an embodiment of the present application;

[0039] FIG16 is a schematic diagram of the structure of a computing subunit provided in an embodiment of the present application;

[0040] FIG17 is a schematic diagram of the structure of a memory calculation instruction provided in an embodiment of the present application;

[0041] FIG18 is a schematic diagram comparing the speedup ratios of a related technology provided by an embodiment of the present application and the memory system of the present application at different embedded vector lengths;

[0042] FIG19 is a schematic diagram comparing the speedup ratios of a related technology provided by an embodiment of the present application and the memory system of the present application at different embedding operation batch sizes;

[0043] FIG20 is a schematic diagram comparing the energy consumption of a related technology provided in an embodiment of the present application and the memory system of the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0045] To facilitate understanding, the technology and background involved in the embodiments of this application are explained below.

[0046] Near memory processing (NMP) technology has emerged in recent years. By integrating a near memory processing element (NMP PE) within memory, NMP technology can perform computations directly on data retrieved from memory, eliminating the need to fetch the data from memory to the central processing unit (CPU) for processing. This significantly reduces the data transfer overhead between the CPU and memory.

[0047] Memory consists of N ranks. As shown in Figure 1, the host connects to the memory through a memory controller. Each memory controller controls an independently accessible memory channel. Each memory channel connects to multiple memory ranks that share a common instruction and data bus. These ranks are physically housed in dual inline memory modules (DIMMs). A single DIMM can accommodate multiple ranks to maximize memory parallelism and meet the requirements of large memory capacity. Because dynamic random access memory (DRAM) chips (such as the chip in Figure 1) have a limited input / output (I / O) data width (typically 4, 8, or 16 bits), multiple DRAM chips are typically combined into a single rank to achieve a 64-bit data width. Each DRAM chip internally consists of multiple memory bank groups, each of which contains multiple memory banks. The dashed boxes in Figure 1 indicate bank groups. A bank is a physical storage unit within the memory, representing the structure in which data is actually stored. Both bank groups and ranks are logical concepts for managing banks. Data stored in a bank group or rank is actually stored in the banks it contains. Figure 1 is a schematic diagram of a DDR-based memory system, but most current memory systems have similar structures. You can refer to Figure 1 for more information.

[0048] Logically, a bank can be viewed as a memory cell array with a row decoder, column decoder, row buffer, and multiple sub-arrays. However, implementing a large memory cell array often results in long bit lines and word lines, which poses a great challenge to signal reliability and delay. Therefore, today's banks are usually composed of multiple small memory cell arrays, each of which is usually 512x512 in size. As shown in Figure 1, all arrays in the horizontal direction share the same row decoder and global word line. Such a row of arrays is called a subarray. In DRAM operation, only one row in one of the sub-arrays in a bank can be activated at a time for read or write operations, so that continuous access to the same bank must be serial operations, which leaves ample opportunities for utilizing higher levels of memory parallelism.

[0049] Currently, a near-memory computing unit is typically configured for each highest-level memory set. The near-memory computing unit of the memory set is responsible for computing the data stored in the memory set and all the memory sets it contains. Configuring near-memory computing units for multiple highest-level memory sets enables parallel execution of computing operations. For example, a near-memory computing unit is configured for each rank. This near-memory computing unit is responsible for computing the data stored in the rank and its included bank groups and banks, and the near-memory computing units configured for multiple ranks execute computing operations in parallel.

[0050] However, due to the limited number of memory sets at the highest level in memory, this method of configuring near-memory computing units has very limited effect on improving memory computing performance.

[0051] To this end, an embodiment of the present application provides a memory system. The memory system includes: a memory controller and a memory. The memory controller is used to manage the memory. The memory has N levels of memory sets. Each memory set includes multiple memory blocks. The i+1th level memory set in the N levels of memory sets includes multiple i-th level memory sets. In the N levels of memory sets, each Nth level memory set is configured with a computing unit, and some memory sets in some or all levels of the first to N-1 levels of memory sets are configured with the computing unit. The computing unit is used to perform computing operations based on data stored in the memory set. N and i are positive integers.

[0052] In this way, in this memory system, not only is each Nth level memory set configured with a computing unit, but some levels or some memory sets in all levels of the first to N-1 level memory sets are also configured with the computing units, which increases the total number of computing units configured in the memory and helps to further improve the computing performance of the memory.

[0053] The implementation environment involved in the memory system provided in the embodiment of the present application includes: a computing device. The computing device is configured with the memory system provided in the embodiment of the present application. The computing device can be a server or a terminal, etc. Figure 2 is a structural diagram of a computing device provided in the embodiment of the present application. As shown in Figure 2, the computing device 20 includes: a processor 201, a communication interface 202, an external memory 203, a memory (also called physical memory) 204, a basic input output system 205 (basic input output system, BIOS) and a bus 206. The memory 204 is configured with a computing unit 1041. Among them, the processor 201, the communication interface 202, the external memory 203, the memory 204 and the basic input output system 105 are interconnected via a bus 206.

[0054] Processor 201 can be a dedicated hardware chip. A dedicated hardware chip is a high-performance processing hardware module. Dedicated hardware chips include digital signal processors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or a combination thereof. The above-mentioned PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. Processor 201 can also be a general-purpose processor, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), a network processor (NP), or a combination of a CPU and an NP. The CPU is, for example, a single-core processor (single-CPU) or a multi-core processor (multi-CPU).

[0055] The communication interface 202 uses a transceiver module, such as, but not limited to, a transceiver, to implement communication with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. For example, the communication interface 202 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, or other device with network access capabilities.

[0056] The external memory 203 is used to store executable code or data generated when executing an application program. The executable code stored in the external memory 203 can be read into the internal memory and executed by the processor 201 to implement the function corresponding to the executable code. Similarly, the data stored in the external memory 203 can be read into the internal memory and accessed by the processor 201 to perform corresponding processing based on the data. Alternatively, the executable code stored in the external memory 203 can be read into the internal memory and executed by the computing unit 1041 to implement the computing function corresponding to the executable code. Similarly, the data stored in the external memory 203 can be read into the internal memory and accessed by the computing unit 1041 to perform corresponding computing operations based on the data.

[0057] The external memory 203 may be, for example, a read-only memory or other type of static storage device capable of storing static information and instructions, a random access memory or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired executable code in the form of instructions or data structures and accessible by a computer, but not limited thereto. For example, the external memory 203 is used to store an outbound port queue, etc. The external memory 203 may exist independently and be connected to the processor 201 via the bus 206. Alternatively, the external memory 203 and the processor 201 may be integrated together. For example, the external memory 203 may include a volatile memory, such as a random-access memory (RAM); the external memory 203 may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or, the external memory 203 may also include a combination of the above types of memory.

[0058] The memory 204 is used to temporarily store data required by the processor 201 during operation, such as reading the operating system from the external memory 203 into the memory 204 when the computing device is started, and data generated when executing the application. Since the memory 204 has a relatively fast read and write speed, the memory 204 is usually used as a cache between the external memory 203 and the processor 201. That is, during the operation of the computing device, the data required by the processor 201 can be read from the external memory 203 into the memory 204 in advance, so that the processor 201 reads the required data from the memory 204 to ensure the computing speed of the processor 201. In the embodiment of the present application, the memory 204 is configured with a computing unit 1041, which can read the data stored in the memory 204 and perform corresponding computing operations based on the data.

[0059] The BIOS 205 is used to perform self-test and initialization on various components within the computing device, and serves as a transmission medium between software programs and hardware to transmit information between the software programs and hardware.

[0060] The bus 206 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., external memory 203, processor 201, communication interface 202). For example, the bus 206 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG2 , but this does not mean that there is only one bus or one type of bus. The embodiment of the present application is illustrated by the example of the above-mentioned devices inside the computing device being interconnected via the bus 206. Optionally, the above-mentioned devices inside the computing device 20 can also be connected to each other in communication using other connection methods besides the bus 206. For example, the above-mentioned devices inside the computing device 20 are interconnected via an internal logical interface.

[0061] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether each device is independently arranged on different chips or integrated on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For parts that are not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0062] In the above embodiments, all or part of the embodiments may be implemented through software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing a program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, all or part of the functions of the memory method provided in the embodiments of the present application are implemented in whole or in part.

[0063] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0064] In one implementation, the memory system provided by the embodiment of the present application can be used to accelerate computing operations. Acceleration here refers to the execution of computing operations based on data in the memory by a computing unit configured in memory. Since there is no need to read the data from the memory to the CPU, the overall time consumed by the calculation is shortened, thereby achieving the effect of accelerating the calculation. For example, the memory system can be used to accelerate the computing operations of the embedded layer in the recommendation system, that is, the computing units in the memory system are used to execute the computing operations of the embedded layer. In this way, there is no need to read the data required for the embedded layer calculation from the memory to the CPU for processing, which shortens the time consumed by the embedded layer to execute the calculation, thereby achieving the purpose of accelerating the calculation.

[0065] Due to their industrial importance, recommender systems have become essential internet infrastructure. They have been widely deployed in various applications, including e-commerce, search engines, media services, and social networks. To provide fast, accurate, and personalized recommendation services, deep learning has been applied to recommender systems and has attracted increasing attention. Figure 3 illustrates a deep learning-based recommendation model. As shown in Figure 3, the recommendation model consists of two fully connected layers (fully connected layer 1 and fully connected layer 2) and an embedding layer with multiple embedding tables. Fully connected layer 1 processes dense input containing personal information. Personal information includes, for example, user age, gender, and other preferences. The embedding layer processes sparse input containing behavioral information collected from past user interactions. Behavioral information includes, for example, information about items purchased and browsed by the user and the interactions generated during that time. The outputs of fully connected layer 1 and the embedding layer are combined as the input to fully connected layer 2 to determine the user's preference for the current scenario.

[0066] The embedding layer needs to perform computations based on a number and size of embedding tables (depending on the recommendation model). Each embedding table is a two-dimensional matrix containing millions of rows. For example, each row in the embedding table is an embedding vector with dimensions ranging from 32 to 256. The embedding layer looks up the embedding table based on a set of indices and then performs aggregation operations (e.g., direct addition, averaging, weighted sum, etc.) based on the embedding vectors in the embedding table (typically 20 to 80).

[0067] Compared with the fully connected layer, the embedding layer exhibits the following characteristics: (1) Large data volume. Compared with the fully connected layer that occupies a relatively small storage space, for recommendation models in production environments, the embedding layer can occupy hundreds of gigabytes (GB) to several terabytes (TB) of storage space. Therefore, such a large embedding table is usually stored in a large-capacity memory; (2) Low computational density. Compared with the computationally intensive fully connected layer, the embedding layer often performs simple operations, with a computational density of 0.25 FLOPs / Byte, which is much lower than the 5.5 FLOPs / Byte of the recurrent neural network (RNN), the 18 FLOPs / Byte of the fully connected (FC) network structure, and the 10 FLOPs / Byte of the convolutional neural network. (1) The embedding layer has a performance bottleneck of 141 FLOPs / Byte, which is higher than that of the previous recommendation system. FLOPs / Byte indicates the computational density, which means how many floating-point operations are performed per Byte of memory exchange during the computation process. (2) Irregular memory access: due to the large size of the embedding table, the embedding vectors to be searched are often irregularly distributed in the embedding table. This highly irregular memory access results in a poor cache hit rate. These characteristics cause the embedding layer to become a performance bottleneck of the recommendation system, and the problem becomes worse as the amount of data grows. Therefore, the present application performs computational operations of the embedding layer through the computing units in the memory system, which greatly reduces the data transmission between the CPU and the memory, and can utilize the parallelism of the computing units in the memory system to ensure computing performance.

[0068] It should be understood that the above content is an illustrative description of the application scenarios of the memory method provided in the embodiments of the present application, and does not constitute a limitation on the application scenarios of the memory method. A person skilled in the art will know that as business needs change, its application scenarios can be adjusted according to application needs, and the embodiments of the present application do not list them one by one.

[0069] The memory system provided in the embodiment of the present application is described below by way of example. FIG4 is a schematic diagram of a memory system provided in the embodiment of the present application. As shown in FIG4 , the memory system 40 includes a memory controller 401 and a memory 402 .

[0070] Memory controller 401 is used to manage memory 402. For example, memory controller 401 is used to receive a memory computing request, generate a memory computing instruction based on the memory computing request, and send the memory computing instruction to memory 402, so that memory 402 performs a corresponding computing operation based on the memory computing instruction using computing unit 4022. For another example, memory controller 401 is used to receive a memory computing instruction and distribute the memory computing instruction to memory 402, so that computing unit 4022 of memory 402 performs a corresponding computing operation based on the memory computing instruction using computing unit 4022. Memory controller 401 is also used to receive a computing result provided by computing unit 4022 in memory 402 and output the computing result.

[0071] Memory 402 has N levels of memory sets 4021, each memory set 4021 including multiple memory blocks (not shown in FIG. 4 ). The (i+1)th level memory set 4021 in the N levels of memory sets 4021 includes multiple (i)th level memory sets 4021. Within the N levels of memory sets 4021, each level N memory set 4021 is configured with a computing unit 4022, and some or all levels of the first to N-1 levels of memory sets 4021 are configured with computing units 4022, where N and i are positive integers.

[0072] In one implementation, some of the levels in the first to N-1 level memory sets 4021 or some of the memory sets 4021 in all levels are configured with computing units 4022, including at least the following situations: In the first situation, when viewed from the perspective of the level to which the memory sets 4021 belong, each level of memory sets 4021 in the first to N-1 level memory sets 4021 is configured with computing units 4022, but some levels of memory sets 4021 in the first to N-1 levels where there are one or more levels of memory sets 4021 are not configured with computing units 4022; In the second situation, when viewed from the perspective of the level to which the memory sets 4021 belong, some levels of memory sets 4021 in the first to N-1 levels are not configured with computing units 4022, and some memory sets 4021 in the remaining levels of memory sets 4021 are configured with computing units 4022.

[0073] Memory set 4021 includes multiple memory subsets, each of which includes multiple memory blocks arranged in an array, each of which is used to store data. Computing units 4022 configured within memory set 4021 are configured to perform computing operations based on the data stored within memory set 4021. For example, the data stored within the memory blocks of memory set 4021 is data required for computing. Computing units 4022 configured within memory set 4021 are configured to read the data required for computing from the memory blocks within memory set 4021 and then perform computing operations based on the data.

[0074] In this way, in the memory system 40, not only is each Nth level memory set 4021 configured with a computing unit 4022, but some memory sets 4021 in some levels or all levels of the first to N-1 level memory sets 4021 are also configured with the computing unit 4022, thereby increasing the total number of computing units 4022 configured in the memory 402, which helps to further improve the computing performance of the memory 402.

[0075] In one implementation, a portion of the first-level memory sets 4021 is configured with computing units 4022. The first-level memory sets 4021 include multiple memory subsets. Each memory subset includes multiple memory blocks arranged in an array. Because the multiple memory subsets exhibit parallelism, if an operation on one memory subset is not completed, an operation on another memory subset can be performed without waiting for the operation on the first memory subset to complete. Therefore, the parallelism of the multiple memory subsets ensures the computational parallelism of the computing units 4022 configured for the multiple memory subsets. However, considering the characteristics of the data required for computation, such as the varying access frequencies of different data, configuring all first-level memory sets 4021 with computing units 4022 would result in uneven load balancing and low computing resource utilization, as well as increased chip area and power consumption. Therefore, configuring computing units 4022 for a portion of the first-level memory sets 4021 balances computational parallelism, computational load balancing, computing resource utilization, chip area, and power consumption.

[0076] Furthermore, among the second to N-1 level memory sets 4021, at least some of the memory sets 4021 included in any level memory set 4021 configured with computing units 4022 are not configured with computing units 4022. For example, when the memory 402 has three levels of memory sets 4021, each level-3 memory set 4021 is configured with a computing unit 4022, some level-2 memory sets 4021 are configured with computing units 4022, and some level-1 memory sets 4021 are configured with computing units 4022, then some level-1 memory sets 4021 included in the level-2 memory sets 4021 configured with computing units 4022 are not configured with computing units 4022. In this way, the computing unit 4022 configured in the second-level memory set 4021 can be used to process the computing operations of the data stored in the first-level memory set 4021 that is not configured with the computing unit 4022, while the computing operations of the data stored in the first-level memory set 4021 configured with the computing unit 4022 are executed by the computing unit 4022 of the first-level memory set 4021, thereby ensuring that the computing operations of the data stored in the memory set 4021 can all be executed by the computing unit 4022 configured in the memory 402, and avoiding the problem of large chip area and power consumption overhead caused by configuring a computing unit 4022 for each memory set 4021.

[0077] The following takes the memory system 40 shown in Figure 1 as an example to illustrate the configuration of the computing unit 4022 in the memory system 40 provided in the embodiment of the present application. According to the previous description, the memory system 40 includes multiple ranks, each rank includes multiple bank groups, and each bank group includes multiple banks. That is, the memory system 40 includes three levels of memory sets 4021, the third level memory set 4021 is a rank, the second level memory set 4021 is a bank group, and the third level memory set 4021 is a bank. In the embodiment of the present application, each rank is configured with a computing unit 4022, some bank groups and some banks are configured with computing units 4022, and the bank group configured with computing units 4022 includes a bank that is not configured with computing units 4022.

[0078] In one implementation, the memory controller 401 is specifically configured to provide a computing instruction to the computing unit 4022. Optionally, the computing instruction may be generated by the memory controller 401, or received by the memory controller 401 and passed to the computing unit 4022. The computing unit 4022 is specifically configured to obtain the data required for the calculation from the memory set 4021 to which the computing unit 4022 belongs based on the computing instruction, and perform the calculation based on the obtained data. For example, the memory controller 401 is configured to receive a memory computing request, generate a memory computing instruction based on the memory computing request, and send the memory computing instruction to the memory 402, so that the memory 402 performs the corresponding computing operation based on the memory computing instruction using the computing unit 4022. Alternatively, the memory computing instruction may be generated by a driver layer or programming library of the computing device, and the memory controller 401 is configured to receive the memory computing instruction and distribute the memory computing instruction to the memory 402, so that the computing unit 4022 of the memory 402 performs the corresponding computing operation based on the memory computing instruction using the computing unit 4022.

[0079] In embodiments of the present application, when customizing a memory system for a specific usage scenario, the configuration of computing units 4022 in memory 402, i.e., which memory sets 4021 require and do not require the configuration of computing units 4022, can be determined based on the usage scenario. The usage scenario determines the data characteristics of the data involved in the calculation and the computing requirements of the usage scenario. The storage method of the data involved in the calculation in memory 402 and the configuration of computing units 4022 in memory 402 can also be determined based on the data characteristics and computing requirements.

[0080] In one implementation, the computational parallelism requirement for data stored in the i-th level memory set 4021 configured with computing units 4022 may be higher than the computational parallelism requirement for data stored in the i-th level memory set 4021 not configured with computing units 4022. Optionally, the computational parallelism is derived based on one or more of the following characteristics: data access frequency, the size of the data set to which the data belongs, the rated data volume processed by each computational batch of the computing units 4022, and the rated capacity and rated bandwidth of the N-level memory sets 4021. The rated data volume processed by each computational batch of the computing units 4022 refers to the maximum data volume that can be processed by each computational batch when the data is distributed across multiple computational batches for computation, given a large amount of data involved in the computation. For example, when memory 402 stores an embedding table and computing units 4022 perform computational operations based on the embedding table, the computational parallelism is derived based on one or more of the following characteristics: the access frequency of the embedded vectors in the embedding table, the size of the embedding table, the batch size for computations performed on the embedding table, and information such as the capacity and bandwidth of memory sets 4021 at different levels in the memory system 40.

[0081] After obtaining the above features, the data set required for the calculation can be first divided based on the above features. Then, the data in the data set is stored in memory 402 according to the division scheme. Then, based on the storage method of the data in memory 402, the degree to which the data stored in each level of memory set 4021 needs to use computing units 4022 is determined. Then, based on this degree of necessity, it is determined whether to configure computing units 4022 for the corresponding memory set 4021. In one implementation, this degree of necessity can be reflected by the data's requirement for computational parallelism. When the data's requirement for computational parallelism is higher, it is necessary to configure computing units 4022 for memory sets 4021 at lower levels within the memory sets 4021 to which the memory blocks storing the data belong. For example, taking a memory system 40 including three-level memory sets 4021 as an example, when the data has a high requirement for computational parallelism, a computing unit 4022 is configured for the first-level memory set 4021 to which the memory block storing the data belongs; when the data has a generally high requirement for computational parallelism, a computing unit 4022 is configured for the second-level memory set 4021 to which the memory block storing the data belongs; and when the data has a low requirement for computational parallelism, a computing unit 4022 is configured for the third-level memory set 4021 to which the memory block storing the data belongs. For example, in an embodiment of the present application, the data capacity ratio of the memory blocks in the rank, bank group, and bank-level memory sets 4021 is 4:3:1.

[0082] Alternatively, in a non-customized memory, the configuration of the computing units in the memory and the characteristics of the data to be stored can be first obtained. Then, based on the configuration and data characteristics, the data to be stored in different memories can be determined. The configuration of the computing units and the characteristics of the data can be referred to in the previous description and will not be repeated here.

[0083] It should be noted that after the data is stored in the memory 402, a data mapping table can also be generated for the data. The data mapping table is used to indicate the correspondence between the storage location of the data in the memory 402 and the original storage location of the data in the external memory, so that other components of the computing device can access the data based on the data mapping table.

[0084] In an embodiment of the present application, the computing unit 4022 of the i+1-level memory set 4021 is specifically used to: after obtaining a computing instruction, determine whether the computing task indicated by the computing instruction belongs to the computing unit 4022 of the i+1-level memory set 4021; when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-level memory set 4021, provide a computing instruction to the computing unit 4022 of the i-level memory set 4021 included in the i+1-level memory set 4021; when the computing task indicated by the computing instruction belongs to the computing unit 4022 of the i+1-level memory set 4021, execute the computing task indicated by the computing instruction.

[0085] It should be noted that, in some computing logics, when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-level memory set 4021, the computing unit 4022 of the i+1-level memory set 4021 is specifically further used to: when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-level memory set 4021, receive the computing result of the computing task provided by the computing unit 4022 of the i-level memory set 4021 included in the i+1-level memory set 4021, process the computing result and output the processed computing result. For example, when the computing unit 4022 performs the computing operation of the embedded layer, according to the computing logic of the embedded layer, the computing unit 4022 of the higher-level memory set 4021 needs to accumulate the computing results of the computing units 4022 of the memory set 4021 it includes. In this case, when the computing task indicated by the computing instruction does not belong to the computing unit 4022 of the i+1-level memory set 4021, the computing unit 4022 of the i+1-level memory set 4021 is also used to receive the computing result of the computing task provided by the computing unit 4022 of the i-level memory set 4021 included in the i+1-level memory set 4021, accumulate the computing result and output the accumulated result.

[0086] Since the Nth-rank memory set 4021 is the highest-rank memory set 4021 in memory 402, the output of memory 402 at any one time is limited. Therefore, when memory 402 has multiple Nth-rank memory sets 4021, as shown in FIG5 , memory 402 also includes an output processing unit 4023. This output processing unit 4023 is used to process the calculation results of the computing units 4022 of multiple Nth-rank memory sets 4021 and output the processed calculation results. FIG5 is a schematic diagram of a DIMM including two Nth-rank memory sets 4021. In this case, the output processing unit 4023 is used to process the calculation results of the computing units 4022 of the two Nth-rank memory sets 4021. For example, when memory 402 has multiple ranks, each rank is configured with a computing unit 4022. Memory 402 is also configured with an output processing unit 4023. This output processing unit 4023 is used to process the calculation results received from the computing units 4022 of multiple ranks, so that memory 402 outputs a single result at each moment. Among them, when the calculation unit 4022 of the rank performs the calculation operation of the embedded layer, the processing of the calculation result by the output processing unit 4023 can be to accumulate the calculation results of the calculation units 4022 of multiple ranks. Alternatively, when the calculation unit 4022 of the rank performs other operation logic, the processing performed by the output processing unit 4023 on the calculation result of the calculation unit 4022 can be the processing indicated by the operation logic. Alternatively, when the output processing unit 4023 does not need to perform operation logic on the calculation results of the calculation units 4022 of multiple ranks, its processing can be to queue the calculation results of multiple ranks.

[0087] Optionally, when the memory 402 includes an output processing unit 4023, the memory controller 401 is specifically used to send computing instructions to the output processing unit 4023, and the output processing unit 4023 is specifically used to send computing instructions to the Nth level memory set 4021 to which the computing unit 4022 used to process the computing instructions belongs.

[0088] The following takes the example of memory 402 including three-level memory sets 4021, and the three-level memory sets 4021 including rank, bank group and bank to illustrate the implementation method of the output processing unit 4023 and the computing unit 4022 configured in each memory set 4021.

[0089] Figure 6 is a schematic diagram of the structure of an output processing unit 4023 provided in an embodiment of the present application. As shown in Figure 6, the output processing unit 4023 includes: a physical layer (PHY) chip, a multiplexer (MUX) and a processing subunit. The physical layer chip is used to interact with the memory controller 401. For example, the physical layer chip is used to receive instructions (C / A) and data (DQ) such as memory calculation instructions sent by the memory controller 401, and output calculation results to the memory controller 401. The multiplexer is used to distribute memory calculation instructions to the calculation unit 4022 of the N-th level memory set 4021. For example, the memory calculation instruction carries an identifier of the memory set 4021 used to process the memory calculation instruction (such as the RANK ID in Figure 6), and the multiplexer is used to send the memory calculation instruction to the memory set 4021 indicated by the identifier. The processing unit is used to process the calculation results of the calculation units 4022 of multiple N-th level memory sets 4021 and output the processed calculation results.

[0090] Optionally, as shown in FIG7 , the output processing unit 4023 further includes an instruction queue, multiple first caches, and a second cache. The instruction queue is used to queue memory computing instructions received by the physical layer chip. The multiple first caches are respectively used to cache computing results sent by computing units 4022 of multiple N-th level memory sets 4021. The second caches are used to cache computing results output by the processing subunits.

[0091] In one implementation, when computing units 4022 in memory 402 are used to perform computational operations at the embedded layer, the processing subunit is configured to accumulate the computation results of multiple N-th level memory sets 4021. As shown in FIG7 , the processing subunit includes multiple accumulators. After accumulating the computation results of memory sets 4021, the processing subunit can output the accumulated results via the physical layer chip and the memory 402 data bus. FIG7 is a schematic diagram of the processing subunit accumulating the computation results of two computing units 4022.

[0092] Figure 8 is a schematic diagram of the structure of a computing unit 4022 configured in an N-th level memory set 4021 provided in an embodiment of the present application. As shown in Figure 8, the computing unit 4022 includes: a first multiplexer, an instruction decoder, a computing subunit, and a second multiplexer. The first multiplexer is used to determine, based on the memory computing instruction sent by the output processing unit 4023, whether to send a memory computing instruction (such as the BG-level memory computing instruction in Figure 8) to the computing unit 4022 of the N-1th level memory set 4021 included therein, or to send the memory computing instruction to the instruction decoder. For example, a memory computing instruction carries an identifier of a memory set 4021 for processing the memory computing instruction (such as BGTag in FIG8 ). The first multiplexer is used to send the memory computing instruction to the instruction decoder of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the first multiplexer belongs. When the memory set 4021 indicated by the identifier is the memory set 4021 included in the memory set 4021 to which the first multiplexer belongs, the memory computing instruction is sent to the N-1th level memory set 4021 included therein. The instruction decoder is used to decode the received memory computing instruction, obtain the algorithm used to execute the memory computing instruction and the address of the required computing data, and provide the address to the Nth level memory set 4021 (such as the DDR signal in FIG8 ), so that the Nth level memory set 4021 provides the computing subunit with the computing data stored at the address, and provides the computing subunit with the algorithm so that the computing subunit executes the memory computing instruction based on the algorithm. For example, when the computing unit 4022 in the memory 402 is used to perform the computing operation of the embedded layer, the algorithm provided by the instruction decoder to the computing subunit may be the weight required to perform the computing operation of the embedded layer. The computing subunit is used to perform computing operations, such as executing computing operations of the embedded layer, based on the computing data provided by the memory set 4021 to which it belongs and the algorithm provided by the instruction decoder. The second multiplexer is used to determine whether to send the computing result to the computing subunit of the N-1 level memory set 4021 based on the identifier of the memory set 4021 that processes the memory computing instruction (such as BGTag in Figure 8), or to send the computing result to the computing unit 4022 of the N-1 level memory set 4021 directly output.For example, a memory computing instruction carries an identifier of a memory set 4021 used to process the memory computing instruction (such as BGTag in Figure 8). The second multiplexer is used to send the calculation result of the calculation subunit of the N-1 level memory set 4021 to the calculation subunit of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the second multiplexer belongs. When the memory set 4021 indicated by the identifier is the memory set 4021 included in the memory set 4021 to which the first multiplexer belongs, the calculation result (such as the BG result in Figure 8) is sent to the calculation unit 4022 of the N-1 level memory set 4021 for direct output, such as outputting the calculation result to the output processing unit 4023 through the memory 402 data bus.

[0093] Optionally, as shown in FIG9 , the computing unit 4022 configured in the N-th level memory set 4021 further includes: an instruction queue and a result register. The instruction queue is used to queue received memory computing instructions. The result register is used to temporarily store the computing results, including temporarily storing the computing results sent by the computing sub-unit, and temporarily storing the computing results sent by the second multiplexer that need to be output directly. When data is distributed among multiple computing batches for calculation, the result register is also used to determine the timing of outputting the temporarily stored data based on which batch the data calculated by the current memory computing instruction indicated by the batch tag belongs to.

[0094] Figure 10 is a schematic diagram of the structure of a computing unit 4022 configured in a j-1 level memory set 4021 in a second to N-1 level memory set 4021 provided by an embodiment of the present application. As shown in Figure 10, the computing unit 4022 includes: a first multiplexer, an instruction decoder, a computing subunit, and a second multiplexer. The first multiplexer is used to determine whether to send a memory computing instruction (such as a bank-level memory computing instruction in Figure 10) to the computing unit 4022 of the j-2 level memory set 4021 included therein, or to send the memory computing instruction to the instruction decoder based on the memory computing instruction sent by the computing unit 4022 of the j-2 level memory set 4021. For example, a memory computing instruction carries an identifier of a memory set 4021 for processing the memory computing instruction (such as the bankTag in FIG10 ). The first multiplexer is used to send the memory computing instruction to the instruction decoder of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the first multiplexer belongs. When the memory set 4021 indicated by the identifier is a memory set 4021 included in the memory set 4021 to which the first multiplexer belongs, the first multiplexer is used to send the memory computing instruction to the j-2-th level memory set 4021 included therein. The instruction decoder is used to decode the received memory computing instruction, obtain the algorithm used to execute the memory computing instruction and the address of the required computing data, and provide the address to the j-1-th level memory set 4021 so that the j-1-th level memory set 4021 can provide the computing subunit with the computing data stored at the address, and provide the computing subunit with the algorithm so that the computing subunit can execute the memory computing instruction based on the algorithm. For example, when the computing unit 4022 in the memory 402 is used to perform a computing operation of the embedded layer, the algorithm provided by the instruction decoder to the computing sub-unit may be the weight required to perform the computing operation of the embedded layer. The computing sub-unit is used to perform a computing operation, such as a computing operation of the embedded layer, based on the computing data provided by the memory set 4021 to which it belongs and the algorithm provided by the instruction decoder. The second multiplexer is used to determine, based on the identifier of the memory set 4021 that processes the memory computing instruction, whether to send the computing result to the computing sub-unit of the j-2 level memory set 4021, or to directly output the computing result sent by the computing unit 4022 of the j-2 level memory set 4021.For example, a memory computing instruction carries an identifier of a memory set 4021 used to process the memory computing instruction (such as the bankTag in Figure 10), and the second multiplexer is used to send the calculation result of the calculation sub-unit of the j-2 level memory set 4021 to the calculation sub-unit of the memory set 4021 when the memory set 4021 indicated by the identifier is the memory set 4021 to which the second multiplexer belongs. When the memory set 4021 indicated by the identifier is the memory set 4021 included in the memory set 4021 to which the first multiplexer belongs, the calculation result (such as the Bank result in Figure 10) is sent to the calculation unit 4022 of the j-2 level memory set 4021 and output directly, such as outputting the calculation result to the calculation unit 4022 of the j-2 level memory set 4021 through the memory 402 data bus.

[0095] Optionally, as shown in FIG11 , the computing unit 4022 configured in the j-1 level memory set 4021 further includes: an instruction queue and a result register. The instruction queue is used to queue received memory computing instructions. The result register is used to temporarily store the computing results, including temporarily storing the computing results sent by the computing subunit, and temporarily storing the computing results sent by the second multiplexer that need to be directly output. When data is distributed among multiple computing batches for calculation, the result register is also used to determine the timing of outputting the temporarily stored data based on which batch the data calculated by the current memory computing instruction indicated by the batch tag belongs to.

[0096] Figure 12 is a structural diagram of a computing unit 4022 configured with a first-level memory set 4021 provided in an embodiment of the present application. As shown in Figure 12, the computing unit 4022 includes: an instruction decoder and a computing subunit. The instruction decoder is used to decode the received memory computing instruction, obtain the algorithm used to execute the memory computing instruction and the address of the required computing data, and provide the address to the first-level memory set 4021 so that the first-level memory set 4021 provides the computing data stored at the address to the computing subunit, and provides the algorithm to the computing subunit so that the computing subunit executes the memory computing instruction based on the algorithm. For example, when the computing unit 4022 in the memory 402 is used to perform the computing operation of the embedded layer, the algorithm provided by the instruction decoder to the computing subunit can be the weight required to perform the computing operation of the embedded layer. The computing subunit is used to perform computing operations based on the computing data provided by the memory set 4021 to which it belongs and the algorithm provided by the instruction decoder, such as executing the computing operation of the embedded layer.

[0097] Optionally, as shown in FIG13 , the computing unit 4022 configured in the first-level memory set 4021 further includes an instruction queue and a result register. The instruction queue is used to queue received memory computing instructions. The result register is used to temporarily store the calculation results sent by the computing sub-unit and output the results when the output conditions are met, such as outputting the calculation results to the computing unit 4022 of the second-level memory set 4021 via the memory 402 data bus.

[0098] The following describes how the memory set 4021 provides the computing subunit with the data required for calculation based on the address provided by the instruction decoder.

[0099] Memory set 4021 includes: multiple memory subsets, multiple local buffers, multiple control subunits, and a global buffer. The multiple memory subsets correspond to the multiple local buffers. For example, the multiple memory subsets correspond one-to-one with the multiple local buffers. The multiple memory subsets correspond to the multiple control subunits. For example, the multiple memory subsets correspond one-to-one with the multiple control subunits. The memory subsets include multiple memory blocks arranged in an array.

[0100] The control subunit corresponding to any memory subset is used to select a target first memory unit from the multiple first memory units included in any memory subset based on the address of the data required for calculation, determine the target first memory unit's access rights to the global buffer, and select a target second memory unit from the multiple second memory units included in the target first memory unit, one of the first memory unit and the second memory unit being a row and the other being a column. The local buffer corresponding to any memory subset is used to cache data stored in the memory block of the target first memory unit. The global buffer is used to cache target data stored in the memory block of the target second memory unit that has access rights to the global buffer. The computing unit 4022 includes a computing subunit, which is used to obtain target data and perform computing operations based on the target data. Among them, when the multiple control subunits in the memory set 4021 determine the access rights of the target first memory unit to the global buffer, the multiple control subunits need to ensure that at the same time there is a target first memory unit that has access rights to the global buffer to avoid data transmission conflicts and ensure the correct transmission of data stored in the global buffer.

[0101] The implementation method of the control subunit provided in the embodiment of the present application enables multiple memory subsets in the memory set 4021 to simultaneously activate a target first memory unit, and cache the selected target first memory unit in the local buffer corresponding to the memory subset, cache it to the global buffer when the target first memory unit has the right to use the global buffer, and provide the target second memory unit stored in the global buffer to the computing subunit, so that the selected target first memory unit is not necessarily transmitted to the computing subunit, thereby realizing the separation of data activation and data transmission.

[0102] FIG14 is a schematic diagram of the structure of a memory set 4021 provided in an embodiment of the present application. As shown in FIG14 , the memory set 4021 includes: a global buffer, multiple memory subsets, multiple local buffers corresponding one-to-one with the multiple memory subsets, and multiple control subunits corresponding one-to-one with the multiple memory subsets. In FIG14 , the first memory unit is a row, the second memory unit is a column, and the memory subset is a memory block subarray comprising multiple rows and columns of memory blocks. The global buffer is also referred to as a global row buffer, and the local buffer is also referred to as a row buffer.

[0103] As shown in Figure 14, the control subunit corresponding to any memory subarray is used to select a target row from the multiple rows included in the memory subarray based on the address of the data required for calculation, determine the target row's access rights to the global row buffer, and select a target column from the multiple columns included in the target row. After selecting the target row in the memory subarray, the data stored in the memory blocks in the target row can be transferred to the row buffer corresponding to the memory subarray, and the data stored in the memory blocks in the target row can be cached in the row buffer corresponding to the memory subarray. After determining that the target row has access rights to the global row buffer, a target column can be selected from the multiple columns included in the target row, and the data stored in the memory blocks in the target column can be transferred to the global row buffer, and the data stored in the memory blocks in the target column can be cached in the global row buffer. The target data stored in the memory blocks in the target column can be read from the global row buffer and provided to the calculation unit 4022. The calculation unit 4022 includes a calculation subunit, which is used to obtain the target data and perform calculation operations based on the target data.

[0104] It should be noted that to facilitate data address recognition within the memory blocks, each memory block subarray is also equipped with a row decoder, and each level of memory set 4021 is also equipped with a column decoder. As shown in Figure 14, the row decoder is used to decode signals sent by the control subunit and provide the decoded signals to the memory subset. The column decoder is used to decode signals sent by the instruction decoder and provide the decoded signals to the global buffer.

[0105] In one implementation, the control logic of the control subunit is shown in Figure 15. The control subunit has four input signals: a row address signal, a memory subarray identifier, a subarray select signal, and a column select signal. Upon input, the row address signal is first stored in a temporary register. When the input memory subarray identifier matches the identifier of the memory subarray corresponding to the control subunit, the row address signal is transmitted to the row decoder, activating the row indicated by the row address signal, i.e., selecting that row as the target row and transferring the data stored in the memory block in the target row to the row buffer corresponding to the memory subarray. The subarray select signal is used to indicate whether the memory subarray corresponding to the control subunit has access to the global buffer, thus avoiding data transmission conflicts. The column select signal is used to select a target column from the multiple columns included in the target row. The row address signal, memory subarray identifier, subarray select signal, and column select signal are derived based on the address provided by the instruction decoder.

[0106] In an embodiment of the present application, the implementation of the calculation subunit can be determined according to the operation requirements. For example, when the calculation subunit is used to perform the calculation operation of the embedding layer, as shown in Figure 16, the calculation subunit may include multiple circular accumulators. The input of the calculation subunit is the embedding vector and its corresponding weight scalar, and the circular accumulator is used to multiply the weight and the embedding vector, and then add the product to the previous result. Optionally, the calculation subunit also includes a register, which is used to temporarily store the calculation result. It should be noted that the implementation of the calculation subunit shown in Figure 16 is an example, and its implementation can change according to the operation requirements. For example, when the calculation subunit is used to obtain the mean, weight sum, maximum and minimum values ​​of the embedding vector, the implementation of the calculation subunit can change accordingly.

[0107] This embodiment of the present application also provides a memory calculation instruction. This memory calculation instruction compresses all instruction and address information into a single instruction and transmits it to the calculation unit 4022 of the memory 402. Figure 17 is a schematic diagram of the structure of a memory calculation instruction provided by an embodiment of the present application. As shown in Figure 17, the memory calculation instruction includes the following fields: a bit operation code, a memory operation code, the memory address and data size of the data required for the calculation, an algorithm indicator code, a batch indicator code, an identification code of the memory set 4021 to which the calculation unit 4022 belongs, and a last batch indicator code. The bit operation code is used to indicate the type of calculation operation performed by the calculation unit 4022, for example, indicating whether the type of embedded operation performed is addition or weighted sum. The memory operation code is used to indicate the type of operation required for the memory 402 operation, such as indicating whether the memory 402 operation mode is to activate a row (ACT), read (RD), or precharge (PRE). The data required for the calculation can be an embedding table, whose memory address is the memory address of the embedded vector, and whose data size can be the length of the embedded vector. The algorithm indicator code is used to indicate the algorithm used for the calculation performed by the calculation unit 4022. For example, when the computing unit 4022 performs an embedding operation, the algorithm indicator code is used to indicate the weight value used for the embedding operation. The batch indicator code (batch Tag) is used to indicate to which batch the data calculated by the current memory computing instruction belongs when data is distributed in multiple computing batches for calculation. The identification code of the memory set 4021 to which the computing unit 4022 belongs is used to indicate the memory set 4021 to which the computing unit 4022 that performs the computing task indicated by the memory computing instruction belongs. Optionally, when the memory 402 includes N levels of memory sets 4021, the memory computing instruction may include the identification codes of the memory sets 4021 to which N-1 computing units 4022 corresponding to the N levels of memory sets 4021 belong, and the identification code corresponding to any memory set 4021 is used to indicate whether the memory set 4021 to which the computing unit 4022 that performs the computing task indicated by the memory computing instruction belongs is the memory set 4021. For example, when the memory 402 includes a rank, a bank group, and a bank memory set 4021, the memory calculation instruction includes two identification codes of the memory set 4021 to which the calculation units 4022 belong, namely BGTag and bankTag. BGTag is used to indicate whether the memory set 4021 to which the calculation unit 4022 that performs the calculation task indicated by the memory calculation instruction belongs is a rank or a bank group, and bankTag is used to indicate whether the memory set 4021 to which the calculation unit 4022 that performs the calculation task indicated by the memory calculation instruction belongs is a bank group or a bank. The last batch indicator code (lastTag) is used to indicate whether the data calculated by the current memory calculation instruction belongs to the last batch when the data is distributed among multiple calculation batches for calculation. For example, when the last batch indicator code indicates that the data belongs to the last batch, it means that the current embedding operation has ended.The size of each field in a memory calculation instruction can be determined based on application requirements. For example, the bit operation code can be 3 bits, the memory operation code can be 3 bits, the memory address of the data required for calculation can be 34 bits, the data size of the data required for calculation can be 3 bits, the algorithm indicator code can be 32 bits, the batch indicator code can be 4 bits, the identification code of the memory set 4021 to which each calculation unit 4022 belongs can be 1 bit, and the final batch indicator code can be 1 bit.

[0108] Figure 18 is a schematic diagram comparing the acceleration ratios of the related art and the memory system 40 of the present application at different embedding vector lengths. Figure 19 is a schematic diagram comparing the acceleration ratios of the related art and the memory system 40 of the present application at different embedding operation batch sizes. Figure 20 is a schematic diagram comparing the energy consumption of the related art and the memory system 40 of the present application. Table 1 compares the additional area overhead of the memory system 40 of the related art and the present application. In the present application scheme shown in Figures 18, 19, and 20, the memory system 40 has three levels of memory sets 4021: rank, bank group, and bank. The data capacity ratio of the memory blocks in the rank, bank group, and bank-level memory sets 4021 is 4:3:1, and each rank is configured with a computing unit 4022, and some bank groups and some banks are configured with computing units 4022. In Figures 18 and 19, the dot-filled rectangles represent the CPU baseline scheme, in which no computing unit 4022 is provided in the memory 402. The rectangle filled with horizontal lines represents the TensorDIMM solution, which has a computing unit 4022 in each rank. The rectangle filled with vertical lines represents the RecNMP solution, which has a computing unit 4022 in each rank and is equipped with an additional cache. The rectangle filled with diagonal lines represents the NMP TRiM-G solution, which has a computing unit 4022 in each bank group. The unfilled rectangle represents the NMP TRiM-B solution, which has a computing unit 4022 in each bank group and each bank. The black-filled rectangle represents the solution of the present application (abbreviated as ReCross).

[0109] As shown in Figures 18 and 19, the present application solution has a performance improvement of 9.3 times and 7.9 times respectively compared to TensorDIMM and RecNMP with computing units 4022 configured at the Rank level, and a performance improvement of approximately 2.5 times and 1.8 times respectively compared to NMP TRiM-G with computing units 4022 configured at the bank group level and NMP TRiM-B with computing units 4022 configured at the bank level. The related art mainly relies on the number of computing units 4022 configured in the memory 402 to improve the parallelism and internal bandwidth of the memory 402, while the present application solution provides different levels of memory 402 parallelism for different data types, making more effective use of bandwidth. In particular, for frequently accessed data, sub-array-level parallelism is also utilized, greatly reducing data access latency.

[0110] In Figure 20, "1" represents the energy consumption of the computing unit, "2" represents the static power consumption of the computing unit, "3" represents the energy consumption of off-chip data input / output, and "4" represents the energy consumption of on-chip memory access. The black-filled portion of the energy consumption of the present application's solution represents the energy consumption of sub-array parallelism. As shown in Figure 20, compared with the CPU baseline, TensorDIMM, RecNMP, TRiM-G, and TRiM-B, the present application's solution reduces energy consumption by 58.5%, 57.2%, 51.9%, 28.5%, and 23.7%, respectively. This is primarily because the sub-array-level parallelism in the present application reduces data access latency, and most embedding operations are completed within the bank, reducing off-chip I / O energy consumption. In addition, the shorter execution time also reduces the static energy consumption of the memory 402 subsystem.

[0111] As shown in Table 1, since TensorDIMM and RecNMP are only configured with computing units 4022 in rank-level memory sets 4021, while TRiM-B, TRiM-G, and ReCross are configured with computing units 4022 in bank group and bank-level memory sets 4021, the additional area overhead of ReCross on DIMMs is 0.34 square millimeters per DIMM and 2.35 square millimeters per DRAM chip. Its additional area overhead is slightly different from that of TRiM-G because they have the same number of computing units 4022, but is much smaller than that of TRiM-B, which is equipped with more computing units 4022.

[0112] Table 1

[0113] As can be seen from the above, in the memory system provided in the embodiment of the present application, since each N-th level memory set is configured with a computing unit, some of the levels or some of the memory sets in all the levels of the first to N-1 level memory sets are also configured with the computing units, which increases the total number of computing units configured in the memory, helping to further improve the computing performance of the memory. In addition, since multiple memory subsets in the memory set have parallelism, by configuring computing units for some of the first-level memory sets, it is possible to take into account the parallelism of computing, the load balancing of computing, the utilization of computing resources, the chip area and the power consumption overhead. On the basis of effectively improving the computing performance of the memory system, for example, compared with the related art, it is possible to achieve a performance improvement of 1.8 to 2.5 times, configure fewer computing units, and save memory chip area and energy consumption overhead.

[0114] It should be noted that the components in the memory system provided in the embodiments of the present application can be increased or decreased accordingly. Any person skilled in the art who can easily conceive of variations within the technical scope disclosed in this application should be included in the scope of protection of this application, and therefore will not be described in detail.

[0115] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements the memory method provided in the embodiment of the present application.

[0116] An embodiment of the present application also provides a computer program product containing instructions. When the computer program product is run on a computer, the computer implements the memory method provided by the embodiment of the present application.

[0117] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0118] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the image information and executable code involved in this application are all obtained with full authorization.

[0119] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "plurality" refers to a plurality, unless otherwise expressly limited.

[0120] In this application, the term "and / or" simply describes an association between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0121] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0122] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0123] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the concepts and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A memory system, characterized in that: The memory system comprises: a memory controller and a memory; The memory controller is used to manage the memory; The memory has N levels of memory sets, each memory set includes multiple memory blocks, the i+1-th level memory set in the N-level memory sets includes multiple i-th level memory sets, and in the N-level memory sets, each N-th level memory set is configured with a computing unit, and some memory sets in some levels or all levels of the first to N-1 level memory sets are configured with the computing units, and the computing units are used to perform computing operations based on data stored in the memory sets, and N and i are positive integers.

2. The system according to claim 1, characterized in that Part of the first level memory set is configured with the computing unit.

3. The system according to claim 1 or 2, characterized in that: In the second to N-1 level memory sets, at least some memory sets included in any level memory set configured with the computing unit are not configured with the computing unit.

4. The system according to any one of claims 1 to 3, characterized in that: The memory controller is specifically used to provide computing instructions to the computing unit; The computing unit is specifically used to obtain data required for the calculation from the memory set to which the computing unit belongs based on the computing instruction, and perform the calculation based on the data.

5. The system according to claim 4, characterized in that The requirement for computing parallelism of the data stored in the i-th level memory set configured with the computing unit is higher than the requirement for computing parallelism of the data stored in the i-th level memory set not configured with the computing unit, and the computing parallelism is obtained based on one or more of the following: the access frequency of the data, the size of the data set to which the data belongs, the rated amount of data processed by each computing batch of the computing unit, and the rated capacity and rated bandwidth of the N-level memory set.

6. The system according to any one of claims 1 to 5, characterized in that: The computing unit of the i+1th level memory set is specifically used for: When the computing task indicated by the computing instruction does not belong to the computing unit of the (i+1)th level memory set, providing the computing instruction to the computing unit of the i-th level memory set included in the (i+1)th level memory set; When the computing task indicated by the computing instruction belongs to the computing unit of the (i+1)th level memory set, the computing task indicated by the computing instruction is executed.

7. The system according to claim 6, characterized in that The computing unit of the i+1th level memory set is further specifically used for: When the computing task indicated by the computing instruction does not belong to the computing unit of the i+1th level memory set, the computing unit of the i-th level memory set included in the i+1th level memory set receives the computing result provided by the computing unit, processes the computing result and outputs the processed computing result.

8. The system according to claim 7, characterized in that When the memory has multiple N-th level memory sets, the memory further includes: an output processing unit, which is used to process the calculation results of the calculation units of the multiple N-th level memory sets and output the processed calculation results.

9. The system according to claim 8, characterized in that The memory controller is specifically used to send the calculation instruction to the output processing unit; The output processing unit is specifically used to send the computing instruction to the Nth level memory set to which the computing unit used to process the computing instruction belongs.

10. The system according to any one of claims 1 to 9, characterized in that: The memory set includes a plurality of memory subsets, a plurality of local buffers, a plurality of control subunits and a global buffer, the plurality of memory subsets correspond to the plurality of local buffers, the plurality of memory subsets correspond to the plurality of control subunits, and the memory subsets include a plurality of memory blocks arranged in an array; The control subunit corresponding to any memory subset is used to select a target first memory unit from a plurality of first memory units included in any memory subset based on the address of the data required for calculation, determine the use authority of the target first memory unit to the global buffer, and select a target second memory unit from a plurality of second memory units included in the target first memory unit, wherein one of the first memory unit and the second memory unit is a row and the other is a column; The local buffer corresponding to any one of the memory subsets is used to cache the data stored in the memory block in the target first memory unit; The global buffer is used to cache target data stored in a memory block in a target second memory unit having the right to use the global buffer; The computing unit includes a computing subunit, which is used for the target data and performs the computing operation based on the target data.

11. A computing device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program instructions, and the processor executes the program instructions to implement the memory system according to any one of claims 1 to 10.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device implements the memory system according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device implements the memory system according to any one of claims 1 to 10.

Images