Memory system and operation method of in-memory computing

Abstract

The memory system includes a storage controller, memory, a first-level filter, and a second-level filter. The storage controller receives a first-stage result. The memory performs in-memory operations on the first-stage result to generate a second-stage result. The first-level filter filters the second-stage result to generate a third-stage result. The second-level filter filters the third-stage result to generate a fourth-stage result. The memory system outputs the fourth-stage result when the number of data results in the third-stage result exceeds a preset threshold, and outputs the third-stage result when the number of data results in the third-stage result is less than or equal to the preset threshold.

Description

Technical Field

[0001] This disclosure relates to a memory technology, and more particularly to a memory system and a method for operating computations within the memory. Prior Technology

[0002] Artificial intelligence (AI) has recently become a highly effective solution for many classification tasks. A key operation of AI involves a large number of input feature maps and weights to perform multiply-accumulate (MAC) operations. However, current Von Neumann architectures based on complementary metal-oxide-semiconductor (CMOS), i.e., central processing units (CPUs) or graphics processing units (GPUs) equipped with external memory, encounter input / output bottlenecks and inefficient MAC operation flows. Therefore, how to solve these problems is an important issue in this field. Summary of the Invention

[0003] This disclosure includes a memory system. The memory system includes a storage controller and memory. The storage controller receives a first-stage result. The memory performs in-memory operations on the first-stage result to generate a second-stage result. The memory device includes a first-level filter and a second-level filter. The first-level filter filters the second-stage result to generate a third-stage result. The second-level filter filters the third-stage result to generate a fourth-stage result. When the number of data results in the third-stage result is greater than a preset threshold, the memory system outputs the fourth-stage result; and when the number of data results in the third-stage result is less than or equal to the preset threshold, the memory system outputs the third-stage result.

[0004] This disclosure includes a method for computation within memory. The method includes: filtering a first-stage result using a first-level filter to generate a second-stage result; comparing the number of data results in the second-stage result with a preset threshold number; and when the number of data results in the second-stage result exceeds the preset threshold number, filtering the second-stage result using a second-level filter to generate a third-stage result. Simple Explanation of the Diagram

[0005] Figure 1 is a schematic diagram illustrating a method of computation within memory according to some embodiments of this case. Figure 2 is a schematic diagram of a memory system according to some embodiments of this case. Figure 3 is a schematic diagram illustrating the accuracy of a test dataset according to some embodiments of this case. Implementation

[0006] In this document, when an element is referred to as a "connection" or "coupled," it may mean an "electrical connection" or "electrical coupling." "Connection" or "coupled" can also be used to indicate the operation or interaction between two or more elements. Furthermore, although terms such as "first," "second," etc., are used to describe different elements, these terms are only used to distinguish elements or operations described using the same technical terminology. Unless the context clearly indicates otherwise, these terms do not specifically refer to or imply order or sequence, nor are they intended to limit the scope of this document.

[0007] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this case pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant technical context and this case, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0008] The terminology used herein is for the purpose of describing particular embodiments only and is not restrictive. As used herein, unless the content clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms, including "at least one." "Or" means "and / or." As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It should also be understood that, when used in this specification, the terms "comprising" and / or "including" specify the presence of the stated features, areas, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, areas, integrals, steps, operations, elements, components, and / or combinations thereof.

[0009] The following diagrams will disclose several embodiments of this invention. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details should not be used to limit this invention. That is, these practical details are not necessary in the embodiments disclosed herein. Furthermore, for the sake of simplicity, some conventional structures and components will be shown in the diagrams in a simple schematic manner.

[0010] Figure 1 is a schematic diagram illustrating an in-memory computation method 100 according to some embodiments of this invention. In some embodiments, the operation method 100 may reduce the amount of data results transferred between different devices to accelerate in-memory computation. In some embodiments, the operation method 100 may be implemented by a memory system including memory and a processor.

[0011] As shown in Figure 1, operation method 100 includes operations OP11 to OP17. In some embodiments, operations OP11 to OP17 are performed sequentially.

[0012] When operating OP11, the system begins operating method 100.

[0013] During operation OP12, the system performs a coarse search on stage result DS1 to generate stage result DS2.

[0014] During operation OP13, the system inputs the stage result DS2 into memory and performs memory operations on the stage result DS2 and the database DTB stored in memory to generate the stage result DS3.

[0015] During operation OP14, the system filters stage result DS3 using the first-level filter to generate stage result DS4. The number of data results in stage result DS4 is less than the number of data results in stage result DS3.

[0016] In some embodiments, the first-level filter can be implemented by setting a threshold or through machine learning. For example, the first-level filter sets the threshold to 3. When a data result in stage result DS3 has a data value less than 3, the first-level filter allows the data result to pass and assign it to stage result DS4. When a data result in stage result DS3 has a data value greater than or equal to 3, the first-level filter blocks the data result and does not assign it to stage result DS45. In other words, in the above example, stage result DS4 includes data results in stage result DS3 with data values ​​less than 3, and does not include data results in stage result DS3 with data values ​​greater than or equal to 3.

[0017] However, the embodiments disclosed herein are not limited to the above examples. In other embodiments, the first-level filter may also allocate data results with data values ​​greater than a threshold to stage result DS4. Specifically, when the data value of a data result in stage result DS3 is greater than the threshold, the first-level filter allows the data result to pass and allocate it to stage result DS4. When the data value of a data result in stage result DS3 is less than or equal to the threshold, the first-level filter blocks the data result and does not allocate it to stage result DS45. In other words, in the above example, stage result DS4 includes data results in stage result DS3 with data values ​​greater than the threshold, and does not include data results in stage result DS3 with data values ​​less than or equal to the threshold.

[0018] Next, the system compares the number of data results in stage result DS4 with a preset threshold number PTQ. When the number of data results in stage result DS4 is greater than the preset threshold number PTQ, after operation OP14, the system activates the second-level filter to perform operation OP15 on stage result DS4. When the number of data results in stage result DS4 is less than or equal to the preset threshold number PTQ, after operation OP14, the system performs operation OP16 on stage result DS4, without performing operation OP15. In some embodiments, the preset threshold number PTQ can be equal to 10,000.

[0019] During operation OP15, the system filters stage result DS4 using a second-level filter to generate stage result DS5. The number of data results in stage result DS5 is less than the number of data results in stage result DS4.

[0020] In some embodiments, the second-level filter can be implemented using a ranking algorithm. However, the embodiments disclosed herein are not limited thereto. In other embodiments, the second-level filter can be implemented using an algorithm other than a ranking algorithm, such as the Top K selection algorithm. Specifically, each data result in the stage result DS4 has a corresponding data value. In the implementation of the ranking algorithm, the second-level filter ranks these data results according to the data values ​​of the data results in the stage result DS4. For example, the smaller the data value of a data result, the higher the priority of that data result for the ranking algorithm. Conversely, the larger the data value of a data result, the lower the priority of that data result for the ranking algorithm.

[0021] However, the embodiments disclosed herein are not limited to the examples described above. In other embodiments, depending on different distance algorithms, a data result may have a higher priority when its data value is larger, and vice versa.

[0022] Next, the second-level filter selects N1 data results with higher priority from stage result DS4 as stage result DS5, where N1 is a positive integer less than or equal to a preset critical number PTQ. In other words, the priority of each of these N1 data results is higher than the priority of the other data results in stage result DS4.

[0023] For example, a positive integer N1 can be equal to five thousand. Correspondingly, the second-level filter selects five thousand data results with smaller data values ​​from stage result DS4 as stage result DS5. The data value of each of these five thousand data results in stage result DS5 is less than the data values ​​of the other data results in stage result DS4. In the above example, the data values ​​of each of the five thousand data results in stage result DS5 are less than a threshold value, and the data values ​​of each of the other data results in stage result DS4 are greater than or equal to that threshold value.

[0024] In some variations, the second-level filter can also be implemented using algorithms other than sorting algorithms. For example, the second-level filter can be implemented using the Top K selection algorithm, where the positive integer K can be equal to the positive integer N1.

[0025] Following operation OP15, in operation OP16, the system performs a fine search on the stage result DS5 to produce the final filtered result DS6. In some embodiments, the final filtered result DS6 corresponds to the search result of operation method 100 computed in memory.

[0026] Please refer to operations OP12 and OP15. In some embodiments, a coarse search searches for data over a larger range or in a more general manner, while a fine search is a query targeting a specific data point or a specific range. In some embodiments, a coarse search is a preprocessing step for a fine search.

[0027] If the number of data results in stage result DS4 is less than or equal to the preset critical number PTQ, after operation OP14, the system performs a fine search on stage result DS4 in operation OP16 to generate the final filtered result DS6.

[0028] Operation OP17, operation method 100 ends.

[0029] Figure 2 is a schematic diagram illustrating system 200 according to some embodiments of this invention. As shown in Figure 2, system 200 includes a processing device 210 and a storage system 220. In some embodiments, system 200 may be implemented using a von Neumann architecture. Processing device 210 may include a processor. Storage system 220 may also include a processor.

[0030] In some embodiments, the processing device 210 and the storage system 220 are used to transfer data to each other for data processing. For example, the storage system 220 can transfer stage results DS4 or DS5 to the processing device 210, so that the processing device 210 can perform a fine search on the stage results DS4 or DS5 to produce a final filtered result DS6.

[0031] In some embodiments, the processing device 210 may be implemented by a central processing unit (CPU), and the storage system 220 may be implemented by a solid state disk (SSD). However, the embodiments disclosed herein are not limited thereto. In some embodiments, the operating method 100 may also be applied to an in-memory computing (IMC) system based on non-volatile memory.

[0032] As shown in Figure 2, the storage system 220 includes a storage controller 221 and a memory 222. The storage controller 221 may include a processor. The memory 222 may include one or more memory chips, wherein the memory chip may include one or more memory dies (one or more memory dies may be packaged into a single memory chip). In some embodiments, the memory 222 includes a plurality of memory chips MC1 to MC4.

[0033] In the embodiment shown in Figure 2, memory chips MC1-MC4 each include first-stage filters FLF1-FLF4, and the storage controller 221 includes a second-stage filter SLF1. However, the embodiments disclosed herein are not limited thereto. In some variations, memory chips MC1-MC4 may include both first-stage and second-stage filters, enabling memory chips MC1-MC4 to perform each of operations OP14 and OP15. In other variations, the storage controller 221 may include both first-stage and second-stage filters, enabling the storage controller 221 to perform each of operations OP14 and OP15.

[0034] In some embodiments, memory 222 may be implemented using multiple logic chips. For example, in the embodiment shown in Figure 2, memory 222 is implemented using multiple 3D NAND flash memory chips having 16 channels and 4 ways.

[0035] In some embodiments, memory chips MC1-MC4 can be implemented using logic chips. For example, in the embodiment shown in Figure 2, memory chips MC1-MC4 are implemented using NAND flash memory logic chips. In summary, memory 222 can be implemented using multiple memory chips (logic chips). Memory chips can be implemented using NAND flash memory chips. NAND flash memory chips can be implemented using two-dimensional (2D) or three-dimensional NAND flash memory chips. NAND flash memory chips can contain NAND flash memory dies.

[0036] Please refer to Figures 1 and 2. In some embodiments, the operation method 100 can be executed by the system 200.

[0037] Specifically, during operation OP12, processing device 210 performs a coarse search on stage result DS1 to generate stage result DS2. In various embodiments, processing device 210 may be implemented by a CPU, GPU, or other processing device. In other words, the coarse search may be performed by a CPU, GPU, or other processing device. Next, storage controller 221 receives stage result DS2 from processing device 210 and transfers stage result DS2 to memory 222.

[0038] During operation OP13, memory 222 receives stage result DS2 from storage controller 221, causing memory chips MC1~MC4 to perform in-memory operations on stage result DS2 and database DTB stored in memory chips MC1~MC4 to generate stage result DS3.

[0039] During operation OP14, the first-stage filters FLF1~FLF4 filter the stage result DS3 to generate stage result DS4. Next, the storage controller 221 receives the stage result DS4 and compares the number of data results in the stage result DS4 with a preset threshold quantity PTQ. In some embodiments, the storage controller 221 stores the preset threshold quantity PTQ.

[0040] In some embodiments, the storage controller 221 is used to count the data results in stage result DS4 to generate the number of data results in stage result DS4. However, this disclosure is not limited thereto. In some variations, the memory 222 is used to count the data results in stage result DS4 to generate the number of data results in stage result DS4 and transmits the number to the storage controller 221.

[0041] When the number of data results in stage result DS4 is greater than the preset threshold number PTQ, storage controller 221 activates the second-level filter SLF1 to perform operation OP15 on stage result DS4. When the number of data results in stage result DS4 is less than or equal to the preset threshold number PTQ, storage controller 221 outputs stage result DS4 to processing device 210, and processing device 210 performs operation OP16 on stage result DS4.

[0042] During operation OP15, the storage controller 221 filters the stage result DS4 using the second-stage filter SLF1 to generate the stage result DS5. Then, the storage controller 221 outputs the stage result DS5 to the processing device 210, and the processing device 210 performs operation OP16 on the stage result DS5.

[0043] During operation OP16, the processing unit 210 performs a fine search on the stage results DS4 or DS5 to produce the final filtered result DS6.

[0044] In some implementations, when performing operations within memory, the storage controller needs to transfer large amounts of data results to the processing device. This leads to input / output bottlenecks and computational bottlenecks. To address these bottlenecks, a primary filter is added to the system to reduce computational and data transfer loads. However, in some situations, the filtering effect remains unsatisfactory.

[0045] Compared to the above approach, in this embodiment, when the number of data results in stage result DS4 exceeds a preset threshold number PTQ, the storage controller 221 activates the second-level filter SLF1 to generate stage result DS5. In this way, both the data transfer volume of the storage controller 221 and the computational load of the processing device 210 can be reduced.

[0046] In some embodiments, the in-memory operational filter can also be implemented in the von Neumann solution. In the embodiments shown in Figures 1 and 2, the system includes a first-level filter and a second-level filter. However, the embodiments disclosed herein are not limited thereto. In various embodiments, the system may include more levels of filters.

[0047] Figure 3 is a schematic diagram illustrating the accuracy of the test dataset TDS1 according to some embodiments of this case. The vertical axis of Figure 3 corresponds to the accuracy. Figure 3 includes accuracies AC1 and AC2.

[0048] Please refer to Figures 1 and 3. Accuracy values ​​AC1 and AC2 can be used to determine the positive integer N1 of operation OP15. In some embodiments, the system sequentially performs operations OP12 to OP14 on test dataset TDS1 to generate test dataset TDS31, and performs operation OP15 on test dataset TDS31 to generate test dataset TDS32.

[0049] In other words, the system filters the test dataset TDS1 using the first-level filter to generate the test dataset TDS31, and then filters the test dataset TDS31 using the second-level filter to generate the test dataset TDS32.

[0050] Next, the system performs operation OP16 on test dataset TDS31 to generate test dataset TDS33, and performs operation OP16 on test dataset TDS32 to generate test dataset TDS34. Specifically, the system performs a fine-grained search operation on test dataset TDS31 to generate test dataset TDS33, and performs a fine-grained search operation on test dataset TDS32 to generate test dataset TDS34.

[0051] In some embodiments, the positive integer N1 can be determined offline. After the positive integer N1 is determined, system 200 comes online and begins operation. While system 200 is online and beginning operation, the positive integer N1 can remain unchanged.

[0052] Specifically, system 200 can filter the test dataset TDS1 without using the second-level filter to produce filtered results without the second-level filter. Alternatively, system 200 can use the second-level filter to filter the test dataset TDS1 to produce filtered results with the second-level filter. Next, system 200 can verify whether there is a significant difference in accuracy between the filtered results without the second-level filter and the filtered results with the second-level filter. Specifically, system 200 can verify whether the accuracy of the filtered results with the second-level filter meets a preset accuracy. If the accuracy of the filtered results with the second-level filter does not meet the preset accuracy, system 200 can increment a positive integer N1. If the accuracy of the filtered results with the second-level filter meets the preset accuracy, system 200 can go online and begin operation.

[0053] For example, when the positive integer N1 equals two thousand, system 200 uses a second-level filter to generate an accuracy corresponding to two thousand. In response to the accuracy corresponding to two thousand being less than the preset accuracy, system 200 increases the positive integer N1 to three thousand. Next, system 200 uses the second-level filter to generate an accuracy corresponding to three thousand. In response to the accuracy corresponding to three thousand being greater than the preset accuracy, system 200 uses three thousand as the positive integer N1 and goes online and begins operation.

[0054] In summary, the positive integer N1 can be determined based on the validation index of the corresponding test dataset TDS1. In some embodiments, the validation index can be precision, recall, or other validation indices.

[0055] As shown in Figure 3, accuracy AC1 is the accuracy of the test dataset TDS33, and accuracy AC2 is the accuracy of the test dataset TDS34. In other words, accuracy AC1 is the accuracy without a second-level filter and corresponds to test dataset TDS31. Accuracy AC2 is the accuracy with a second-level filter and corresponds to test dataset TDS32. In some variations, the accuracy described herein can be replaced with recall rate.

[0056] Generally, accuracy is proportional to the number of data results in the corresponding stage. Specifically, accuracy AC1 is proportional to the number of data results in test dataset TDS33, and accuracy AC2 is proportional to the number of data results in test dataset TDS34. If the number of data results in test dataset TDS34 is less than the number of data results in test dataset TDS33, accuracy AC2 is less than accuracy AC1.

[0057] However, the embodiments disclosed herein are not limited to this. The positive integer N1 can be determined online. For example, in some embodiments, the system can compare accuracy AC2 with a preset accuracy. When accuracy AC2 is less than the preset accuracy, the system increments the positive integer N1 to increase the number of data results in the test dataset TDS34, thereby increasing accuracy AC2 accordingly. When accuracy AC2 is greater than or equal to the preset accuracy, the system determines the positive integer N1 and performs operation OP15 on the stage result DS4 shown in Figure 1 with the determined positive integer N1. In some embodiments, the preset accuracy may be equal to accuracy AC1. Correspondingly, in this embodiment, accuracy AC2 will not be greater than the preset accuracy. However, the disclosure is not limited to this. In other embodiments, the preset accuracy may be less than accuracy AC1. Correspondingly, in this embodiment, accuracy AC2 may be between the preset accuracy and accuracy AC1.

[0058] In other words, the system determines a positive integer N1 based on the accuracy AC2 corresponding to the test dataset TDS1, and the second-level filter can filter the stage result DS4 based on the determined positive integer N1 to ensure that the subsequent final filtered result DS6 has sufficient accuracy. In some embodiments, the test dataset TDS1 may contain 10,000 data points.

[0059] In some embodiments, the preset accuracy rate can be equal to the accuracy AC1 multiplied by the preset accuracy rate percentage. For example, the preset accuracy rate is equal to the accuracy AC1 multiplied by 0.8, where the preset accuracy rate percentage is equal to 0.8. However, this disclosure is not limited thereto. In various embodiments, the preset accuracy rate percentage can also be equal to 0.9 or other values.

[0060] Referring to Figures 2 and 3, in some embodiments, system 200 may further include a processor (not shown) for performing the operations related to Figure 3. For example, the processor may compare accuracy AC2 with a preset accuracy to determine a positive integer N1. Furthermore, the first-level filter performing the operations related to Figure 3 may be implemented using first-level filters FLF1 to FLF4, and the second-level filter performing the operations related to Figure 3 may be implemented using second-level filter SLF1.

[0061] Although the present disclosure has been presented above with reference to embodiments, it is not intended to limit the scope of the present disclosure. Anyone skilled in the art may make some modifications and refinements without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the appended claims.

[0062] 100: Operating Instructions OP11~OP17: Operation DS1~DS5: Phase Results DS6: Final Filtering Results 200: System 210: Processing device 220: Storage System 221: Storage Controller 222: Memory MC1~MC4: Memory chips FLF1~FLF4: First-stage filters SLF1: Second-stage filter AC1, AC2: Accuracy TDS1, TDS31, TDS32: Test Data Sets

[0063] Domestic storage information (please note in order of storage institution, date, and number) none Overseas storage information (please note in the order of storage country, institution, date, and number) none

Claims

1. A memory system comprising: a storage controller for receiving a first-stage result; a memory for performing in-memory operations on the first-stage result to generate a second-stage result; a first-level filter for filtering the second-stage result to generate a third-stage result; and a second-level filter for filtering the third-stage result to generate a fourth-stage result, wherein the memory system outputs the fourth-stage result when a number of multiple data results in the third-stage result is greater than a preset threshold number, and the memory system outputs the third-stage result when the number of data results in the third-stage result is less than or equal to the preset threshold number, wherein the second-level filter is further configured to sort the data results in the third-stage result according to multiple data values, and select N1 data results from the third-stage result as the fourth-stage result, wherein N1 is a positive integer less than or equal to the preset threshold number.

2. The memory system as described in claim 1, wherein the storage controller is configured to output the third-stage result or the fourth-stage result to a processing device.

3. The memory system as described in claim 2, wherein N1 is determined based on a verification index corresponding to a first test dataset.

4. The memory system as described in claim 3, wherein the verification index is accuracy.

5. The memory system as claimed in claim 4, wherein the first-level filter is further configured to filter the first test dataset to generate a second test dataset, the second-level filter is further configured to filter the second test dataset to generate a third test dataset, and the memory system is configured to adjust N1 based on a first accuracy corresponding to the second test dataset and a second accuracy corresponding to the third test dataset.

6. The memory system as described in claim 5, wherein the memory system is further configured to compare the second accuracy with a preset accuracy, wherein when the second accuracy is less than the preset accuracy, the memory system increases N1, and when the second accuracy is greater than or equal to the preset accuracy, the memory system determines N1.

7. A method for computation within memory, comprising: filtering a first-stage result using a first-level filter to generate a second-stage result; comparing the number of data results in the second-stage result with a preset threshold number; and when the number of data results in the second-stage result is greater than the preset threshold number, filtering the second-stage result using a second-level filter to generate a third-stage result, wherein filtering the second-stage result to generate the third-stage result comprises: sorting the data results in the second-stage result according to multiple data values ​​of the data results in the second-stage result using the second-level filter; and selecting N1 data results from the second-stage result as the third-stage result using the second-level filter, wherein N1 is a positive integer less than or equal to the preset threshold number.

8. The method of operation as described in claim 7, wherein the first-level filter is used to set a threshold value to filter the first-stage result.

Images