Memory system and operating method of in-memory

Abstract

The invention provides a memory system and an operation method for computing in a memory. The memory system comprises a storage controller, a memory, a first-stage filter and a second-stage filter. The storage controller is used for receiving a first stage result. The memory is used for performing in-memory operation on the first-stage result to generate a second-stage result. The first-stage filter is used for filtering the second-stage result to generate a third-stage result. The second-stage filter is used for filtering the third-stage result to generate a fourth-stage result. The memory system outputs the fourth stage result when a number of the plurality of data results in the third stage result is greater than a preset threshold number, and outputs the third stage result when the number of the data results in the third stage result is less than or equal to the preset threshold number.

Description

Technical Field

[0001] This invention relates to a memory technology, and more particularly to a memory system and a method for operating computation within the memory. Background Technology

[0002] Artificial intelligence (AI) has recently become a highly effective solution for many classification tasks. A key aspect of AI operation is that it involves a large number of input feature maps and weights to perform multiply-accumulate (MAC) operations. However, current Von Neumann architectures based on complementary metal-oxide-semiconductor (CMOS), i.e., central processing units (CPUs) or graphics processing units (GPUs) equipped with external memory, encounter input / output bottlenecks and inefficient MAC operation flows. Therefore, addressing these issues is a crucial task in this field. Summary of the Invention

[0003] This invention includes a memory system. The memory system includes a storage controller and a memory. The storage controller receives a first-stage result. The memory performs in-memory operations on the first-stage result to generate a second-stage result. The memory device includes a first-level filter and a second-level filter. The first-level filter filters the second-stage result to generate a third-stage result. The second-level filter filters the third-stage result to generate a fourth-stage result. When the number of data results in the third-stage result is greater than a preset threshold number, the memory system outputs the fourth-stage result; and when the number of data results in the third-stage result is less than or equal to the preset threshold number, the memory system outputs the third-stage result.

[0004] This invention includes an in-memory computation method. The method includes: filtering a first-stage result through a first-level filter to generate a second-stage result; comparing the number of data results in the second-stage result with a preset threshold number; and when the number of data results in the second-stage result exceeds the preset threshold number, filtering the second-stage result through a second-level filter to generate a third-stage result. Attached Figure Description

[0005] Figure 1 This is a schematic diagram illustrating an in-memory computation method according to some embodiments of the present invention.

[0006] Figure 2 This is a schematic diagram illustrating a memory system according to some embodiments of the present invention.

[0007] Figure 3 This is a schematic diagram illustrating the accuracy of a test dataset according to some embodiments of the present invention.

[0008] Explanation of reference numerals in the attached figures:

[0009] 100: Operating Instructions

[0010] OP11~OP17: Operation

[0011] DS1~DS5: Phase Results

[0012] DS6: Final Filtering Results

[0013] 200: System

[0014] 210: Processing device

[0015] 220: Storage System

[0016] 221: Storage Controller

[0017] 222: Memory

[0018] MC1~MC4: Memory chips

[0019] FLF1~FLF4: First-stage filters

[0020] SLF1: Second-stage filter

[0021] AC1, AC2: Accuracy

[0022] TDS1, TDS31, TDS32: Test datasets Detailed Implementation

[0023] In this document, when an element is referred to as a "connection" or "coupled," it may mean an "electrical connection" or "electrical coupling." "Connection" or "coupled" can also be used to indicate the operation or interaction between two or more elements. Furthermore, although terms such as "first," "second," etc., are used herein to describe different elements, these terms are merely used to distinguish elements or operations described using the same technical terms. Unless the context clearly indicates otherwise, these terms do not specifically refer to or imply any order or sequence, nor are they intended to limit the invention.

[0024] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology and this invention, and will not be interpreted as having idealized or overly formal meanings unless expressly defined herein.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not restrictive. As used herein, unless the content clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms, including "at least one." "Or" means "and / or." As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It should also be understood that, when used in this specification, the terms "comprising" and / or "including" specify the presence of the stated features, areas, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, areas, integrals, steps, operations, elements, components, and / or combinations thereof.

[0026] The following describes several embodiments of the present invention with reference to the accompanying drawings. For clarity, many practical details will be described in conjunction with the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential. Furthermore, for the sake of simplicity, some known and conventional structures and elements will be shown in the drawings in a simple schematic manner.

[0027] Figure 1 This is a schematic diagram illustrating an in-memory computation operation method 100 according to some embodiments of the present invention. In some embodiments, the operation method 100 may reduce the amount of data results transferred between different devices to accelerate in-memory computation. In some embodiments, the operation method 100 may be implemented using a memory system including a memory and a processor.

[0028] Figure 1 As shown, operation method 100 includes operations OP11 to OP17. In some embodiments, operations OP11 to OP17 are performed sequentially.

[0029] When operating OP11, the system begins operating method 100.

[0030] During operation OP12, the system performs a coarse search on stage result DS1 to generate stage result DS2.

[0031] During operation OP13, the system inputs the stage result DS2 into the memory and performs in-memory operations on the stage result DS2 and the database DTB stored in the memory to generate the stage result DS3.

[0032] In operation OP14, the system filters stage result DS3 through the first-level filter to generate stage result DS4. The number of data results in stage result DS4 is less than the number of data results in stage result DS3.

[0033] In some embodiments, the first-level filter can be implemented by setting a threshold or through machine learning. For example, the first-level filter sets the threshold to 3. When a data result in stage result DS3 has a value less than 3, the first-level filter allows the data result to pass and assign it to stage result DS4. When a data result in stage result DS3 has a value greater than or equal to 3, the first-level filter blocks the data result and does not assign it to stage result DS45. In other words, in the above example, stage result DS4 includes data results in stage result DS3 with values ​​less than 3, and does not include data results in stage result DS3 with values ​​greater than or equal to 3.

[0034] However, embodiments of the present invention are not limited to the above examples. In other embodiments, the first-level filter may also allocate data results with data values ​​greater than a threshold to stage result DS4. Specifically, when the data value of a data result in stage result DS3 is greater than the threshold, the first-level filter allows the data result to pass and allocate it to stage result DS4. When the data value of a data result in stage result DS3 is less than or equal to the threshold, the first-level filter blocks the data result and does not allocate it to stage result DS45. In other words, in the above example, stage result DS4 includes data results in stage result DS3 with data values ​​greater than the threshold, and does not include data results in stage result DS3 with data values ​​less than or equal to the threshold.

[0035] Next, the system compares the number of data results in stage result DS4 with a preset threshold number PTQ. When the number of data results in stage result DS4 is greater than the preset threshold number PTQ, after operation OP14, the system activates the second-level filter to perform operation OP15 on stage result DS4. When the number of data results in stage result DS4 is less than or equal to the preset threshold number PTQ, after operation OP14, the system performs operation OP16 on stage result DS4, without performing operation OP15. In some embodiments, the preset threshold number PTQ can be equal to ten thousand.

[0036] In operation OP15, the system filters stage result DS4 through a second-level filter to generate stage result DS5. The number of data results in stage result DS5 is less than the number of data results in stage result DS4.

[0037] In some embodiments, the second-level filter can be implemented using a ranking algorithm. However, embodiments of the invention are not limited thereto. In other embodiments, the second-level filter can be implemented using an algorithm other than a ranking algorithm, such as the Top K selection algorithm. Specifically, each data result in the stage result DS4 has a corresponding data value. In the implementation of the ranking algorithm, the second-level filter ranks these data results according to the data values ​​of the data results in the stage result DS4. For example, the smaller the data value of a data result, the higher the priority of that data result for the ranking algorithm. Conversely, the larger the data value of a data result, the lower the priority of that data result for the ranking algorithm.

[0038] However, embodiments of the present invention are not limited to the above examples. In other embodiments, depending on different distance algorithms, a data result with a larger value may have a higher priority. Conversely, a data result with a smaller value may have a lower priority.

[0039] Next, the second-level filter selects N1 data results with higher priority from stage result DS4 as stage result DS5, where N1 is a positive integer less than or equal to a preset threshold number PTQ. In other words, the priority of each of these N1 data results is higher than the priority of other data results in stage result DS4.

[0040] For example, a positive integer N1 can be equal to five thousand. Correspondingly, the second-level filter selects five thousand data results with smaller data values ​​from stage result DS4 as stage result DS5. The data value of each of these five thousand data results in stage result DS5 is less than the data values ​​of the other data results in stage result DS4. In the example above, each of the five thousand data results in stage result DS5 is less than a threshold, and each of the other data results in stage result DS4 is greater than or equal to that threshold.

[0041] In some variations, the second-level filter can also be implemented using algorithms other than sorting algorithms. For example, the second-level filter can be implemented using the Top K selection algorithm, where the positive integer K can be equal to the positive integer N1.

[0042] Following operation OP15, in operation OP16, the system performs a fine-grained search on the stage result DS5 to produce the final filtered result DS6. In some embodiments, the final filtered result DS6 corresponds to the search result of operation method 100 computed in memory.

[0043] Please refer to operations OP12 and OP15. In some embodiments, a coarse search searches data over a broad range or in a coarse manner, while a fine search is a query targeting a specific data point or a specific range. In some embodiments, a coarse search is a preprocessing step for a fine search.

[0044] If the number of data results in stage result DS4 is less than or equal to the preset threshold number PTQ, after operation OP14, the system performs a fine search on stage result DS4 in operation OP16 to generate the final filtered result DS6.

[0045] Operation OP17, operation method 100 ends.

[0046] Figure 2 A schematic diagram illustrating a system 200 according to some embodiments of the present invention. Figure 2 As shown, system 200 includes a processing device 210 and a storage system 220. In some embodiments, system 200 can be implemented using a von Neumann architecture. The processing device 210 may include a processor. The storage system 220 may also include a processor.

[0047] In some embodiments, the processing device 210 and the storage system 220 are used to transfer data to each other for data processing. For example, the storage system 220 can transfer stage results DS4 or DS5 to the processing device 210, so that the processing device 210 can perform a fine search on the stage results DS4 or DS5 to produce a final filtered result DS6.

[0048] In some embodiments, the processing device 210 may be implemented using a central processing unit (CPU), and the storage system 220 may be implemented using a solid state disk (SSD). However, the embodiments of the present invention are not limited thereto. In some embodiments, the operating method 100 may also be applied to an in-memory computing (IMC) system based on non-volatile memory.

[0049] like Figure 2 As shown, the storage system 220 includes a storage controller 221 and a memory 222. The storage controller 221 may include a processor. The memory 222 may include one or more memory chips, wherein the memory chip may include one or more memory cells (one or more memory cells may be packaged into a single memory chip). In some embodiments, the memory 222 includes a plurality of memory chips MC1 to MC4.

[0050] exist Figure 2In the illustrated embodiment, memory chips MC1-MC4 each include first-level filters FLF1-FLF4, and the storage controller 221 includes a second-level filter SLF1. However, the embodiments of the present invention are not limited thereto. In some variations, memory chips MC1-MC4 may include both first-level and second-level filters, such that memory chips MC1-MC4 can perform each of operations OP14 and OP15. In other variations, storage controller 221 may include both first-level and second-level filters, such that storage controller 221 can perform each of operations OP14 and OP15.

[0051] In some embodiments, the memory 222 may be implemented using multiple logic units. For example, in Figure 2 In the embodiment shown, memory 222 is implemented using multiple 3D NAND flash memory cells with 16 channels and 4 ways.

[0052] In some embodiments, memory chips MC1-MC4 can be implemented using logic chips. For example, in Figure 2 In the illustrated embodiment, memory chips MC1-MC4 are implemented using NAND flash memory logic chips. In summary, memory 222 can be implemented using multiple memory chips (logic chips). Memory chips can be implemented using NAND flash memory chips. NAND flash memory chips can be implemented using two-dimensional (2D) or three-dimensional NAND flash memory chips. NAND flash memory chips can contain NAND flash memory cells.

[0053] Please refer to Figure 1 and Figure 2 In some embodiments, the operation method 100 can be executed by the system 200.

[0054] Specifically, during operation OP12, processing device 210 performs a coarse search on stage result DS1 to generate stage result DS2. In various embodiments, processing device 210 can be implemented by a CPU, GPU, or other processing device. In other words, the coarse search can be performed by a CPU, GPU, or other processing device. Next, storage controller 221 receives stage result DS2 from processing device 210 and transfers stage result DS2 to memory 222.

[0055] During operation OP13, memory 222 receives stage result DS2 from storage controller 221, causing memory chips MC1~MC4 to perform in-memory operations on stage result DS2 and database DTB stored in memory chips MC1~MC4 to generate stage result DS3.

[0056] During operation OP14, the first-stage filters FLF1~FLF4 filter the stage result DS3 to generate the stage result DS4. Next, the storage controller 221 receives the stage result DS4 and compares the number of data results in the stage result DS4 with a preset threshold number PTQ. In some embodiments, the storage controller 221 stores the preset threshold number PTQ.

[0057] In some embodiments, the storage controller 221 is used to count the data results in stage result DS4 to generate the number of data results in stage result DS4. However, the invention is not limited thereto. In some variations, the storage 222 is used to count the data results in stage result DS4 to generate the number of data results in stage result DS4 and transmits the number to the storage controller 221.

[0058] When the number of data results in stage result DS4 exceeds a preset threshold number PTQ, storage controller 221 activates the second-level filter SLF1 to perform operation OP15 on stage result DS4. When the number of data results in stage result DS4 is less than or equal to the preset threshold number PTQ, storage controller 221 outputs stage result DS4 to processing device 210, and processing device 210 performs operation OP16 on stage result DS4.

[0059] In operation OP15, the storage controller 221 filters the stage result DS4 through the second-stage filter SLF1 to generate the stage result DS5. Then, the storage controller 221 outputs the stage result DS5 to the processing device 210, and the processing device 210 performs operation OP16 on the stage result DS5.

[0060] During operation OP16, the processing unit 210 performs a fine search on the stage results DS4 or DS5 to produce the final filtered result DS6.

[0061] In some implementations, during in-memory operations, the storage controller needs to transfer large amounts of data results to the processing device. This can lead to input / output bottlenecks and computational bottlenecks. To address these bottlenecks, a primary filter is added to the system to reduce computational and data transfer loads. However, in some situations, the filtering effect remains unsatisfactory.

[0062] Compared to the above approach, in this embodiment of the invention, when the number of data results in stage result DS4 exceeds a preset threshold number PTQ, the storage controller 221 activates the second-level filter SLF1 to generate stage result DS5. In this way, both the transmission load of the storage controller 221 and the computational load of the processing device 210 can be reduced.

[0063] In some embodiments, in-memory operational filters can also be implemented in the von Neumann solution. Figure 1 and Figure 2 In the illustrated embodiment, the system includes a first-stage filter and a second-stage filter. However, the embodiments of the present invention are not limited thereto. In various embodiments, the system may include more stages of filters.

[0064] Figure 3 A schematic diagram illustrating the accuracy of the test dataset TDS1 according to some embodiments of the present invention. Figure 3 The vertical axis corresponds to the accuracy rate. Figure 3 Includes accuracy rates AC1 and AC2.

[0065] Please refer to Figure 1 and Figure 3 Accuracy values ​​AC1 and AC2 can be used to determine the positive integer N1 of operation OP15. In some embodiments, the system sequentially performs operations OP12 to OP14 on the test dataset TDS1 to generate the test dataset TDS31, and performs operation OP15 on the test dataset TDS31 to generate the test dataset TDS32.

[0066] In other words, the system filters the test dataset TDS1 through the first-level filter to generate the test dataset TDS31, and filters the test dataset TDS31 through the second-level filter to generate the test dataset TDS32.

[0067] Next, the system performs operation OP16 on the test dataset TDS31 to generate test dataset TDS33, and performs operation OP16 on the test dataset TDS32 to generate test dataset TDS34. Specifically, the system performs a fine-grained search operation on the test dataset TDS31 to generate test dataset TDS33, and performs a fine-grained search operation on the test dataset TDS32 to generate test dataset TDS34.

[0068] In some embodiments, the positive integer N1 can be determined offline. After the positive integer N1 is determined, system 200 comes online and begins operation. While system 200 is online and beginning operation, the positive integer N1 can remain unchanged.

[0069] Specifically, system 200 can filter the test dataset TDS1 without using the second-level filter to produce filtered results without the second-level filter. Alternatively, system 200 can use the second-level filter to filter the test dataset TDS1 to produce filtered results with the second-level filter. Next, system 200 can verify whether there is a significant difference in accuracy between the filtered results without the second-level filter and the filtered results with the second-level filter. Specifically, system 200 can verify whether the accuracy of the filtered results with the second-level filter meets a preset accuracy. If the accuracy of the filtered results with the second-level filter does not meet the preset accuracy, system 200 can increment a positive integer N1. If the accuracy of the filtered results with the second-level filter meets the preset accuracy, system 200 can go online and begin operation.

[0070] For example, when the positive integer N1 equals two thousand, system 200 uses a second-level filter to generate an accuracy corresponding to two thousand. In response to the accuracy corresponding to two thousand being less than a preset accuracy, system 200 increases the positive integer N1 to three thousand. Next, system 200 uses the second-level filter to generate an accuracy corresponding to three thousand. In response to the accuracy corresponding to three thousand being greater than the preset accuracy, system 200 uses three thousand as the positive integer N1 and goes online and begins operation.

[0071] In summary, the positive integer N1 can be determined based on the validation index of the corresponding test dataset TDS1. In some embodiments, the validation index can be precision, recall, or other validation indices.

[0072] like Figure 3 As shown, accuracy AC1 is the accuracy of the test dataset TDS33, and accuracy AC2 is the accuracy of the test dataset TDS34. In other words, accuracy AC1 is the accuracy without a second-level filter and corresponds to the test dataset TDS31. Accuracy AC2 is the accuracy with a second-level filter and corresponds to the test dataset TDS32. In some variations, the accuracy described herein can be replaced with recall rate.

[0073] Generally, accuracy is proportional to the number of data results in the corresponding stage. Specifically, accuracy AC1 is proportional to the number of data results in the test dataset TDS33, and accuracy AC2 is proportional to the number of data results in the test dataset TDS34. Since the number of data results in the test dataset TDS34 is less than the number of data results in the test dataset TDS33, accuracy AC2 is less than accuracy AC1.

[0074] However, embodiments of the present invention are not limited thereto. The positive integer N1 can be determined online. For example, in some embodiments, the system can compare the accuracy AC2 with a preset accuracy. When the accuracy AC2 is less than the preset accuracy, the system increments the positive integer N1 to increase the number of data results in the test dataset TDS34, thereby increasing the accuracy AC2 accordingly. When the accuracy AC2 is greater than or equal to the preset accuracy, the system determines the positive integer N1 and uses the determined positive integer N1 to... Figure 1 The stage result DS4 shown executes operation OP15. In some embodiments, the preset accuracy can be equal to the accuracy AC1. Correspondingly, in this embodiment, the accuracy AC2 will not be greater than the preset accuracy. However, the invention is not limited thereto. In other embodiments, the preset accuracy can be less than the accuracy AC1. Correspondingly, in this embodiment, the accuracy AC2 may be between the preset accuracy and the accuracy AC1.

[0075] In other words, the system determines a positive integer N1 based on the accuracy AC2 corresponding to the test dataset TDS1, and the second-level filter can filter the stage result DS4 based on the determined positive integer N1 to ensure that the subsequent final filtered result DS6 has sufficient accuracy. In some embodiments, the test dataset TDS1 may contain 10,000 data points.

[0076] In some embodiments, the preset accuracy rate can be equal to the accuracy AC1 multiplied by the preset accuracy rate percentage. For example, the preset accuracy rate is equal to the accuracy AC1 multiplied by 0.8, where the preset accuracy rate percentage is equal to 0.8. However, the invention is not limited thereto. In various embodiments, the preset accuracy rate percentage can also be equal to 0.9 or other values.

[0077] Please refer to Figure 2 and Figure 3 In some embodiments, system 200 may further include components for performing Figure 3 The processor (not shown in the diagram) performs the relevant operations. For example, the processor can compare the accuracy AC2 with a preset accuracy to determine the positive integer N1. Furthermore, it executes... Figure 3 The first-level filter for related operations can be implemented through first-level filters FLF1~FLF4, and executed. Figure 3 The second-level filter for related operations can be implemented through the second-level filter SLF1.

[0078] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A memory system comprising: A storage controller is used to receive a first-stage result; A memory is used to perform in-memory operations on the first stage result to produce a second stage result; A first-level filter is used to filter the second-stage results to produce a third-stage result; as well as A second-stage filter is used to filter the results of the third stage to produce a fourth-stage result. When the number of multiple data results in the third stage exceeds a preset threshold, the memory system outputs the fourth stage result. When the number of these data results in the third stage is less than or equal to the preset threshold number, the memory system outputs the third stage result.

2. The memory system according to claim 1, wherein The storage controller is used to output the third-stage result or the fourth-stage result to a processing device.

3. The memory system of claim 2, wherein the second-stage filter is further configured to sort the data results in the third-stage result according to a plurality of data values ​​of the data results in the third-stage result, and select N1 data results from the third-stage result as the fourth-stage result, and N1 is a positive integer less than or equal to the preset threshold number.

4. The memory system of claim 3, wherein N1 is determined based on a verification index corresponding to a first test dataset.

5. The memory system of claim 4, wherein the verification index is accuracy.

6. The memory system of claim 5, wherein the first-level filter is further configured to filter the first test dataset to generate a second test dataset. The second-level filter is also used to filter the second test dataset to generate a third test dataset, and The memory system is used to adjust N1 based on a first accuracy corresponding to the second test dataset and a second accuracy corresponding to the third test dataset.

7. The memory system of claim 6, wherein the memory system is further configured to compare the second accuracy rate with a preset accuracy rate. When the second accuracy is less than the preset accuracy, the memory system increases by N1, and When the second accuracy is greater than or equal to the preset accuracy, the memory system determines N1.

8. A method for in-memory computation, comprising: A first-stage result is filtered through a first-level filter to produce a second-stage result; Compare the number of data results in the second stage with a preset threshold number; and When the number of these data results in the second stage exceeds the preset threshold, the second stage results are filtered by a second-level filter to generate a third stage result.

9. The method of operation according to claim 8, wherein the first-stage filter is used to set a threshold to filter the first-stage result.

10. The method of operation according to claim 9, wherein filtering the second-stage result to produce the third-stage result comprises: The second-level filter sorts the data results in the second-stage results based on multiple data values; and The second-level filter selects N1 data results from the second-stage results as the third-stage results. Where N1 is a positive integer less than or equal to the preset threshold number.

Images