Method and apparatus for performing video processing matrix operations within a memory array
By converting the memory array into a matrix structure and performing matrix operations therein, the problem of processor-memory interface limitations is solved, efficient video processing is achieved, computing speed is improved and power consumption is reduced.
Patent Information
- Application Number
- CN202011251972.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-11-20
- Filing Date
- 2020-11-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-11-11
AI Technical Summary
In the prior art, the interface limitation between the processor and the memory has affected the video processing speed, especially in applications with frequent communication, it is difficult to effectively improve the speed of traditional video processing.
Convert the memory array into a matrix structure and perform matrix operations in it, perform matrix transformation operations through analog media storage and sensing analog values, and implement matrix multiplication using resistive random access memory (ReRAM) units to reduce interface transactions and improve calculations efficiency.
Calculating multiple elements of the matrix atomically reduces the number of communications between the processor and memory, improves the speed and efficiency of video processing, reduces power consumption, and maintains the non-volatile characteristics of the matrix structure when not in use. Save power.
Smart Images

Figure CN112825257B_ABST
Abstract
Description
[0001] Priority and related applications
[0002] This application claims priority to U.S. Patent Application No. 16 / 689,981, filed on November 20, 2019, entitled “METHODS AND APPARATUS FOR PERFORMING VIDEO PROCESSING MATRIXOPERATIONS WITHIN A MEMORY ARRAY,” the entire contents of which are incorporated herein by reference.
[0003] The subject matter of this application is also related to the following: commonly owned and co-pending U.S. Patent Application No. 16 / 002,644, filed on June 7, 2018, entitled “AN IMAGE PROCESSOR FORMED IN AN ARRAY OF MEMORY CELLS”; U.S. Patent Application No. 16 / 211,029, filed on December 5, 2018, entitled “METHODS AND APPARATUS FOR INCENTIVIZING PARTICIPATION IN FOG NETWORKS”; U.S. Patent Application No. 16 / 211,029, filed on December 5, 2018, entitled “METHODS AND APPARATUS FOR ROUTINEBASED FOG NETWORKS”; U.S. Patent Application No. 16 / 211,029, filed on December 5, 2018, entitled “METHODS AND APPARATUS FOR ROUTINEBASED FOG NETWORKS NETWORKING”; Serial No. 16 / 276,461, filed on February 14, 2019, entitled “METHODS AND APPARATUS FOR CHARACTERIZING MEMORY DEVICES”; Serial No. 16 / 276,471, filed on February 14, 2019, entitled “METHODS AND APPARATUS FOR CHECKING THE RESULTS OF CHARACTERIZED MEMORY SEARCHES”; and Serial No. 16 / 276,489, filed on February 14, 2019, entitled “Methods and Apparatus for Maintaining Characterized Memory Devices,” each of which is incorporated herein by reference in its entirety. Technical Field
[0004] The following generally relates to the field of data processing and device architecture. Specifically, in one exemplary aspect, a processor-memory architecture is disclosed that converts a memory array into a matrix structure for matrix transformation and performs video processing matrix operations therein. Background Art
[0005] Memory devices are widely used to store information in various electronic devices such as computers, wireless communication devices, cameras, digital displays, etc. Information is stored by programming different states of the memory device. For example, a binary device has two states, usually represented by a logical "1" or a logical "0". To access the stored information, the memory device can read (or sense) the stored state in the memory device. To store information, the memory device can write (or program) the state into the memory device. So-called volatile memory devices may require power to maintain this stored information, while non-volatile memory devices can store information persistently even after the memory device itself has been power cycled, for example. Different memory manufacturing methods and constructions achieve different capabilities. For example, dynamic random access memory (DRAM) provides high-density volatile storage inexpensively. Initial research is directed to resistive random access memory (ReRAM), which has non-volatile performance similar to DRAM.
[0006] Processor devices are often used in conjunction with memory devices to perform a myriad of different tasks and functions. During operation, the processor executes computer-readable instructions (commonly referred to as "software") from the memory. The computer-readable instructions define basic arithmetic, logic, control, input / output (I / O) operations, etc. As is well known in the computing arts, relatively basic computer-readable instructions can perform a variety of complex behaviors when combined in sequence. Processors tend to emphasize circuit construction and manufacturing techniques that are different from memory devices. For example, processing performance is generally related to clock rate, so most processor manufacturing methods and constructions emphasize very high-speed transistor switch structures, etc.
[0007] Over time, the speed and power consumption of processors and memory have increased. Typically, these improvements are the result of reduced device size, as electrical signaling is physically limited by the dielectric and distance of the transmission media. As mentioned earlier, most processors and memories are made with different manufacturing materials and technologies. Therefore, even if processors and memories continue to improve, the physical interface between the processor and the memory is the "bottleneck" of the overall system performance. More directly, no matter how fast the processor or memory can run independently, the performance of the combined system of processor and memory is limited by the transfer rate allowed by the interface. This phenomenon has several common names, such as "processor-memory wall", "von Neumann bottleneck effect", etc.
[0008] Video processing (e.g., compression) methods are one type of processing application known to require frequent communication between memory and the processor. As a result, the speed of conventional video processing is particularly affected by the above-mentioned processor-memory interface limitations. Therefore, there is a clear need for improved apparatus and methods to reduce the impact of these limitations, including in such video processing applications. Summary of the invention
[0009] In particular, the present invention provides a method and apparatus for converting a memory array into a matrix structure for matrix transformation and performing matrix operations therein.
[0010] In one aspect of the present disclosure, a non-transitory computer-readable medium is disclosed. In one exemplary embodiment, the non-transitory computer-readable medium includes: a memory cell array, wherein each memory cell of the memory cell array is configured to store a digital value as an analog value in an analog medium; a memory sensing component, wherein the memory sensing component is configured to read the analog value of a first memory cell as a first digital value; and logic. In one embodiment, the memory cell array includes a stack of at least two memory cell sub-arrays. In one variant, at least some of the memory cell sub-arrays are configured to operate in parallel with each other.
[0011] In one embodiment, at least some of the memory cell subarrays are connected to a single memory sensing component.
[0012] In another implementation, individual memory cell subarrays are connected to individual memory sensing elements.
[0013] In another variation, the memory cell is configured to store the analog value as impedance or conductance in the memory cell.
[0014] In another exemplary embodiment, the logic is further configured to: receive a subjective operation code; cause the memory cell array to operate as a matrix multiplication unit (MMU) based on the matrix transformation operation code; wherein each memory cell of the MMU modifies an analog value in an analog medium according to the matrix transformation operation code and a matrix transformation operand; configure the memory sensing component to convert the analog value of the first memory cell into a second digital value according to the matrix transformation operation code and the matrix transformation operand; and in response to reading the matrix transformation operand into the MMU, write a matrix transformation result based on the second digital value.
[0015] In one variant, the matrix transform opcode indicates the size of the MMU. In one such variant, the matrix transform opcode corresponds to a frequency domain transform operation. In one embodiment, the frequency domain transform operation is a DCT (discrete cosine transform) operation, and at least some aspects of the size of the MMU correspond to the size of the DCT operation (e.g., the size of the MMU may correspond to the size of a DC operation, or a multiple or fraction thereof). In an exemplary variant, the frequency domain transform operation spans at least one other MMU.
[0016] In another variant, the matrix transformation operand includes a vector derived from the pixel values of the pixel image block. In one embodiment, the pixel value corresponds to red, green, blue, brightness (Y), chromaticity red (Cr) or chromaticity blue (Cr) pixel value. In an exemplary configuration, the pixel image block is an NxN image block (e.g., 8x8 pixels, 16x16 pixels, etc.).
[0017] In one variant, the matrix transformation operation code identifies one or more analog values corresponding to one or more memory cells. In one such variant, one or more analog values corresponding to one or more memory cells are stored in a lookup table (LUT) data structure. In one embodiment, one or more analog values include DCT transformation matrix values.
[0018] In one variation, each memory unit of the MMU comprises a resistive random access memory (ReRAM) unit; and each memory unit of the MMU multiplies analog values in the analog medium according to a matrix transformation operation code and a matrix transformation operand.
[0019] In one variation, each memory unit of the MMU further accumulates the analog value in the analog medium with a previous analog value.
[0020] In one variation, the first digital value is characterized by a first base of two (2); and the second digital value is characterized by a second base greater than two (2).
[0021] In one aspect of the present disclosure, a device is disclosed. In one embodiment, the device includes a processor coupled to a non-transitory computer-readable medium; wherein the non-transitory computer-readable medium includes one or more instructions, which, when executed by the processor, cause the processor to: write a matrix transformation operation code and a matrix transformation operand to the non-transitory computer-readable medium; wherein the matrix transformation operation code causes the non-transitory computer-readable medium to operate a memory cell array as a matrix structure; wherein the matrix transformation operand modifies one or more analog values of the matrix structure; and read a matrix transformation result from the matrix structure.
[0022] In one variation, the non-transitory computer-readable medium further comprises one or more instructions which, when executed by the processor, cause the processor to: capture image data comprising one or more captured color values; and wherein a matrix transformation operand comprises the one or more captured color values and a matrix transformation result comprises the one or more shifted color values.
[0023] In one variant, the non-transitory computer-readable medium further comprises one or more instructions that, when executed by the processor, cause the processor to: receive video data comprising one or more image blocks; wherein a matrix transform operand comprises one or more image blocks, and a matrix transform result comprises one or more frequency domain image coefficients; and wherein one or more analog values of the matrix structure accumulate one or more frequency domain image coefficients from the video data over time. In one embodiment, the one or more frequency domain image coefficients comprise DCT image coefficients.
[0024] In one variation, the matrix transformation operation code causes the non-transitory computer-readable medium to operate another memory cell array as another matrix structure; and logically combine a matrix transformation result associated with the matrix structure and another matrix transformation result associated with another matrix structure.
[0025] In one variation, one or more analog values of the matrix structure are stored in a look-up table (LUT) data structure. In one variation, at least some of the values of the matrix structure are stored in a portion of the memory cell that is not configured as a matrix structure.
[0026] In one aspect of the present disclosure, a method for performing a transformation matrix operation is disclosed. In one embodiment, the method includes: receiving a matrix transformation operation code; configuring a memory cell array of a memory into a matrix structure based on the matrix transformation operation code; configuring a memory sensing component based on the matrix transformation operation code; and writing a matrix transformation result from the memory sensing component in response to reading a matrix transformation operand into the matrix structure. In one variant, configuring the matrix structure includes configuring the memory cell array with values of a DCT transformation matrix, and the matrix transformation result includes a matrix of DCT image coefficients. In one embodiment, the matrix structure is configured with values of two matrices: a matrix of positive DCT transformation matrix values and a matrix of negative DCT transformation matrix values. In one variant, multiple memory cell arrays are configured with values of the DCT transformation matrix.
[0027] In one variation, configuring the memory cell array includes connecting a plurality of word lines and a plurality of bit lines corresponding to row and column dimensions associated with the matrix structure.
[0028] In one variation, the method further includes determining row and column dimensions from the matrix transform opcode.
[0029] In one variation, configuring the memory cell array includes setting one or more analog values in a matrix structure based on a look-up table (LUT) data structure.
[0030] In one variation, the method includes identifying an entry from a LUT data structure based on a matrix transform opcode.
[0031] In one variation, the memory sensing component is configured such that the matrix transformation result has a cardinality greater than two (2).
[0032] In one aspect, an apparatus configured to configure a memory device into a matrix structure is disclosed. In one embodiment, the apparatus includes: a memory; a processor configured to access the memory; and a pre-processor logic configured to allocate one or more memory portions for use as a matrix structure.
[0033] In another aspect of the present disclosure, a computer image processing device is disclosed, which is configured to dynamically configure a memory into a matrix structure. In one embodiment, the computer image processing device includes: a camera interface; a digital processor device in data communication with the camera interface; and a memory in data communication with the digital processor device and containing at least one computer program.
[0034] In another aspect of the present disclosure, a computer video processing device is disclosed that is configured to dynamically configure a memory into a matrix structure. In one embodiment, the computer video processing device includes: a camera interface; a digital processor device in data communication with the camera interface; and a memory that is in data communication with the digital processor device and contains at least one computer program.
[0035] In another aspect of the present disclosure, a computer wireless access node device is disclosed that is configured to dynamically configure a memory into a matrix structure. In one embodiment, the computer wireless access node includes: a wireless interface that is configured to transmit and receive RF waveforms in a portion of a spectrum; a digital processor device that is in data communication with the wireless interface; and a memory that is in data communication with the digital processor device and includes at least one computer program.
[0036] In additional aspects of the present disclosure, a computer readable device is described. In one embodiment, the device includes a storage medium configured to store one or more computer programs in or in conjunction with a characterization memory. In one embodiment, the device includes a program memory or a HDD or SDD on a computer controller device. In another embodiment, the device includes a program memory, HDD or SSD on a computer access node.
[0037] These and other aspects will become apparent when considered in light of the disclosure provided herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1A is a diagram of the processor-memory architecture and a graphical depiction of the associated matrix operations.
[0039] Figure 1B is a diagram of the processor-PIM architecture and a graphical depiction of the associated matrix operations.
[0040] Figure 2 is a logical block diagram of one exemplary implementation of a memory device according to various principles of the present disclosure.
[0041] Figure 3 is an exemplary side-by-side illustration of a first memory device configuration and a second memory device configuration.
[0042] Figure 4 is a graphical depiction of a half-matrix operation performed in accordance with the principles of the present disclosure.
[0043] Figure 5A is a block diagram of a first embodiment of a memory device configured for DCT matrix operations according to aspects of the present invention.
[0044] Figure 5B is a block diagram of a second embodiment of a memory device configured for DCT matrix operations according to aspects of the present invention.
[0045] Fig. 6A is a block diagram of a third embodiment of a memory device configured for DCT matrix operations according to aspects of the present invention.
[0046] Figure 6B is a block diagram of a fourth embodiment of a memory device configured for DCT matrix operations according to aspects of the present invention.
[0047] Fig. 7A is a logical block diagram of an exemplary implementation of a processor-memory architecture.
[0048] Figure 7B is a ladder diagram illustrating one exemplary embodiment of performing a set of matrix operations according to the principles of the present disclosure.
[0049] Figure 7C is a ladder diagram illustrating another exemplary embodiment of performing a set of matrix operations according to the principles of the present disclosure.
[0050] Figure 8 is a logic flow diagram of an exemplary method of converting a memory array into a matrix structure and performing matrix operations therein.
[0051] Fig. 8A is a logic flow diagram of an exemplary method for performing DCT matrix operations using a matrix structure.
[0052] All drawings © 2019 Micron Technology, Inc. All rights reserved. DETAILED DESCRIPTION
[0053] Reference is now made to the drawings, wherein like numerals refer to like parts throughout.
[0054] As used herein, the term "application" (or "app") is generally, but not limited to, an executable software unit that implements a specific function or theme. The themes of applications vary widely in any number of specifications and functions (e.g., on-demand content management, e-commerce transactions, brokerage transactions, home entertainment, calculators, etc.), and an application may have more than one theme. An executable software unit typically runs in a predetermined environment; for example, the unit may include a downloadable application program that runs in an operating system environment.
[0055] As used herein, the term "computer program" or "software" is meant to include any sequence or human or machine recognizable steps for performing a function. Such a program may be presented in virtually any programming language or environment, including, for example, C / C++, Fortran, COBOL, PASCAL, assembly language, markup languages (e.g., HTML, SGML, XML, VoXML), etc., and object-oriented environments such as Common Object Request Broker Architecture (CORBA), Java, and the like. TM (including J2ME, Java Bean, etc.), Register Transfer Language (RTL), VHSIC (Very High Speed Integrated Circuit) Hardware Description Language (VHDL), Verilog, etc.
[0056] As used herein, the term "decentralized" or "distributed" is not limited to a configuration or network architecture of multiple computer devices capable of performing data communications with each other, rather than requiring a given device to communicate through a designated (e.g., central) network entity such as a server device. For example, a decentralized network enables direct peer-to-peer data communications between multiple UEs (e.g., wireless user devices) that make up the network.
[0057] As used herein, the term "distributed unit" (DU) is not limited to a distributed logical node within a wireless network infrastructure. For example, a DU may be embodied as a next-generation Node B (gNB) DU (gNB-DU) controlled by the gNB CU described above. A gNB-DU may support one or more cells; a given cell is supported by only one gNB-DU.
[0058] As used herein, the terms "Internet" and "internet" are used interchangeably to refer to cross-networks including, but not limited to, the Internet. Other common examples include, but are not limited to: a network of external servers, "cloud" entities (e.g., storage or memory that is not local to a device, typically accessible at any time via a network connection, etc.), service nodes, access points, controller devices, client devices, etc. The 5G service core network and network components (e.g., DU, CU, gNB, small cell or femtocell, external nodes with 5G capabilities) residing in backhaul, fronthaul, crosshaul, or the "edge" of its neighboring residences, businesses, and other occupied areas can be included in the "Internet".
[0059] As used herein, the term "memory" includes any type of integrated circuit or other storage device suitable for storing digital data, including but not limited to random access memory (RAM), pseudo-static RAM (PSRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM) including double data rate (DDR) class memory and graphics DDR (GDDR) and its variants, ferroelectric RAM (FeRAM), magnetic RAM (MRAM), resistive RAM (ReRAM), read-only memory (ROM), programmable ROM (PROM), electrically erasable PROM (EEPROM or EEPROM), etc. 2 PROM), DDR / 2SDRAM, EDO / FPMS, reduced latency DRAM (RLDRAM), static RAM (SRAM), “flash” memory (e.g., NAND / NOR), phase change memory (PCM), 3-dimensional cross-point memory (3D Xpoint), stacked memory such as HBM / HBM2, and magnetoresistive RAM (MRAM) such as spin torque transfer RAM (STT RAM).
[0060] As used herein, the terms "microprocessor" and "processor" or "digital processor" are generally meant to encompass all types of digital processing devices, including but not limited to digital signal processors (DSPs), reduced instruction set computers (RISCs), general purpose processors (GPPs), microprocessors, gate arrays (e.g., FPGAs), PLDs, reconfigurable computer structures (RCFs), array processors, secure microprocessors, and application specific integrated circuits (ASICs). Such digital processors may be contained on a single integral IC die, or distributed across multiple components.
[0061] As used herein, the term "server" refers to any computer component, system, or entity, regardless of form, suitable for providing data, files, applications, content, or other services to one or more other devices or entities on a computer network.
[0062] As used herein, the term "memory" is not limited to a computer hard drive (e.g., a hard disk drive (HDD), a solid-state drive (SDD)), a flash drive, a DVR device, memory, a RAID device or array, optical media (e.g., CD-ROM, laser disc, Blu-ray, etc.), or any other device or media capable of storing content or other information, including semiconductor devices capable of maintaining data in the absence of power (e.g., those devices described herein as memory). Common examples of memory devices for storage include, but are not limited to, ReRAM, DRAM (e.g., SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, DDR4 SDRAM, GDDR, RLDRAM, LPDRAM, etc.), DRAM modules (e.g., RDIMM, VLP RDIMM, UDIMM, VLP UDIMM, SODIMM, SORDIMM, Mini-DIMM, VLP Mini-DIMM, LRDIMM, NVDIMM, etc.), managed NAND, NAND flash memory (e.g., SLC NAND, MLC NAND, TLS NAND, serial NAND, 3D NAND, etc.), NOR flash memory (e.g., parallel NOR, serial NOR, etc.), multi-chip packages, hybrid memory cubes, memory cards, solid-state storage (SSS), and any number of other memory devices.
[0063] As used herein, the term "Wi-Fi" is not limited to and, where applicable, refers to any variant of the following: IEEE Standard 802.11 or related standards including 802.11a / b / g / n / s / v / ac / ad / av or 802.11-2012 / 2013, 802.11-2016, and Wi-Fi Direct (including, in particular, the "Wi-Fi Peer-to-Peer (P2P) Specification", the entire contents of which are incorporated herein by reference).
[0064] As used herein, the term "wireless" means any wireless signal, data, communication or other interface, including but not limited to Wi-Fi, Bluetooth / BLE, 3G / 4G / 4.5G / 5G / B5G (3GPP / 3GPP2), HSDPA / HSUPA, TDMA, CBRS, CDMA (e.g., IS-95A, WCDMA, etc.), FHSS, DSSS, GSM, PAN / 802.15, WiMAX (802.16), 802.20, Z-wave, narrowband / FDMA, OFDM, PCS / DCS, LTE / LTE-A / LTE-U / LTE-LAA, analog cellular, CDPD, satellite systems, millimeter wave or microwave systems, acoustic, and infrared (i.e., IrDA).
[0065] Overview
[0066] The above-described "processor-memory wall" performance limitation can be very severe in situations where a processor-memory architecture repeats similar operations on large data sets. In such situations, the processor-memory architecture must repeatedly transfer, operate on, and store each element of the data set individually. For example, a 4x4 (sixteen (16) elements) matrix multiplication requires four (4) times as many 2x2 (four (4) elements) matrix multiplications. In other words, matrix operations scale exponentially with the size of the matrix.
[0067] Matrix transforms are commonly used in many different applications and may take up a disproportionate amount of processing and / or memory bandwidth. For example, many image signal processing (ISP) techniques commonly use matrix transforms for, for example, color interpolation, white balance, color correction, color conversion, etc. Many communication technologies employ fast Fourier transforms (FFTs) and matrix multiplications for beamforming and / or massive multiple-input multiple-output (MIMO) channel processing.
[0068] In particular, video, audio, and image compression typically use discrete cosine transforms (DCTs) to identify video image data that can be removed with minimal fidelity loss. For example, DCT-based compression is used in JPEG, MP3, AAC, MPEG, H.264 / AVC, H.265 / HEVC image / image / audio standards, and HDR (high dynamic range) applications. DCT-based image / video compression involves several matrix multiplication operations. Various embodiments of the present disclosure are directed to converting a memory array into a matrix structure for matrix transformations and performing video processing matrix operations therein.
[0069] The exemplary embodiments described herein perform matrix transformations in a memory device comprising a matrix structure and a matrix multiplication unit (MMU). In an exemplary embodiment, the matrix structure uses a "crossbar" construction of resistor elements. Each resistor element stores an impedance level representing a corresponding matrix coefficient value. The crossbar connection can be driven by an electrical signal representing an input vector as an analog voltage. The resulting signal can be converted from an analog voltage to a digital value by the MMU to generate a vector-matrix product. In some cases, the MMU can also perform various other logical operations in the digital domain.
[0070] Different from the existing solutions that iterate each element of the matrix to calculate the element value, the vertical and horizontal matrix structure described below calculates multiple elements of the matrix "in an atomic manner" (i.e., in a single processing cycle). For example, at least a portion of the vector-matrix product can be calculated in parallel. The "atomicity" of the calculation based on the matrix structure produces significant processing improvements relative to the iterative replacement scheme. In particular, when the iterative technique grows with the matrix size, the atomic matrix structure calculation is independent of the matrix dimension. In other words, the NxN vector-matrix product can be completed in a single atomic instruction.
[0071] Various embodiments of the present disclosure derive and / or use matrix coefficient values internally to further minimize interface transactions. As described in more detail herein, many useful matrix transforms can be characterized by "structurally defined dimensions" and performed with "structurally defined coefficients". Structural definition refers to those aspects of matrix calculations that are defined for a particular matrix structure (e.g., the rank and / or size of the matrix); in other words, the matrix coefficients can be inferred from the matrix structure without being explicitly provided through the processor-memory interface. For example, as described in more detail below, the DCT transform matrix values (i.e., DCT "rotation factors") are a function of the matrix size. The "rotation factors" used for the Fast Fourier Transform (FFT) are also a function of the matrix size, while ISP filtering and / or massive MIMO channel coding techniques can use, for example, codebooks of predefined matrices and / or matrices with known structures and weights.
[0072] In short, practical limitations in component manufacturing limit the capabilities of each element within an individual memory device. For example, most memory arrays are designed to discern only two (2) states (logical "1", logical "0"). While existing memory sensing components can be expanded to discern higher levels of precision (e.g., four (4) states, eight (8) states, etc.), increasing the precision of memory sensing components may be impractical to support the precision required for large transforms typically used in, for example, video compression, mathematical transforms, etc.
[0073] To these ends, various embodiments of the present disclosure logically combine one or more matrix structures and / or MMUs to provide higher precision and / or processing complexity than would otherwise be possible. In one such embodiment, a first matrix structure and / or MMU may be used to compute positive vector-matrix products, while a second matrix structure and / or MMU may be used to compute negative vector-matrix products. Positive and negative vector-matrix products may be summed to determine a net vector-matrix product.
[0074] In another such embodiment, a plurality of simple matrix transformations can be used to implement a larger matrix transformation. For example, in order to perform a DCT operation, the matrix-matrix multiplication of two NxN matrices can be decomposed into N individual matrix-vector multiplications (that is, an NxN matrix multiplied by a vector of N size). The various parts of the DCT operation in the processing matrix structure can also be sorted or parallelized according to many design considerations. Given the contents of the present disclosure, other examples of logical matrix operations can be replaced with equivalent successes (e.g., decomposition, common matrix multiplication, etc.).
[0075] Some applications can save a lot of power by shutting down system components when not in use. For example, video compression can benefit from "sleep" during video blanking intervals (when there is no video data activity), etc. However, the sleep program usually requires the processor and / or memory to transfer data from the operating volatile memory to the non-volatile storage memory so that the data is not lost when the power is off. A wake-up process is also required to retrieve the stored information from the non-volatile memory. Transferring data back and forth between memories is an inefficient use of processor-memory bandwidth. Therefore, various embodiments disclosed herein utilize the "non-volatile" characteristics of the matrix structure. In such an embodiment, the matrix structure can maintain its matrix coefficient values even when the memory has no power. More directly, the non-volatile characteristics of the matrix structure enable the processor and memory to switch to a dormant / low-power mode or perform other tasks without shuffling data from volatile memory to non-volatile memory, and vice versa.
[0076] Various other combinations and / or variations of the foregoing will be readily apparent to those of ordinary skill in the art given the present disclosure.
[0077] Detailed Description of Exemplary Embodiments
[0078] Exemplary embodiments of the apparatus and method of the present disclosure are now described in detail. Although these exemplary embodiments are described in the context of a specific processor and / or memory configuration, the general principles and advantages of the present disclosure can be extended to other types of processors and / or memory technologies, so the following is only exemplary in nature.
[0079] It will also be understood that, while generally described in the context of consumer devices (within camera devices, video codecs, cellular phones, and / or network base stations), the present disclosure may be readily adapted for use with other types of devices, including, for example, server devices, Internet of Things (IoT) devices, and / or devices for personal, corporate, or even governmental use. Other applications are also possible.
[0080] Other features and advantages of the present disclosure will be immediately recognized by those skilled in the art with reference to the accompanying drawings and the detailed description of exemplary embodiments given below.
[0081] Processor Memory Architecture -
[0082] Figure 1A A common processor-memory architecture 100 for illustrating matrix operations is shown. Figure 1A As shown, the processor 102 is connected to the memory 104 via the interface 106. In the illustrative example, the processor multiplies the elements of the input vector a with the matrix M to calculate the vector-matrix product b. Mathematically, the input vector a is considered to be a single-column matrix with a number of elements equal to the number of rows in the matrix M.
[0083] To compute the first element of the vector-matrix product b0, the processor must iterate over each permutation of the elements of the input vector a for each element within a row of the matrix M. During the first iteration, the first element of the input vector a0 is read, the current value of the vector-matrix product b0 is read, and the corresponding matrix coefficient value M is read. 0,0 The three (3) read values are used in a multiply-accumulate operation to produce the "intermediate" vector-matrix product b0. Specifically, the multiply-accumulate operation computes: (a0·M 0,0 )+b0 and write the resulting value back to b0. It is worth noting that b0 is an "intermediate value". After the first iteration but before the second iteration, the intermediate value b0 may not correspond to the final value of the vector-matrix product b0.
[0084] During the second iteration, the second element of the input vector a1 is read, the previously calculated intermediate value b0 is retrieved, and the second matrix coefficient value M is read. 1,0 The three (3) read values are used in a multiply-accumulate operation to generate the first element of the vector-matrix product b0. The second iteration completes the calculation of b0.
[0085] Although not explicitly shown, the above iterative process is also performed to generate the second element of the vector-matrix product b1. In addition, although the above example is a 2x2 vector-matrix product, the techniques described therein are generally extended to support vector-matrix calculations of any size. For example, for each of the three (3) rows of the matrix, the 3x3 vector-matrix product calculation is iterated on the input vector of three (3) elements; therefore, nine (9) iterations are required. A 1024x1024 matrix operation (which is not uncommon for many applications) will require more than one million iterations. More directly, the above iterative process scales exponentially according to the matrix dimension.
[0086] Although the foregoing discussion is presented in the context of a vector-matrix product, one of ordinary skill in the art will readily appreciate that a matrix-matrix product can be performed as a series of vector-matrix products. For example, a first vector-matrix product corresponding to a first single-column matrix of the input vector is calculated, a second vector-matrix product corresponding to a second single-column matrix of the input vector is calculated, and so on. Thus, a 2x2 matrix-matrix product will require two (2) vector-matrix calculations (i.e., a total of 2x4=8), and a 3x3 matrix-matrix product will require three (3) vector-matrix calculations (i.e., a total of 3x9=27).
[0087] A person skilled in the art will readily understand that Figure 1A Each iteration of the process described in becomes a bottleneck at the bandwidth limitations of interface 106 (the "processor-memory wall"). Even though the processor and memory may have extremely high bandwidth internal buses, the processor-memory system can only communicate at a speed that interface 106 can support electrical signaling, based on the dielectric properties of the material (typically copper) used in interface 106 and the transmission distance (~1-2 cm). In addition, interface 106 may also contain various additional signal conditioning, amplification, noise correction, error correction, parity calculations, and / or other interface-based logic that further reduces transaction times.
[0088] A common method to improve matrix operation performance is to perform matrix operations in a local processor cache. Unfortunately, the local processor cache takes up processor die space and has a much higher per-bit manufacturing cost compared to, for example, comparable memory devices. As a result, the local cache size of a processor is usually much smaller (e.g., several megabytes) than its memory (which can be several gigabytes). From a practical point of view, a smaller local cache is a hard limit on the maximum amount of matrix operations that can be performed locally in a processor. As another disadvantage, large matrix operations lead to poor cache utilization because only one row and one column are accessed at a time (e.g., for a 1024x1024 vector-matrix product, only 1 / 1024 of the cache is in effective use during a single iteration). Therefore, although the processor cache implementation is acceptable for small matrices, as the complexity of matrix operations increases, this technique becomes increasingly undesirable.
[0089] Another common approach is the so-called Processor-In-Memory (PIM). Figure 1B One such processor-PIM architecture 150 is shown. As shown, processor 152 is connected to memory 154 via interface 156. Memory 154 further includes PIM 162 and memory array 164; PIM 162 is tightly coupled to memory array 164 via internal interface 166.
[0090] Similar to the above Figure 1A The process described in Figure 1B The processor-PIM structure 150 of FIG. 1 multiplies the elements of the input vector a by the matrix M to calculate the vector-matrix product b. However, the PIM 162 performs internal reads, multiply-accumulates, and writes to the memory 164 via the internal interface 166. The internal interface 166 is much shorter than the external interface 156; in addition, the internal interface 166 can operate locally without, for example, signal conditioning, amplification, noise correction, error correction, parity calculation, etc.
[0091] Although processor-PIM architecture 150 provides substantial improvements in performance over, for example, processor-memory architecture 100, processor-PIM architecture 150 may have other disadvantages. For example, manufacturing technology ("silicon processes") differ substantially between processors and memory devices because each silicon process is optimized for different design criteria. For example, a processor silicon process may use thinner transistor structures than a memory silicon process; thinner transistor structures provide faster switching (which improves performance) but suffer from greater leakage (which is undesirable for memory retention). As a result, manufacturing PIM 162 and memory array 164 in the same wafer results in at least one of them being implemented in a suboptimal silicon process. Alternatively, PIM 162 and memory array 164 may be implemented in separate dies and connected together; die-to-die communication typically increases manufacturing cost and complexity, and may suffer from various other impairments (e.g., introduced by process discontinuities, etc.).
[0092] Furthermore, one of ordinary skill in the relevant art will readily appreciate that the PIM 162 and memory array 164 are "hardened" components; the PIM 162 cannot store data, nor can the memory 164 perform computations. In fact, once the memory 154 is manufactured, the memory cannot be altered to, for example, store more data and / or increase / decrease PIM performance / power consumption. Such memory devices are often customized specifically for their application; this is expensive to design and modify, and in many cases the memory is "proprietary" and / or customer / manufacturer specific. Furthermore, because technology changes so rapidly, these devices quickly become obsolete.
[0093] For various reasons, there is a need for an improved solution for matrix operations within a processor and / or memory. Ideally, such a solution would implement matrix operations within a memory device in a manner that minimizes the performance bottleneck of the processor-memory wall. Furthermore, such a solution should be flexible to accommodate a variety of different matrix operations and / or matrix sizes.
[0094] Exemplary Memory Devices -
[0095] Figure 22 is a logical block diagram of an exemplary implementation of a memory device 200 fabricated according to various principles of the present disclosure. The memory device 200 may include a plurality of partitioned memory cell arrays 220. In some implementations, each partitioned memory cell array 220 may be partitioned at the time of device fabrication. In other implementations, the memory cell arrays 220 may be partitioned dynamically (i.e., after device fabrication). The memory cell arrays 220 may each include a plurality of banks, wherein each bank includes a plurality of word lines, a plurality of bit lines, and a plurality of memory cells, the plurality of memory cells being arranged, for example, at intersections of the plurality of word lines and the plurality of bit lines. The selection of the word lines may be performed by a row decoder 216, while the selection of the bit lines may be performed by a column decoder 218.
[0096] The plurality of external terminals included in the semiconductor device 200 may include an address terminal 260, a command terminal 262, a clock terminal 264, a data terminal 240, and a power supply terminal 250. The address terminal 260 may be provided with an address signal and a bank address signal. The address signal and the bank address signal provided to the address terminal 260 are transmitted to the address decoder 204 via the address input circuit 202. The address decoder 204 receives, for example, the address signal and provides a decoded row address signal to the row decoder 216 and a decoded column address signal to the column decoder 218. The address decoder 204 may also receive a bank address signal and provide the bank address signal to the row decoder 216 and the column decoder 218.
[0097] The command terminal 262 is provided with a command signal to the command input circuit 206. The command terminal 262 may include one or more individual signals, such as a row address strobe (RAS), a column address strobe (CAS), and a read / write (R / W). The command signal input to the command terminal 262 is provided to the command decoder 208 by the command input circuit 206. The command decoder 208 may decode the command signal 262 to generate various control signals. For example, RAS may be asserted to specify the row where data is to be read / written, and CAS may be asserted to specify the column where data is to be read / written. In some variants, the R / W command signal determines whether the contents of the data terminal 240 are written to or read from the memory cell 220.
[0098] During a read operation, read data may be output from the data terminal 240 from the outside through the read / write amplifier 222 and the input / output circuit 224. Similarly, when a write command is issued and a row address and a column address are provided in time with the write command, a write data command may be provided to the data terminal 240. The write data command may be provided to a given memory cell array 220 through the input / output circuit 224 and the read / write amplifier 222 and written into a memory cell specified by the row address and the column address. According to some embodiments, the input / output circuit 224 may include an input buffer.
[0099] An external clock signal may be provided to the clock terminal 264 for synchronous operation. In one variation, the clock signal is a single-ended signal; in other variations, the external clock signals may be complementary to each other (differential signaling) and provided to the clock input circuit 210. The clock input circuit 210 receives the external clock signal and conditions the clock signal to ensure that the resulting internal clock signal has sufficient amplitude and / or frequency for subsequent locked loop operation. The conditioned internal clock signal is provided to a feedback mechanism (internal clock generator 212) that provides a stable clock for the internal memory logic. Common examples of the internal clock generation logic 212 include, but are not limited to: digital or analog phase-locked loop (PLL), delay-locked loop (DLL), and / or frequency-locked loop (FLL) operations.
[0100] In an alternative variation (not shown), memory 200 may rely on external timing (i.e., without its own internal clock). For example, a phase-controlled clock signal may be provided externally to input / output (IO) circuitry 224. This external clock may be used to clock in write data, and clock out read data. In this variation, IO circuitry 224 provides clock signals to each corresponding logic block (e.g., address input circuitry 202, address decoder 204, command input circuitry 206, command decoder 208, etc.).
[0101] Power supply potentials may be provided for power supply terminals 250. In some variants (not shown), these power supply potentials may be provided by input / output (I / O) circuits 224. In some embodiments, the power supply potentials may be isolated from the I / O circuits 224 so that the power supply noise generated by the IO circuits 224 does not propagate to other circuit blocks. These power supply potentials are regulated by internal power supply circuits 230. For example, the internal power supply circuits 230 may generate various internal potentials, such as removing noise and / or parasitic activity, and boost or buck potentials provided from the power supply potentials. The internal potentials may be used, for example, for address circuits (202, 204), command circuits (206, 208), row and column decoders (216, 218), RW amplifiers 222, and / or any of various other circuit blocks.
[0102] When the internal power circuit 230 can provide sufficient internal voltage for the power-up sequence, the power-on reset circuit (PON) 228 provides a power-up signal. The temperature sensor 226 can sense the temperature of the semiconductor device 200 and provide a temperature signal; the temperature of the semiconductor device 200 may affect some memory operations.
[0103] In one exemplary embodiment, the memory array 220 may be controlled by one or more configuration registers. In other words, the use of these configuration registers selectively configures the one or more memory arrays 220 into one or more matrix structures and / or matrix multiplication units (MMUs), which are described in more detail herein. In other words, the configuration registers may enable the memory cell architecture within the memory array to dynamically change, for example, its structure, operation, and functionality. These and other variations will be apparent to one of ordinary skill in the art given the content of this disclosure.
[0104] Figure 3 Provides more detailed side-by-side illustrations of memory array and matrix structure circuit configurations. Figure 3 Both the memory array and matrix structure circuit configurations use the same memory cell array, where each memory cell consists of a resistor element 302, which is coupled to a word line 304 and a bit line 306. In the first configuration 300, the memory array circuit is configured to operate as a row decoder 316, a column decoder 318, and a memory cell array 320. In the second configuration 350, the matrix structure circuit is configured to operate as a row driver 317, a matrix multiplication unit (MMU) 319, and an analog crossbar structure (matrix structure) 321. In an exemplary embodiment, a lookup table (LUT) and associated logic 315 can be used to store and configure different matrix multiplication unit coefficient values.
[0105] In an exemplary embodiment of the present disclosure, the memory array 320 is composed of resistive random access memory (ReRAM). ReRAM is a non-volatile memory that changes the resistance of a memory cell across a dielectric solid state material, which is sometimes referred to as a "memristor". Current ReRAM technology can be implemented in a two-dimensional (2D) layer or a three-dimensional (3D) layer stack; however, higher-order dimensions may be used in future iterations. The complementary metal oxide semiconductor (CMOS) compatibility of the vertical and horizontal ReRAM technology can enable logic (data processing) and memory (storage) to be integrated into a single chip. In other possible configurations, the vertical and horizontal ReRAM array can be formed in a one transistor / one resistor (1T1R) configuration and / or a configuration in which one transistor drives n resistive memory cells (1TNR).
[0106] A variety of inorganic and organic material systems can achieve thermal and / or ionic resistance transitions. In various embodiments, such systems can include: phase change chalcogenides (e.g., Ge2Sb2Te5, AgInSbTe, etc.); binary transition metal oxides (e.g., NiO, TiO2, etc.); perovskites (e.g., Sr(ZR)TrO3, PCMO, etc.); solid electrolytes (e.g., GeS, GeSe, SiO x, Cu2S, etc.); organic charge transfer complexes (e.g., Cu tetracyanoquinodimethane (TCNQ), etc.); organic charge acceptor systems (e.g., Al amino-dicyanoimidazole (AIDCN), etc.); and / or 2D (layered) insulating materials (e.g., hexagonal BN, etc.); and other possible systems for resistive switching.
[0107] In the illustrated embodiment, the resistor element 302 is a nonlinear passive two-terminal electrical component that can change its resistance based on the history of the current application (e.g., hysteresis or memory). In at least one exemplary embodiment, the resistor element 302 can form or destroy a conductive filament in response to applying currents of different polarities to a first terminal (connected to the word line 304) and a second terminal (connected to the bit line 306). The presence or absence of a conductive filament between the two terminals changes the conductance between the terminals. Although the present operation is given in the context of a resistor element, it will be readily understood by those of ordinary skill in the relevant art that the principles described herein can be implemented in any circuit characterized by a variable impedance (e.g., resistance and / or reactance). Variable impedance can be implemented by various linear and / or nonlinear elements (e.g., resistors, capacitors, inductors, diodes, transistors, thyristors, etc.).
[0108] For purposes of illustration, the operation of the memory array 320 in the first configuration 300 is briefly summarized. During operation in the first configuration, memory "writes" may be achieved by applying current to memory cells corresponding to rows and columns of the memory array. The row decoder 316 may selectively drive individual row terminals to select a particular row of the memory array circuit 320. The column decoder 318 may selectively sense / drive individual column terminals to "read" and / or "write" to the corresponding memory cell (e.g., a cell uniquely identified by the selected row and column). Figure 3 , emphasized by thicker line widths and blackened cell elements). As described above, the application of current causes the formation (or destruction) of conductive filaments within the dielectric solid state material. In such a case, the low resistance state (on state) is used to represent a logical "1", while the high resistance state (off state) is used to represent a logical "0". To switch a ReRAM cell, a first current of a specific polarity, amplitude, and duration is applied to the dielectric solid state material. Subsequently, a memory "read" can be achieved by applying a second current to the resistive element and sensing whether the resistive element is in the on state or the off state based on the corresponding impedance. The memory read may or may not be destructive (e.g., the second current may or may not be sufficient to form or destroy the conductive filament).
[0109] A person skilled in the relevant art will readily appreciate that the above discussion of the memory array 320 in the first configuration 300 is consistent with existing memory operations according to, for example, ReRAM memory technology. In contrast, the second configuration 350 uses memory cells as an analog crossbar structure (matrix structure) 321 to perform matrix multiplication operations. Figure 3 The exemplary implementation of corresponds to a 2x4 matrix multiplication unit (MMU), but other variants may be substituted with equivalent success. For example, matrices of arbitrarily large size (e.g., 3x3, 4x4, 8x8, etc.) may be implemented (depending on the precision achieved by the digital-to-analog conversion (DAC) 308 and analog-to-digital (ADC) 310 components).
[0110] In analog crossbar (matrix) 321 operation, each of the row terminals is simultaneously driven by an analog input signal, and each of the column terminals is simultaneously sensed for an analog output (which is the analog sum of the voltage potentials across the corresponding resistor elements for each row / column combination). Notably, in the second configuration 350, all row and column terminals associated with matrix multiplication are active (e.g., Figure 3 In other words, the ReRAM crossbar structure (matrix structure) 321 uses the matrix structure to perform "analog calculations" that calculate vector-matrix products (or scalar-matrix products, matrix-matrix products, etc.).
[0111] It is noteworthy that the concurrent vector-matrix product computations within the crossbar structure are atomic. Specifically, the analog computation of the vector-matrix product can be completed in a single access cycle. As previously mentioned, atomic operations are not subject to data race conditions. In addition, the vector-matrix product computation computes all rows and all columns of the matrix operation simultaneously; in other words, the vector-matrix product computation does not scale in complexity with the matrix dimension. Although manufacturing constraints (e.g., ADC / DAC granularity, manufacturing tolerances, etc.) can limit the amount of precision and complexity that a single matrix structure can produce, multiple matrix operations can be mathematically combined to provide much higher precision and complexity.
[0112] For example, in one exemplary embodiment of the present disclosure, DAC 308 converts the input to the analog domain for analog calculations, but ADC 310 can also convert the input back to the digital domain for subsequent digital and / or logic operations. In other words, arithmetic logic unit 312 is capable of performing complex digital operations on the output of matrix structure 321. This capability can be used in situations where the analog domain cannot perform the required calculations due to practical implementation limitations (e.g., manufacturing costs, etc.).
[0113] consider Figure 4, where a simple "FFT butterfly" calculation 400 can be performed by a 2x4 matrix structure. Although the conductance can be increased or decreased, the conductance cannot be made "negative". As a result, subtraction may need to be performed in the digital domain. The FFT butterfly operation (EQN.1) is described in the following multiplication of a matrix (M) and a vector (a):
[0114] EQN.1:
[0115] This simple FFT butterfly 400 of EQN.1 can be decomposed into two different matrices (EQN.2 and EQN.3) representing the positive and negative coefficients.
[0116] EQN.2:
[0117] EQN.3:
[0118] EQN.2 and EQN.3 can be implemented as analog calculations using a matrix structure circuit. Once calculated, the resulting analog value can be converted back to the digital domain via the aforementioned ADC. Existing ALU operations can be used to perform subtraction in the digital domain (EQN.4):
[0119] EQN.4:
[0120] In other words, Figure 4 As shown, the 2x2 matrix can be further subdivided into a 2x2 positive matrix and a 2x2 negative matrix. The ALU can add / subtract the results of the 2x2 positive matrix and the 2x2 negative matrix to generate a single 2x2 matrix. Those of ordinary skill in the relevant art will readily understand the numerous types and / or capabilities enabled by the ALU. For example, the ALU can provide arithmetic operations (e.g., addition, subtraction, carry addition, borrow subtraction, negation, increment, decrement, transfer, etc.), bitwise operations (e.g., AND, OR, XOR, complement), shift operations (e.g., arithmetic shift, logical shift, loop, carry loop, etc.) to implement, for example, multiple precision arithmetic, complex number operations and / or any MMU capabilities extended to any precision, size and / or complexity.
[0121] As used herein, the terms "digital" and / or "logic" in the context of computing refer to processing logic that uses quantized values (e.g., "0" and "1") to represent symbolic values (e.g., "on state", "off state"). In contrast, the term "analog" in the context of computing refers to processing logic that uses continuously variable aspects of physical signaling phenomena to perform computing, such as electrical, chemical, and / or mechanical quantities. Various embodiments of the present disclosure may represent analog input and / or output signals as continuous electrical signals. For example, a voltage potential may have different possible values (e.g., any value between a minimum voltage (0V) and a maximum voltage (1.8V), etc.). Combining analog computing with digital components may be performed using a digital-to-analog converter (DAC), an analog-to-digital converter (ADC), an arithmetic logic unit (ALU), and / or variable gain amplification / attenuation.
[0122] Refer to Figure 3 , in order to configure the memory cells into the vertical and horizontal structure (matrix structure) 321 of the second configuration 350, each resistive element can be written with a corresponding matrix coefficient value. Unlike the first configuration 300, the second configuration 350 can write different degrees of impedance (representing coefficient values) into each ReRAM cell using an amount of current with a polarity, amplitude, and duration selected to set a specific conductance. In other words, by forming / destroying conductive filaments with different conductivities, multiple different conductivity states can be established. For example, applying a first magnitude can result in a first conductance, applying a second amplitude can produce a second conductance, applying the first amplitude for a longer duration can produce a third conductance, etc. Any permutation of the aforementioned write parameters can be replaced with equivalent success. More directly, instead of using two (2) resistance states (on state, off state) to represent two (2) digital states (logic "1", logic "0"), the varying conductance can use multiple states (e.g., three (3), four (4), eight (8), etc.) to represent a continuous range of values and / or a range of values (e.g., [0, 0.33, 0.66, 1], [0, 0.25, 0.50, 0.75, 1], [0, 0.125, 0.250, ..., 1], etc.).
[0123] In one embodiment of the present disclosure, the matrix coefficient values are stored in advance in a lookup table (LUT) and configured by the associated control logic 315. In the initial configuration stage, the matrix structure 321 is written with the matrix coefficient values from the LUT by the control logic 315. It will be readily appreciated by those of ordinary skill in the relevant art that certain memory technologies may also implement write-once-and-use-many operations. For example, even though the formation (or destruction) of the conductive filaments of a ReRAM cell may require a specific duration, amplitude, polarity, and / or direction of the current; subsequent uses of the memory cell may be repeated multiple times (as long as the conductive filaments are not substantially formed or destroyed during the service life). In other words, subsequent uses of the same matrix structure 321 construction may be used to pay for the initial construction time.
[0124] In addition, some memory technologies (such as ReRAM) are non-volatile. Therefore, once the matrix structure circuit is programmed, it can enter a low-power state (or even power off) when not in use to save power. In some cases, the non-volatility of the matrix structure can be used to further improve power consumption. Specifically, unlike the prior art that can reload matrix coefficient values from non-volatile memory for subsequent processing, the exemplary matrix structure can still store matrix coefficient values even when the memory device is powered off. In a subsequent wake-up, the matrix structure can be used directly.
[0125] In one exemplary embodiment, the matrix coefficient values may be derived based on the characteristics of the matrix operation. For example, the coefficients of a particular matrix operation may be derived in advance based on the "size" (or other structurally defined parameters) and stored in the LUT. As just two such examples, the Fast Fourier Transform (EQN.5) and the Discrete Cosine Transform (DCT) (EQN.6) are reproduced below:
[0126] EQN.5:
[0127] EQN.6:
[0128] Note that EQN.6 represents the DCT-II version of the DCT and is commonly used for video / image compression. As can be determined mathematically from the preceding equations, the matrix coefficient values (also called "twiddle factors") are determined according to the size of the transform. For example, once the size of the DCT matrix is known, the values of the matrix can be set a priori (using, for example, EQN.6).
[0129] In another exemplary embodiment, the matrix coefficient values may be stored in advance. For example, the coefficients of certain matrix multiplication operations may be known or defined by, for example, an application or user. For example, image processing calculations such as described in co-owned and co-pending U.S. patent application Ser. No. 16 / 002,644, filed on Jun. 7, 2018, entitled “AN IMAGE PROCESSOR FORMED IN AN ARRAY OF MEMORY CELLS,” previously incorporated above, may define various different matrix coefficient values to implement, for example, defect correction, color interpolation, white balance, color adjustment, gamma brightness, contrast adjustment, color conversion, downsampling, and / or other image signal processing operations.
[0130] In another example, the coefficients of certain matrix multiplication operations may be determined or defined by, for example, user considerations, environmental considerations, other devices, and / or other network entities. For example, wireless devices often experience different multipath effects that may interfere with operation. Various embodiments of the present disclosure determine multipath effects and correct them using matrix multiplication. In some cases, the wireless device may calculate each of the different channel effects independently based on the degradation of known signaling. The difference between the expected reference channel signal and the actual reference channel signal can be used to determine the noise effects it experiences (e.g., attenuation, reflection, scattering, and / or other noise effects over a particular frequency range). In other embodiments, the wireless device may be instructed to use a predetermined "codebook" of beamforming configurations. The codebook of beamforming coefficients may be less precise, but may be preferred for other reasons (e.g., speed, simplicity, etc.).
[0131] As mentioned above, the matrix coefficient values are stored in advance in a lookup table (LUT) and configured by the associated control logic 315. In an exemplary embodiment, the matrix structure can be configured by dedicated hardware logic. This internal hardware logic may not be limited by the processor word size; therefore, the matrix coefficient values of any dimension may be simultaneously configurable (e.g., 4x4, 8x8, 16x16, etc.). Although the present disclosure is presented in the context of the internal control logic 315, the external implementation may be replaced with an equivalent success. For example, in other embodiments, the logic includes an internal memory embedded processor (PIM), which may set the matrix coefficient values based on the LUT values in a series of reads and writes. In other instances, for example, an external processor may perform the LUT and / or logic functions.
[0132] The following references Figures 5A to 5B 6A-6B describe memory arrays configured to perform specific matrix multiplication operations.
[0133] In various embodiments, aspects of the present disclosure may be applied to video / image compression or other operations, whether lossy or not. Discrete cosine transform (DCT) is used by various image / video processing and compression techniques (including JPEG) to represent the pixel values of an image (e.g., Y, Cb, or Cr values) as the sum of different cosine functions. For example, an image can be decomposed into 8x8 pixel blocks, where each of the 64 pixels in the block is represented by the values of brightness (Y) and chrominance (Cb, Cr). DCT can be applied to each 8x8 block to obtain a matrix D of the DCT frequency coefficients of the pixel block. This DCT frequency coefficient matrix is then used to perform the actual compression of the image block. Note that DCT can be similarly applied to pixel blocks of other sizes (e.g., 16x16 pixel blocks).
[0134] The DCT of an NxN pixel block can be performed using a matrix multiplication operation D=TMT', where T represents the DCT transform matrix (whose value can be calculated using EQN.6), T' represents the transpose of the T matrix, M represents a matrix of pixel values (e.g., Y values, Cb values, R values, etc.), and D represents a matrix of DCT frequency coefficients for the pixel block. This operation can be decomposed into two separate matrix-matrix multiplication stages / operations: a first operation (EQN.7) and a second operation (EQN.8).
[0135] EQN.7: B = TM
[0136] EQN.8: D = BT'
[0137] The individual values of the matrices in the first and second matrix-matrix multiplication operations are explicitly written below.
[0138]
[0139] Each matrix-matrix multiplication operation can be performed using a separate matrix-vector multiplication operation, and there are various ways to deconstruct the two matrix-matrix multiplication operations of EQN.7 and EQN.8 into matrix-vector operations, as will be understood by one of ordinary skill in the art given this disclosure. And as previously described, the matrix-vector multiplication operations can be performed using (one or more) matrix structures and (one or more) MMUs.
[0140] In one exemplary embodiment, individual columns of the B matrix may be calculated using EQN.7A.
[0141] EQN.7A:b cn =T·m cn
[0142] where vector b cn represents column n of the B matrix, and vector m cn Represents column n of the M matrix.
[0143] For example, we can use the matrix-vector multiplication operation b c0 =T·m c0 To calculate the first column of the B matrix (b 00 , b 10 , b 20 …b n0 ), as shown below.
[0144]
[0145] By executing b n times cn =T·m cn Matrix-vector operations (for each of the n columns of B) can obtain the entire B matrix. Similarly, to obtain D = BT', the rows of the D matrix can be calculated using EQN.8A:
[0146] EQN.8A:d rn =T·b rn
[0147] where the vector d rn represents row n of the D matrix, and vector b rn Represents row n of the B matrix.
[0148] For example, the first row of the D matrix (d 00 d 01 d 02 …d 0n ) can be calculated using the matrix-vector multiplication shown below. Note that the matrix used in this operation is T, not T'.
[0149]
[0150] Figure 5A and 5B An exemplary embodiment of a matrix structure and MMU circuit 500 is shown, which can be used to perform DCT on an NxN matrix M using the matrix-vector operations described above in EQN.7A and 8A. Note that for simplicity, Figure 5A The matrix structure 521 is shown as a 2x4 cell array. However, the matrix structure 521 represents an array of size Nx2N, wherein the first NxN matrix structure portion 521A can be configured with positive values of the matrix T, and the second NxN matrix structure portion 521B can be configured with negative values of the matrix T.
[0151] In one embodiment, the matrix structure 521 is an 8x16 cell array, the first matrix structure portion 521A is configured with positive values of the 8x8T matrix, and the second matrix structure 521B is configured with negative values of the 8x8T matrix. The 8x16 matrix structure 521 can be used to perform a DCT operation on an 8x8 matrix M representing pixel values of an 8x8 pixel block.
[0152] In another embodiment, the matrix structure 521 is a 16x32 cell array, programmed with positive and negative values of a 16x16 T matrix, and configured to perform a DCT operation on a 16x16 matrix M.
[0153] In other embodiments, the size of the matrix structure 521 can be appropriately scaled up or down to perform DCT operations on matrices M representing pixel blocks of various other sizes. As previously described, the matrix multiplication operations of the present disclosure can be performed on very large matrices because they do not face the time exponential scaling problem of traditional methods. Therefore, according to aspects of the present disclosure, DCT operations based on the matrix structure can be performed on very large pixel blocks (e.g., 64x64 pixels) in a reasonable amount of time.
[0154] exist Figure 5A In the system 500, the matrix structure 521 (comprising a first part 521A and a second part 521B) is programmed with positive and negative values of the DCT transformation matrix T using control logic 515. The values of the T matrix depend on the size of the matrix and can be calculated using EQN.6 as needed, or pre-stored in a lookup table (LUT). In one embodiment, the values of T for commonly used matrix sizes (e.g., 8x8 and 16x16) are stored in the LUT. The values of the matrix M represent the values of individual pixels of the pixel block (e.g., brightness values) and can be stored in the memory array 522.
[0155] To perform the first stage of the DCT operation (EQN.7 or 7A), the vectors m representing the individual columns of the matrix M are cn The analog input signals may be extracted from the memory array 522 and sequentially fed to the row driver 517 by the control logic 515 and then converted into analog input signals by the DAC 508 of the row driver 517. These converted analog input signals are then used to drive all row terminals of the matrix structure 521 simultaneously. The matrix structure results are output to the MMU 519, which converts the results into digital values through its ADC 510. The digital values are then used by the arithmetic logic unit (ALU) 512 of the MMU to obtain the individual b cn A vector of values (from EQN.7A) representing the columns of the B matrix. If necessary, each b cn The values of the vector may be temporarily stored in the memory array 522 until the entire B matrix is obtained.
[0156] Once the entire B matrix has been calculated, the second stage of the DCT operation (EQN.8 or 8A) can be performed using the same matrix structure 521 (configured with T matrix values). Each vector b representing an individual column of the matrix B rn The control logic 515 feeds back to the row driver 517, is converted by the DAC 508, and is used to simultaneously drive all row inputs of the matrix structure 521. The MMU 519 (including the ADC 510 and the ALU 512) is then used to obtain d rn vector, the d rn The vector is stored in the memory array 522 until all values of the D matrix have been calculated. Therefore, Figure 5A The system 500 can be used to calculate DCT frequency matrix coefficients of an NxN matrix M using a single Nx2N matrix structure 521 configured with values of a DCT transform matrix T, by first running a series of matrix-vector operations through the matrix structure 521 and the MMU 519 to obtain a first matrix-matrix product B, and then running a series of matrix-vector operations through the matrix structure 521 and the MMU 519 to obtain a second matrix-matrix product D.
[0157] like Figure 5B As shown, another system 550 can be used to perform matrix-matrix multiplication operations by running several matrix-vector multiplication operations in parallel using two or more matrix structures 521. In one embodiment, Figure 5B The system 550 includes multiple matrix structures 521 connected to its own row driver 517 and MMU 519. In one variation, the multiple matrix structures 521 include, for example, a vertical stack of vertical and horizontal matrix arrays, each vertical and horizontal matrix array is configured as a matrix structure 521. In another variation, the multiple matrix structures can be implemented as matrix arrays arranged on a single level, or even a common semiconductor die.
[0158] It is understandable, although Figure 5B Each matrix structure 521 is connected to its own MMU 519, but multiple matrix structures 521 may be connected to a single MMU 519. For example, a single MMU with sufficient capacity may be used to process the output of each matrix structure 521 in parallel. In addition, a common row driver device 517 may be used (although discretely composed row driver circuits are used to independently drive each matrix structure 521).
[0159] In one embodiment, using memory structure system 550, the entire matrix-matrix multiplication stage of EQN.7 can be executed in the time it takes to drive one matrix structure 521 once, because all n matrix-vector operations of EQN.7A can be executed simultaneously. In this embodiment of the device, control logic 515 is configured to obtain n individual vectors m cn And feed the vector to n individual row drivers simultaneously. Each individual row driver uses its m cn vectors to simultaneously drive the row terminals of their respective matrix structures 521. The matrix structures 521 and their respective MMUs 519 transmit their individual vectors b cn The values of are output to the memory array 522. Therefore, the individual columns of the matrix B are all calculated in parallel and stored in the memory array 522 at the same time. Similarly, in the second stage of the DCT operation, all rows of the matrix D can be calculated by using Figure 5B The multi-matrix structure system 550 performs the matrix-vector multiplication of EQN.8A in parallel to obtain.
[0160] EQNS.7A and 8A show a way to deconstruct the two matrix-matrix multiplications required for the DCT operation. However, the DCT matrix-matrix multiplication operations (EQNS.7 and 8) can be deconstructed using matrix-vector multiplication operations other than EQNS.7A and 8A, as recognized by one of ordinary skill given this disclosure.
[0161] In one exemplary embodiment, the values of the matrix B may be calculated using the following formula:
[0162] EQN.7B:b rn =M'·t rn ,
[0163] where vector b rn Represents row n of the B matrix, vector t rn represents row n of the T matrix, and M' represents the transpose of the M matrix. For example, the first row (b 00 , b 01 , b 02 …b 0n ) can be calculated using the matrix-vector multiplication shown below.
[0164]
[0165] Fig. 6A and 6B Another embodiment of a matrix structure and MMU circuit is shown, which can be used to perform DCT on an NxN matrix M using the matrix-vector operations described in EQN.7B and 8A. In one configuration, during the first stage of the DCT operation, the matrix-vector multiplication of EQN.7B can be used Fig. 6A 600 is implemented as shown in the first part (left) of the circuit 600. The first matrix structure 621 is configured by the control logic 615 to hold the values of the M' matrix (divided into a positive M' matrix 621A and a negative M' matrix 621B). The values of the M' matrix correspond to the pixel values of a particular image block, are unknown or pre-calculated, and are not stored in a lookup table (LUT). Instead, the values of the M' matrix can be stored in a memory array 622 of the device. The control logic 615 can access the memory array 622 and use the M' values stored in the memory array 622 to configure the first matrix structure 621.
[0166] After configuring the first matrix structure 621, the control logic 615 provides the individual vectors t to the first row driver 617. rn (For example, the vector t r0 ,t r1 , etc.), which converts the vector value from digital to analog (using DAC 608) and uses the analog signal to simultaneously drive the row terminals of the first matrix structure 621. The vector value t rn It depends on the size of the matrix T and can be stored in a lookup table (LUT) in advance. The first matrix structure 621 and the first MMU 619 are used to sequentially obtain all b rn Vector (representing the rows of matrix B).
[0167] In one variation, b rn The values of the vector may be temporarily stored in the memory array 622 and then provided to the second row driver from the memory array 622. In another variation, b rn The vector values are fed directly from the first MMU 619 into the second row driver 637. The second stage (EQN.8) of the DCT operation may be implemented using the second (right) portion of the memory system 600, including the second row driver 637, the second matrix structure 641 and the second MMU 639.
[0168] During the second stage of the DCT operation, each vector b rn The vectors are fed to the second row driver 637 via the control logic 615, converted by the DAC 628, and used to simultaneously drive all row inputs of the second matrix structure 641. The individual vector outputs of the first operation (EQN.7B) are used as individual vector inputs of the second operation (EQN.8A). In other words, unlike Figure 5A and 5B In the embodiment of FIG. 6 , it is not necessary to accumulate the values of the entire B matrix in the memory array 622 before performing the second operation. The second matrix structure 641 and the second MMU 639 use b rn vector to obtain d rn vector and convert the d rnThe vector is stored in memory array 622 until all values of the D matrix have been calculated. Therefore, Fig. 6A The system 600 may be used to compute the DCT of an NxN matrix M using two consecutive Nx2N matrix structures 621 and 641 without accumulating an intermediate product B between the first and second operations of the DCT.
[0169] Figure 6B Another embodiment of the apparatus is shown, wherein the system 650 is configured such that all matrix-vector multiplication operations of EQN.7B can be performed in parallel, and all matrix-vector multiplication operations of EQN.8A can be performed in parallel. All rows of the matrix B can be calculated in parallel using multiple (e.g., stacked or common level / die) first row drivers / first matrix structures / first MMU circuits, such as Figure 6B Then b rn The vectors are input to the corresponding second row drivers / second matrix structures / second MMU circuits so that all rows of the D matrix are calculated in parallel using EQN.8A. All individual rows of the D matrix are obtained from the second MMU 639 and stored simultaneously in the memory array 622. Therefore, the entire two-stage DCT operation can be performed in approximately twice the time it takes to pass through the matrix structure / MMU circuit.
[0170] Fig. 7A is a logical block diagram of an exemplary implementation of a processor-memory architecture 700 according to the various principles described herein. Fig. 7A As shown, processor 702 is coupled to memory 704 ; the memory includes a look-up table (LUT) 706 , control logic 708 , a matrix structure and corresponding matrix multiplication unit (MMU) 710 , and a memory array 712 .
[0171] In one embodiment, LUT 706 stores a plurality of matrix value coefficients, dimensions and / or other parameters associated with different matrix operations. In one exemplary embodiment, LUT 706 stores a plurality of discrete cosine transform (DCT) coefficients (e.g., "twiddle factors") associated with different DCT dimensions.
[0172] In another exemplary embodiment, the LUT 706 stores a plurality of Fast Fourier Transform (FFT) "twiddle factors," where various subsets of the twiddle factors are associated with different FFT dimensions. For example, a LUT 706 storing twiddle factors for a 64-point FFT has 64 coefficient values, which includes all 32 coefficients used in a 32-point FFT, all 16 coefficients used in a 16-point FFT, and so on.
[0173] In other embodiments, the LUT 706 stores a plurality of different matrix coefficient values for image signal processing (ISP), such as defect correction, color interpolation, white balance, color adjustment, gamma brightness, contrast adjustment, color conversion, downsampling, and / or other image signal processing operations. In yet another embodiment of the LUT 706, the LUT 706 may contain various channel matrix codebooks, which may be predefined and / or empirically determined based on radio channel measurements.
[0174] In one embodiment, the control logic 708 controls the operation of the matrix structure and the MMU 710 based on instructions received from the processor 702. In an exemplary embodiment, the control logic 708 can form / destroy conductive filaments with different conductivity within each memory cell of the matrix structure according to the aforementioned matrix dimensions and / or matrix value coefficients provided by the LUT 706. In addition, the control logic 708 can configure the corresponding MMU to perform any additional arithmetic and / or logical operations of the matrix structure. In addition, the control logic 708 can select one or more digital vectors to drive the matrix structure and select one or more digital vectors to store the logic output of the MMU.
[0175] In processing technology, an "instruction" typically contains different types of "instruction syllables": for example, an opcode, operands, and / or other associated data structures (eg, registers, scalars, vectors).
[0176] As used herein, the term "operation code" (opcode) refers to an instruction that can be interpreted by processor logic, memory logic, or other logic circuits to implement an operation. More directly, an opcode identifies an operation to be performed on one or more operands (inputs) to generate one or more results (outputs). Both operands and results can be embodied as data structures. Common examples of data structures include, but are not limited to, scalars, vectors, arrays, lists, records, unions, objects, graphs, trees, and / or any number of other forms of data. Some data structures may contain reference data (data that "points" to other data) in whole or in part. Common examples of reference data structures include, for example, pointers, indexes, and / or descriptors.
[0177] In one exemplary embodiment, the opcode may identify one or more of: a matrix operation, a dimension of a matrix operation, and / or a row and / or column of a memory cell.In one such variant, an operand is an encoded identifier specifying one or more numeric vectors to be operated on.
[0178] For example, an instruction to perform a DCT operation on an input digital 8x8 matrix and store the result in an output digital matrix may include an opcode and operands: DCT8X8($input, $output), where: DCT8X8 identifies the size and nature of the operation, $input identifies the input digital matrix base address, and $output identifies the output digital matrix base address. In another such example, the DCT operation may be divided into several different atomic operations, such as (i) DCT8X8($address), which converts the memory array at $address into an 8x8 (64-point) matrix structure, (ii) ROWS8($input, $output), which decomposes the 8x8 digital matrix at $input into 8 separate digital vectors ($output), and (iii) MULT($address, $input, $output), which stores the vector-matrix product at $input and the matrix structure at $address to $output.
[0179] In one embodiment, the memory-processor architecture 700 can be used to perform DCT video processing on consecutive video frames by computing one video frame (e.g., an I frame) in the memory 704, at least partially powering off the processor 702 (or assigning it to other tasks), and then waking up the processor to compute the next frame (e.g., a B frame). Since each DCT matrix operation is performed in memory and the video frame data always resides in memory, the memory bandwidth load is greatly reduced.
[0180] Fig. 7A An instruction interface that is functionally independent of and distinct from an input / output (I / O) memory interface is shown. In one such embodiment, the instruction interface may be physically distinct (e.g., having different dies and / or connections). In other embodiments, the instruction interface may be multiplexed with the I / O memory interface (e.g., sharing the same control signaling, and address and / or data bus, but in a different communication mode). In other embodiments, the instruction interface may be virtually accessible through the I / O memory interface (e.g., as a register located within an address space addressable by the I / O interface). Other variants may be substituted by one of ordinary skill in the art given the contents of this disclosure.
[0181] In one embodiment, the matrix structure and MMU 710 are tightly coupled to the memory array 712 to read and write digital vectors (operands). In an exemplary embodiment, the operands are identified for dedicated data transfer hardware (e.g., direct memory access (DMA)) to and from the matrix structure and MMU 710. In an exemplary variant, the digital vector of data can be of any dimension and is not limited by the processor word size. For example, the operand can specify an N-bit operand (e.g., 2, 4, 8, 16 ... etc.). In other embodiments, the DMA logic 708 can read / write the matrix structure 710 using an existing memory row / column bus interface. In other embodiments, the DMA logic 708 can read / write the matrix structure 710 using existing address / data and read / write control signaling within the internal memory interface.
[0182] Figure 7B A set of exemplary matrix operations 750 is provided in Fig. 7A 700. As shown therein, processor 702 sends a specified operation code (e.g., a matrix M) to interface 707. x,y representation) and operands (e.g., digital vectors a, b) are written into memory 704.
[0183] The control logic 708 determines whether the matrix structure and / or the matrix multiplication unit (MMU) should be configured / reconfigured. For example, a portion of the memory array is converted into one or more matrix structures and used by the matrix MMU. x,y The defined correlation matrix coefficient values are weighted. The digital-to-analog (DAC) row drivers and analog-to-digital (ADC) sense amplifiers associated with the matrix structure may need to be adjusted for dynamic range and / or amplification. Additionally, one or more MMU ALU components may be coupled to one or more matrix structures.
[0184] When the matrix structure and / or the matrix multiplication unit (MMU) are appropriately configured, the input operand a is read by the digital-to-analog (DAC) and applied to the matrix structure M x,y The simulation result can also be converted using analog-to-digital (ADC) conversion for subsequent logic operations in the MMU ALU. The output is written to the output operand b.
[0185] Figure 7C A set of exemplary matrix operations 760 is provided in Fig. 7A An alternative logic flow diagram in the context of the exemplary embodiment 700 described in Figure 7B In contrast to the flowchart, Figure 7CThe system uses explicit instructions to convert the memory array into a matrix structure. Providing a further degree of atomicity in instruction behavior can achieve various related benefits, including, for example, pipelining and / or reduced instruction set complexity.
[0186] More directly, when the matrix structure contains the appropriate matrix-valued coefficients M x,y , matrix operations can be efficiently repeated. For example, image processing calculations such as described in co-owned and co-pending U.S. Patent Application No. 16 / 002,644, filed on June 7, 2018, entitled "AN IMAGE PROCESSOR FORMED IN AN ARRAY OF MEMORY CELLS," previously incorporated above, can be configured with multiple matrix structures and MMU processing elements to pipeline, for example, defect correction, color interpolation, white balance, color adjustment, gamma brightness, contrast adjustment, color conversion, downsampling, and / or other image signal processing operations. Each pipeline stage can be configured once and repeatedly used for each pixel (or group of pixels) of an image. For example, a white balance pipeline stage can operate on each pixel of data using the same matrix structure with matrix coefficient values set for white balance; a color adjustment pipeline stage can operate on each pixel of data using the same matrix structure with matrix coefficient values set for color adjustment, etc. In another such example, a DCT of an 8x8 matrix may be processed, for example, in two stages, each stage containing, for example, eight atomic MMU computations using the same DCT transform matrix T (as described above).
[0187] In addition, those of ordinary skill in the relevant art will further appreciate that some matrix structures may have additional versatility and / or uses beyond their initial construction. For example, as previously described, a 64-point FFT has 64 coefficient values, which contain all 32 coefficients used in a 32-point FFT. Therefore, a matrix structure configured for 64-point operations can be reused for 32-point operations, where the 32-point input operand a is appropriately applied on the appropriate rows of the 64-point FFT matrix structure. Similarly, the FFT rotation factors are a superset of the discrete cosine transform (DCT) rotation factors; therefore, the FFT matrix structure (applying the input operand a appropriately) can also be used to calculate the DCT result.
[0188] Other permutations and / or variations of the foregoing examples will be apparent to those of ordinary skill in the relevant art(s), given the context of this disclosure.
[0189] method-
[0190] Reference now Figure 8, a logic flow diagram of an exemplary method 600 of converting a memory array into a matrix structure and performing matrix operations therein is shown.
[0191] At step 802 of method 800, a memory device receives one or more instructions. In one embodiment, the memory device receives the instructions from a processor. In one such variant, the processor is an application processor (AP) commonly used in consumer electronics. In other such variants, the processor is a baseband processor (BB) commonly used in wireless devices.
[0192] In short, a so-called "application processor" is a processor configured to execute an operating system (OS) and one or more applications, firmware, and / or software. The term "operating system" refers to software that controls and manages access to hardware. The OS typically supports processing functions such as task scheduling, application execution, input and output management, memory management, security, and peripheral device access.
[0193] The so-called "baseband processor" is a processor configured to communicate with a wireless network through a communication protocol stack. The term "communication protocol stack" refers to the software and hardware components that control and manage access to wireless network resources. The communication protocol stack generally includes, but is not limited to: physical layer protocols, data link layer protocols, media access control protocols, network and / or transmission protocols, etc.
[0194] Other peripheral devices and / or coprocessor configurations can be similarly replaced with equivalent success. For example, a server device typically includes multiple processors that share common memory resources. Similarly, many public device architectures pair general-purpose processors with dedicated coprocessors and shared memory resources (such as graphics engines or digital signal processors (DSPs)). Common examples of such processors include, but are not limited to: graphics processing units (GPUs), video processing units (VPUs), tensor processing units (TPUs), neural network processing units (NPUs), digital signal processors (DSPs), image signal processors (ISPs). In other embodiments, the memory device receives instructions from an application-specific integrated circuit (ASIC) or other form of processing logic (e.g., field programmable gate arrays (FPGAs), programmable logic devices (PLDs), camera sensors, audio / video processors, and / or media codecs (e.g., images, video, audio, and / or any combination thereof)).
[0195] In one exemplary embodiment, the memory device is a resistive random access memory (ReRAM) arranged in a "horizontal and vertical" row-column configuration. Although the various embodiments described herein assume a specific memory technology and a specific memory structure, a person of ordinary skill in the relevant art given the contents of the present disclosure will readily understand that the principles described herein can be broadly extended to other technologies and / or structures. For example, certain programmable logic structures (e.g., commonly used in field programmable gate arrays (FPGAs) and programmable logic devices (PLDs)) can have similar characteristics to memories in terms of capabilities and topology. Similarly, certain processors and / or other memory technologies can change resistance, capacitance, and / or inductance; in this case, the varying impedance characteristics can be used to perform analog calculations. In addition, while the "horizontal and vertical" based construction provides a physical structure that is well adapted to a two-dimensional (2D) matrix structure, other topological structures can also be well adapted to higher-order mathematical operations (e.g., matrix-matrix products via three-dimensional (3D) memory stacks, etc.).
[0196] In one exemplary embodiment, the memory device further includes a controller. The controller receives one or more instructions and parses each instruction into one or more instruction components (also commonly referred to as "instruction syllables"). In one exemplary embodiment, an instruction syllable includes at least one opcode and one or more operands. For example, an instruction may be parsed into an opcode, a first source operand, and a destination operand. Other common examples of instruction components may include, but are not limited to, a second source operand (for binary operations), a shift amount, an absolute / relative address, a register (or other reference to a data structure), an immediate data structure (i.e., a data structure provided within the instruction itself), a dependent function, and / or a branch / link value (e.g., executed depending on whether the instruction completed or failed).
[0197] In one embodiment, each received instruction corresponds to an atomic memory controller operation. As used herein, an "atomic" instruction is an instruction that is completed within a single access cycle. In contrast, a "non-atomic" instruction is an instruction that is completed or not completed within a single access cycle. Even though a non-atomic instruction may be completed in a single cycle, the instruction must be considered non-atomic to prevent data race conditions. In the case where data accessed by a processor instruction (read or write) can be accessed by another processor instruction before the first processor instruction has a chance to complete, a race condition occurs; a race condition may unpredictably cause a data read / write error. In other words, an atomic instruction ensures that data cannot be observed in an incomplete state.
[0198] In one exemplary embodiment, an atomic instruction may identify a portion of a memory array that is to be converted into a matrix structure. In some cases, an atomic instruction may identify characteristic attributes of a matrix structure. For example, an atomic instruction may identify the portion of a memory array based on, for example, a location within the memory array (e.g., by offset, row, column), a size (number of rows, number of columns, and / or other dimensional parameters), a granularity (e.g., precision and / or sensitivity). Notably, atomic instructions may provide very fine control over memory device operations; this may be desirable in situations where memory device operations may be optimized in light of various application-specific considerations.
[0199] In other embodiments, non-atomic instructions may specify portions of a memory array that will be converted into a matrix structure. For example, non-atomic instructions may specify various requirements and / or constraints for a matrix structure. The memory controller may allocate resources internally to accommodate the requirements and / or constraints. In some cases, the memory controller may prioritize and / or deprioritize instructions based on current memory usage, memory resources, controller bandwidth, and / or other considerations. Such embodiments may be particularly useful in situations where memory device management is unnecessary or would otherwise burden the processor.
[0200] In one embodiment, the instruction specifies a matrix operation. In one such variant, the matrix operation can be a vector-matrix product. In another variant, the matrix operation can be a matrix-matrix product. Given the contents of the present disclosure, one of ordinary skill in the relevant art can also replace other variants. Such variants can include, for example, scalar-matrix products, high-order matrix products and / or include, for example, other transformations of linear shifts, rotations, reflections, and translations.
[0201] As used herein, the terms "transformation" and the like refer to a mathematical operation that converts an input from a first domain to a second domain. A transformation may be "injective" (each element of the first domain has a unique element in the second domain), "surjective" (each element of the second domain has a unique element in the first domain), or "bijective" (a unique one-to-one mapping of elements from the first domain to the second domain).
[0202] More complex mathematically defined transforms that are often used in the computing arts include Fourier transforms (and their derivatives, such as discrete cosine transforms (DCTs)), Hilbert transforms, Laplace transforms, and Legendre transforms. In an exemplary embodiment of the present disclosure, matrix coefficient values for mathematically defined transforms may be calculated in advance and stored in a lookup table (LUT) or other data structure. For example, rotation factors for a fast Fourier transform (FFT) and / or DCT may be calculated and stored in a LUT. In other embodiments, matrix coefficient values for mathematically defined transforms may be calculated by a memory controller during a matrix structure conversion process (or preparation).
[0203] Other transformations may not be based on the mathematical definition itself, but may be defined based on, for example, an application, another device and / or a network entity. Such transformations may be generally useful for encryption, decryption, geometric modeling, mathematical modeling, neural networks, network management, and / or other graph-theory-based applications. For example, a wireless network may use a codebook of predetermined antenna weight matrices to signal the most commonly used beamforming configurations. In other instances, certain types of encryption may be agreed upon and / or negotiated between different encryption matrices. In such embodiments, the codebook or matrix coefficient values may be agreed upon in advance, exchanged out-of-band, exchanged in-band, or even arbitrarily determined or negotiated.
[0204] Given the content of the present disclosure, empirically determined transformations can also be replaced with equivalent success. For example, empirically derived transformations frequently used in the computing field include radio channel coding, image signal processing, and / or other mathematically modeled environmental effects. For example, a multipath radio environment can be characterized by measuring the impact of the channel on, for example, a reference signal. The resulting channel matrix can be used to constructively interfere with signal reception (e.g., to increase signal strength) while destructively interfering with interference (e.g., to reduce noise). Similarly, the overall color balance of an image with skewed tones can be evaluated and mathematically corrected. In some cases, an image can be intentionally skewed based on, for example, user input in order to give the image aesthetic "warmth."
[0205] Various embodiments of the present invention may implement "unary" operations within a memory device. Other embodiments may implement "binary" or even higher order "N-ary" matrix operations. As used herein, the terms "unary", "binary" and "N-ary" refer to operations that use one, two or N input data structures, respectively. In some embodiments, binary and / or N-ary operations may be subdivided into one or more unary matrix in-place operators. As used herein, an "in-place" operator refers to a matrix operation that stores or converts its result into its own state (e.g., its own matrix coefficient value). For example, a binary operation may be decomposed into two (2) unary operations; a first in-place unary operation is performed (the result is stored "in-place"). Thereafter, a second unary operation may be performed on the matrix structure to produce a binary result (e.g., a multiply-accumulate operation).
[0206] Other embodiments may serialize and / or parallelize matrix operations based on various considerations. For example, sequentially related operations may be performed in a "serial" pipeline. For example, image processing calculations such as described in co-owned and co-pending U.S. patent application Ser. No. 16 / 002,644, filed on June 7, 2018, entitled "AN IMAGE PROCESSOR FORMED IN AN ARRAY OF MEMORY CELLS," previously incorporated above, configure multiple matrix structures and MMU processing elements to pipeline such as defect correction, color interpolation, white balance, color adjustment, gamma brightness, contrast adjustment, color conversion, downsampling, etc. Pipeline processing can typically produce very high throughput data with minimal matrix structure resources. Conversely, unrelated operations can be performed "in parallel" with separate resources. For example, the first stage of a DCT operation performed on an 8x8 matrix can be processed with eight separate matrix structures, such as with respect to Figure 5B and 6B Highly parallelized operations can significantly reduce latency; however, overall memory structure resource utilization may be very high.
[0207] In one exemplary embodiment, instructions are received from a processor via a dedicated interface. The dedicated interface is particularly useful in the case where the matrix computing structure is treated as a coprocessor or hardware accelerator. It is worth noting that the dedicated interface does not require arbitration and can operate at very high speeds (in some cases, at the local processor speed). In other embodiments, instructions are received via a shared interface.
[0208] The shared interface may be multiplexed in time, resources (e.g., channels, channels, etc.), or otherwise with other concurrently active memory interface functionality. Common examples of other memory interface functionality include, but are not limited to: data input / output, memory configuration, processor-in-memory (PIM) communications, direct memory access, and / or any other form of blocking memory access. In some variants, the shared interface may include one or more queuing and / or pipelining mechanisms. For example, some memory technologies may implement a pipelined interface in order to maximize memory throughput.
[0209] In some embodiments, instructions can be received from any entity that can access the memory interface. For example, a camera coprocessor (image signal processor (ISP)) can communicate directly with the memory device to, for example, write captured data. In certain embodiments, the camera coprocessor can offload its processing tasks to the matrix structure of the memory device. For example, the ISP can accelerate / offload / parallelize, for example, color interpolation, white balance, color correction, color conversion, etc. In other instances, a baseband coprocessor (BB) can communicate directly with the memory device to, for example, read / write data for transactions through a network interface. The BB processor can offload, for example, FFT / IFFT, channel estimation, beamforming calculations, and / or any number of other networking tasks to the matrix structure of the memory device. Similarly, video and / or audio codecs typically utilize DCT / IDCT transforms and will benefit from matrix structure operations. Given the content of the present disclosure, a person of ordinary skill in the relevant art will readily recognize variations of the foregoing.
[0210] Various embodiments of the present invention may support queues of multiple instructions. In an exemplary embodiment, matrix operations may be queued together. For example, multiple vector-matrix multiplications may be queued together to implement matrix-matrix multiplication. For example, as described elsewhere in this disclosure, multiple matrix-vector multiplications may be used to efficiently perform the matrix-matrix multiplication operations required for DCT. Similarly, as previously described, higher-order transforms (e.g., FFT1024) may be implemented by queuing multiple iterations of lower-order constituent transforms (e.g., FFT512, etc.). In yet another example, ISP processing of an image may include multiple iterations in the iteration space (each iteration may be pre-queued). Given the contents of this disclosure, a person of ordinary skill in the relevant art may easily replace other queuing schemes with the same success.
[0211] In some cases, matrix operations can be cascaded together to achieve higher order matrix operations. For example, a high order FFT (e.g., 1024x1024) can be decomposed into multiple iterations of a low order FFT (e.g., four (4) iterations of a 512x512 FFT, sixteen (16) iterations of a 256x256 FFT, etc.). In other examples, an N-point DFT of arbitrary size (e.g., not a power of 2) can be achieved by cascading DFTs of other sizes. Other examples of cascaded and / or chained matrix transforms can be replaced with equivalent successes, the above is purely illustrative.
[0212] As previously mentioned, the non-volatile nature of ReRAM enables memory contents to be maintained even when the ReRAM is not powered. Therefore, certain variations of processor-memory architectures may enable one or more processors to independently power the memory. In some cases, the processor may power the memory when the processor is inactive (e.g., keeping the memory active when the processor is at low power). Independent power management of the memory may be particularly useful, for example, for performing matrix operations in the memory even when the processor is dormant. For example, the memory may receive multiple instructions to be executed; the processor may transition to sleep mode until the multiple instructions are completed. Other embodiments may use the non-volatile nature of ReRAM to maintain memory contents when the memory is powered off; for example, certain video and / or image processing calculations may be maintained within the ReRAM during inactivity.
[0213] In step 804 of method 800, the memory array (or a portion thereof) can be converted into a matrix structure based on an instruction. As used herein, the term "matrix structure" refers to a plurality of memory cells with configurable impedance, which generate output vectors and / or matrices when driven with input vectors. In one embodiment, the matrix structure can be associated with a portion of a memory map. In some such variants, the portion is configurable in terms of its size and / or position. For example, a configurable memory register can determine whether a library is configured as a memory or a matrix structure. In other variants, the matrix structure can be reused and / or even blocked memory interface operations. For example, a memory device can allow a memory interface to be based on GPIO (e.g., in one configuration, a pin of a memory interface can be selectively operated as ADDR / DATA during normal operation, or during matrix operations, such as as FFT16, etc.).
[0214] In one embodiment, the instruction identification is characterized by a matrix structure of coefficients defined in the structure. In an exemplary embodiment, the matrix structure contains coefficients for the matrix operations defined in the structure. For example, the matrix structure used for 8x8 DCT is an 8x8 matrix structure, which has been pre-filled with coefficients defined in the structure of DCT. In some variants, the matrix structure can be pre-filled with coefficients of specific symbols (positive, negative) or specific bases (most significant bit, least significant bit or middle bit).
[0215] As used herein, the term "structurally defined coefficients" refers to the fact that the coefficients of a matrix multiplication are defined by the matrix structure (e.g., the size of the matrix), rather than by the characteristics of the operation (e.g., multiplication by an operand). For example, a structurally defined matrix operation (e.g., 8x8, 16x16, 32x32, 64x64, 128x128, 256x256, etc.) may be identified by, for example, row and column designations. Although the foregoing discussion is presented in the context of full rank matrix operations, the deficient matrix operator may be successfully replaced with an equivalent. For example, a matrix operation may have asymmetric columns and / or rows (e.g., 8x16, 16x8, etc.). In fact, a plurality of vector-based operations may be viewed as rows with a single column or columns with a single row (e.g., 8x1, 1x8).
[0216] In some hybrid hardware / software embodiments, control logic (e.g., memory controller, processor, PIM, etc.) may determine whether there are resources to provide a matrix structure. In one such embodiment, matrix operations may be evaluated by a preprocessor to determine whether they should be processed in software or in a dedicated matrix structure. For example, if the use of existing memory and / or matrix structures consumes all memory device resources, it may be necessary to process matrix operations in software rather than through matrix structures. In this case, the instructions may become incomplete (resulting in traditional matrix operations being performed through processor instructions). In another such example, configuring a temporary matrix structure to process simple matrix operations may yield less returns, such that matrix operations should be processed in software.
[0217] Various considerations can be used when determining whether a matrix structure should be used. For example, memory management can allocate portions of a memory array for memory and / or a matrix structure. In some embodiments, portions of a memory array can be allocated statically. Static allocation is preferred for reducing memory management overhead and / or simplifying operational overhead (wear leveling, etc.). In other embodiments, portions of a memory array can be allocated dynamically. For example, wear leveling may be required to ensure that the performance of the memory is uniformly degraded (rather than losing high availability areas). Other variants can allocate different portions statically and / or dynamically; for example, subsets of memory and / or matrix structure portions can be allocated dynamically and / or statically.
[0218] In short, wear leveling of memory cells can be performed in any discrete amount of memory (e.g., memory banks, memory blocks, etc.). Similar techniques can be used to wear level a matrix structure; for example, in one variation, wear leveling a portion of a matrix structure may require the entire matrix structure to be moved as a whole (the vertical and horizontal structures cannot be moved in sections). Alternatively, wear leveling of a portion of a matrix structure can be performed by first decomposing the matrix structure into constituent matrix calculations and distributing the constituent matrix calculations to other locations. More directly, matrix structure wear leveling can indirectly benefit from "logical" matrix operations used in other matrix operations (e.g., decomposition, concatenation, parallelization, etc.). In particular, decomposing a matrix structure into its constituent matrix structures can achieve better wear leveling management, using only slightly more complex operations (e.g., additional steps of logical combination by the MMU).
[0219] In one exemplary embodiment, the conversion includes reconfiguring a row decoder to operate as a matrix structure driver that variably drives multiple rows of a memory array. In one variant, the row driver converts a digital value into an analog signal. In one variant, the digital-to-analog conversion includes changing the conductance associated with the memory cell according to the matrix coefficient value. In addition, the conversion may include reconfiguring a column decoder to perform analog decoding. In one variant, the column decoder is reconfigured to sense an analog signal corresponding to a column of variable conductance cells, which are driven by a corresponding row of variable signaling. The column decoder converts the analog signal into a digital value. Although the foregoing construction is presented in a specific row-column configuration, other embodiments may also be replaced with the same success. For example, the column driver may convert a digital value into an analog signal, and the row decoder may convert an analog signal into a digital value. In another such example, a three-dimensional (3D) row-column-depth memory may implement a 2D matrix with any permutation (e.g., row driver / column decoder, row driver / depth decoder, column driver / depth decoder, etc.) and / or a 3D matrix permutation (e.g., row driver / column decoder driver / depth decoder).
[0220] In one exemplary embodiment, the matrix coefficient values correspond to the structurally determined values. The structurally determined values can be based on the characteristics of the operation. For example, a discrete cosine transform (DCT) performed on an NxN matrix uses an NxN DCT matrix T, where the values of the DCT matrix are calculated as a function of N. In fact, various transforms are similar in this respect. For example, both discrete Fourier transform (DFT) and fast Fourier transform (FFT) use structurally defined coefficients.
[0221] In one exemplary embodiment, the matrix structure itself has structurally determined dimensions. The structurally determined dimensions may be based on the characteristics of the operation; for example, an ISP white balance process may use a 3x3 matrix (corresponding to different values of red (R), green (G), blue (B), luminance (Y), chrominance red (Cr), chrominance blue (Cb), etc.). In another such instance, the channel matrix estimation and / or beamforming codebook is typically defined based on the number of multiple-input multiple-output (MIMO) paths. For example, a 2x2 MIMO channel has a corresponding 2x2 channel matrix and corresponding 2x2 beamforming weights. Given the content of the present disclosure, a person of ordinary skill in the relevant art may substitute various other structurally defined values and / or dimensions that are useful for matrix operations.
[0222] Some variants can also subdivide matrix coefficient values to handle operations that may be impractical. In this case, the matrix structure can contain only a portion of the matrix coefficient values (only a portion of the matrix operation is performed). For example, extremely expensive manufacturing tolerance levels may be required to perform signed operations and / or higher-level cardinality calculations. Signed matrix operations can be divided into positive and negative matrix operations (which are later summed by the matrix multiplication unit (MMU) described elsewhere herein). Similarly, high-radix matrix operations can be divided into, for example, the most significant bit (MSB) part, the least significant bit (LSB) part and / or any intermediate bit (which can be shifted and summed by the aforementioned MMU). Given the contents of this disclosure, those of ordinary skill in the art will readily recognize other variants.
[0223] In one exemplary embodiment, the matrix coefficient values are determined in advance and stored in a lookup table for later reference. For example, matrix operations having structurally determined dimensions and structurally determined values may be stored in advance. As just one such example, an 8x8 DCT matrix T has structurally determined dimensions (8x8) and structurally determined values (e.g., calculated with EQN.6). A DCT8X8 instruction may result in the configuration of an 8x8 matrix structure that is pre-filled with corresponding DCT8X8 structurally determined values. As another such example, antenna beamforming coefficients are typically defined in advance within a codebook; a wireless network may identify corresponding indices within the codebook to configure antenna beamforming. For example, a MIMO codebook may identify possible configurations for a 4x4 MIMO system; during operation, a selected configuration may be retrieved from the codebook based on its index.
[0224] Although the foregoing examples are presented in the context of structurally defined dimensions and / or values, other embodiments may use dimensions and / or values defined based on one or more other system parameters. For example, low-power operations may require a smaller granularity. Similarly, as previously described, various processing considerations may favor (or oppose) performing matrix operations within a matrix structure. In addition, matrix operations may affect other memory considerations, including but not limited to: wear leveling, memory bandwidth, in-memory process bandwidth, power consumption, row and column and / or depth decoding complexity, etc. Given the content of the present disclosure, a person of ordinary skill in the relevant art may replace various other considerations, and the above content is merely illustrative.
[0225] At step 806 of method 800, one or more matrix multiplication units may be configured based on the instruction. As previously described, certain matrix structures may implement logic (mathematical identities) to handle a single stage of a matrix operation; however, multi-level matrix structures may be cascaded together to implement more complex matrix operations. In an exemplary embodiment, a first matrix is used to calculate positive products of matrix operations, and a second matrix is used to calculate negative products of matrix operations. The resulting positive and negative products may be compiled within the MMU to provide signed matrix multiplication. In an exemplary embodiment, a first matrix is used to calculate a first cardinality portion of a matrix operation, and a second matrix is used to calculate a second cardinality portion of a matrix operation. The resulting cardinality portion may be shifted and / or summed within the MMU to provide a larger cardinality product.
[0226] Simply put, logical matrix operations are distinguished from analog matrix operations. The exemplary matrix structure converts analog voltages or currents into digital values that are read by a matrix multiplication unit (MMU). Logical operations can manipulate digital values through mathematical properties (e.g., through matrix decomposition, etc.); analog voltages or currents cannot be manipulated in this way.
[0227] More generally, different logical operations can be performed with matrix groups. For example, a matrix can be decomposed or factored into one or more component matrices. Similarly, multiple component matrices can be gathered or combined into a single matrix. In addition, a matrix can be expanded in rows and / or columns to create a defective matrix of larger dimension (but of the same rank). This logic can be used to implement many high-order matrix operations. For example, as explained elsewhere in this disclosure, multiplying two matrices together can be decomposed into multiple vector-matrix multiplications. These vector-matrix multiplications can be further implemented as multiplication-accumulation logic in a matrix multiplication unit (MMU). In other words, even non-unary operations can also be processed as a series of segmented unary matrix operations. More generally, it will be readily understood by those of ordinary skill in the relevant art that any matrix operation that can be represented as a unary operation in whole or in part can greatly benefit from the various principles described herein.
[0228] Various embodiments of the present disclosure use a matrix multiplication unit (MMU) as glue logic between multiple component matrix structures. In addition, the MMU operation can be selectively switched to connect to various rows and / or columns. Not all matrix structures can be used at the same time; therefore, depending on the current processing and / or memory usage, the matrix structure can be selectively connected to the MMU. For example, a single MMU can be dynamically connected to different matrix structures.
[0229] In some embodiments, control logic (e.g., a memory controller, a processor, a PIM, etc.) can determine whether there are resources to provide MMU operations, such as in a column decoder or elsewhere. For example, the current MMU load can be evaluated by a preprocessor to determine whether the MMU load can be made high. It is worth noting that the MMU is primarily used for logical operations, so any processing entity with equivalent logical functions can assist the MMU's tasks. For example, a processor embedded in memory (PIM) can offload MMU operations. Similarly, the matrix structure results can be provided directly to the main processor (which can perform the logical operations in software).
[0230] More generally, various embodiments of the present disclosure contemplate sharing MMU logic between multiple different matrix structures. Sharing may be based on, for example, a time sharing scheme. For example, an MMU may be assigned to a first matrix structure during one time slot and to a second matrix structure during another time slot. In other words, unlike the physical structure of the matrix structure (which is statically assigned for the duration of the matrix operation), the MMU performs logical operations that may be scheduled, subdivided, allocated, reserved, and / or divided in any number of ways. More generally, various embodiments of the matrix structure are based on memory and non-volatility. As a result, the matrix structure may be pre-configured and read when needed; the non-volatile property ensures that the matrix structure retains content without requiring processing overhead even if, for example, the memory device is powered off.
[0231] If both the matrix structure and the corresponding matrix multiplication unit (MMU) are successfully converted and configured, then at step 808 of method 800, the matrix structure is driven based on the instruction, and the logic result is calculated using one or more matrix multiplication units at step 810. In one embodiment, one or more operands are converted into electrical signals for analog calculations by the matrix structure. The result of the analog calculation is to drive the electrical signal through the matrix structure elements; for example, the voltage drop is a function of the coefficients of the matrix structure. The analog calculation result is sensed and converted back to a digital domain signal. Thereafter, the one or more digital domain values are processed using one or more matrix multiplication units (MMUs) to create a logic result.
[0232] Fig. 8A A logic flow diagram of an exemplary method 850 is shown, which implements Figure 8Steps 808 and 810 are used to perform DCT matrix operations. In one embodiment, the Figure 5A The exemplary method 850 may be implemented using the memory structure described in , although this is merely exemplary.
[0233] Return to reference Figure 8 In the context of DCT, at step 802, a memory device receives one or more instructions from, for example, a processor. The instructions may specify a DCT operation to be performed on a matrix M of size NxN.
[0234] At step 804 of method 800, a portion of the memory array is converted into a matrix structure configured to perform a DCT matrix operation. In one embodiment, at least one Nx2N portion of the memory array is configured as a matrix structure having values of a DCT matrix T (divided into positive T and negative T matrix values), such as Figure 5A The values of the matrix T may be retrieved from a look-up table (LUT) by control logic within the memory device.
[0235] At step 806, one or more matrix multiplication units may be configured to, for example, compile matrix-vector products from a positive T matrix structure and a negative T matrix structure to provide a signed matrix multiplication (similar to Figure 4 described in ).
[0236] Reference now Fig. 8A , method 850 is used to perform a DCT operation on the matrix M (ie, Figure 8 Steps 808 and 810).
[0237] In step 811 of method 850, one or more vectors m representing columns of matrix M are obtained from a memory array. c In one embodiment, the control logic is used to access one vector at a time based on the instruction received in step 802 of method 800 .
[0238] In step 812, a vector m is used cn to drive a matrix structure that has been configured with positive and negative T matrix values.
[0239] In step 814, the MMU obtains the result of the matrix structure and calculates the bcn vector by subtracting the matrix-vector product obtained on the negative T side of the matrix structure from the matrix-vector product obtained on the positive T side of the matrix structure.
[0240] In step 816, the obtained b cn The values of the vector are stored in a memory array.
[0241] Then repeat steps 811 to 816 until the entire B matrix is calculated and accumulated in the memory array. Additional details can be found elsewhere in this disclosure. Figure 5A Found in the discussion.
[0242] Once the entire B matrix has been calculated according to steps 811 to 816, Fig. 8A At step 817, the control logic obtains one or more vectors b representing rows of matrix B from the memory array. r .
[0243] In step 818, a vector b is used rn to drive a matrix structure that has been configured with positive and negative T matrix values, such as Figure 5A described.
[0244] In step 820, the MMU obtains the result of the matrix structure and calculates d by subtracting the matrix-vector product obtained on the negative T side of the matrix structure from the matrix-vector product obtained on the positive T side of the matrix structure. rn vector.
[0245] In step 822, the resulting value of the drn vector is stored in a memory array.
[0246] Steps 817 to 822 are then repeated until the entire D matrix is calculated and the DCT operation is complete. It will be appreciated that the foregoing method can be easily modified by one of ordinary skill in the art for use in Figure 5B and 6B Multi-matrix structure architecture.
[0247] It will also be appreciated that, although certain aspects of the present disclosure are described in terms of a particular order of steps of the method, these descriptions are merely illustrations of a broader method of the present disclosure and may be modified as required for a particular application. Certain steps may be unnecessary or optional in certain circumstances. In addition, certain steps or functions may be added to the disclosed embodiments, or the order of execution of two or more steps may be arranged. In addition, features from two or more methods may be combined. All of these variations are considered to be included in the disclosure disclosed and claimed herein.
[0248] The description set out herein describes example configurations in conjunction with the accompanying drawings and does not represent all examples that can be implemented or within the scope of the claims. The term "exemplary" as used herein means "serving as an example, instance, or illustration," rather than "preferred" or "superior to other examples." The detailed description contains specific details that provide an understanding of the described techniques. However, these techniques may be practiced without these specific details. In some examples, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described examples.
[0249] Any of a variety of different technologies and techniques may be used to represent the information and signals described herein. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.
[0250] Although the above detailed description has shown, described and pointed out the novel features of the present disclosure as applied to various embodiments, it should be understood that various omissions, substitutions and changes in the form and details of the devices or processes shown may be made by those skilled in the art without departing from the present disclosure. This description is in no way meant to be limiting, but should be considered as an illustration of the general principles of the present disclosure. The scope of the present disclosure should be determined by reference to the claims.
[0251] It should also be understood that while certain steps and aspects of the various methods and apparatus described herein can be performed by humans, the disclosed aspects and the various methods and apparatus are typically computerized / computer-implemented. For many reasons, including but not limited to commercial feasibility, practicality, and even usability (i.e., humans cannot simply perform certain steps / processes in any feasible manner), computerized apparatus and methods are required to fully implement these aspects.
[0252] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on a computer-readable medium or transmitted by the computer-readable device (e.g., storage medium) as one or more instructions or codes. Computer-readable media include non-transitory computer storage media and communication media, and communication media include any media that facilitates the transmission of computer programs from one place to another. Non-transitory storage media can be any available media that can be accessed by a general or special computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves are included in the definition of media. As used herein, the disk and optical disc include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blue disc, wherein the disk usually reproduces data magnetically, and the optical disc reproduces data optically by laser. Combinations of the above are also included within the scope of computer-readable media.
Claims
1. A non-transitory computer-readable device comprising: a memory cell array, wherein each memory cell of the memory cell array is configured to store a digital value as an analog value in an analog medium; a memory sensing component, wherein the memory sensing component is configured to read the analog value of a first memory cell as a first digital value; as well as Logic configured to: Receive discrete cosine transform DCT matrix transform operation code; Enable the memory cell array to operate as a matrix multiplication unit MMU based on the DCT matrix transformation operation code; wherein each memory unit of the MMU modifies the analog value in the analog medium according to the DCT matrix transformation operation code and the matrix transformation operand; configuring the memory sensing component to convert the analog value of the first memory cell into a second digital value according to the DCT matrix transform operation code and the matrix transform operand; and In response to reading the matrix transformation operand into the MMU, a matrix transformation result is written based on the second digital value. 2 . The non-transitory computer-readable device of claim 1 , wherein the DCT matrix transform opcode indicates a size of the MMU.
3. The non-transitory computer-readable device of claim 2, wherein the DCT matrix transform operation code corresponds to a frequency domain transform operation. 4 . The non-transitory computer-readable device of claim 3 , wherein the frequency domain transform operation spans at least one other MMU.
5. The non-transitory computer-readable device of claim 1, wherein the DCT matrix transform operation code identifies one or more analog values corresponding to one or more memory cells. 6 . The non-transitory computer-readable device of claim 5 , wherein the one or more analog values corresponding to the one or more memory cells are stored within a look-up table (LUT) data structure.
7. The non-transitory computer-readable device of claim 1 , wherein each memory cell of the MMU comprises a Resistive Random Access Memory (ReRAM) cell; and wherein each memory unit of the MMU multiplies the analog values in the analog medium according to the DCT matrix transform operation code and the matrix transform operand.
8. The non-transitory computer-readable device of claim 7, wherein each memory unit of the MMU further accumulates the analog value in the analog medium with a previous analog value.
9. The non-transitory computer-readable device of claim 1 , wherein the first digital value is characterized by a first base of two (2); and Wherein the second digital value is represented by a second base number greater than two (2).
10. A device for video processing, comprising: a processor in data communication with a non-transitory computer-readable device; wherein the non-transitory computer-readable device comprises one or more instructions that, when executed by the processor, are configured to cause the device to: Writing discrete cosine transform (DCT) matrix transform operation codes and matrix transform operands into the non-transitory computer readable device; wherein the DCT matrix transform operation code causes the non-transitory computer-readable medium to operate a memory cell array as a matrix structure and causes a memory sensing component to convert an analog value to a digital value; wherein the matrix transformation operand modifies one or more analog values corresponding to one or more memory cells of the matrix structure; and The matrix transformation result is read from the matrix structure.
11. The apparatus of claim 10, wherein the non-transitory computer-readable device further comprises one or more instructions that, when executed by the processor, cause the apparatus to: Receiving video data including one or more image blocks; wherein the matrix transformation operands include the one or more image blocks, and the matrix transformation results include one or more DCT frequency domain image coefficients; and The one or more analog values of the matrix structure are accumulated over a period of time from the one or more DCT frequency domain image coefficients of the video data.
12. The apparatus of claim 10, wherein the DCT matrix transform operation code causes the non-transitory computer readable device to operate another memory cell array as another matrix structure; and The matrix transformation result associated with the matrix structure and another matrix transformation result associated with the another matrix structure are logically combined.
13. The device of claim 10, wherein the one or more analog values of the matrix structure are stored within a look-up table (LUT) data structure.
14. A method for performing a matrix transformation operation for video processing, comprising: Receiving a matrix transformation operation code, wherein the matrix transformation operation code includes an instruction to perform a discrete cosine transform (DCT) operation; configuring a memory cell array of a memory into a matrix structure based at least on the matrix transformation operation code; configuring a memory sensing component to convert an analog value into a digital value based on the matrix transformation operation code; and In response to reading a matrix transformation operand into the matrix structure, a matrix transformation result from the memory sensing component is written.
15. The method of claim 14, wherein the configuring the memory cell array comprises connecting a plurality of word lines and a plurality of bit lines corresponding to row and column dimensions associated with the matrix structure.
16. The method of claim 15, further comprising determining the row dimension and the column dimension from the matrix transform opcode.
17. The method of claim 14, wherein the configuring the memory cell array comprises setting one or more analog values of the matrix structure based on a look-up table (LUT) data structure.
18. The method of claim 17, further comprising identifying an entry from the LUT data structure based on the matrix transform opcode.
19. The method of claim 14, wherein said configuring said memory sensing component causes a matrix transformation result to have a cardinality greater than two (2).
20. The method of claim 14, wherein configuring the array of memory cells into the matrix structure comprises configuring impedance values of individual memory cells based on values of a DCT transform matrix.
Citation Information
Patent Citations
Image processor formed in an array of memory cells
US10440341B1
Methods and apparatus for incentivizing participation in fog networks
US20200186607A1
Methods and apparatus for performing video processing matrix operations within a memory array
US20210149984A1
Hardware implementing apparatus and method for inverse discrete cosine transformation during video decoding process
CN101330616A
Encoding and decoding techniques using low-density parity check codes
CN103534952A
