Dual-port, dual-function memory device
By integrating the AI circuit system in the memory device and directly processing the data of the imaging device, the problems of multiple data transmission and resource consumption in the prior art are solved, and the performance and image processing capabilities of the imaging system are improved.
Patent Information
- Application Number
- CN202180080993.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-12-09
- Filing Date
- 2021-11-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-29
AI Technical Summary
The existing memory devices have problems of multiple data transfer and resource consumption during data transfer, resulting in increased consumption of system resources (such as power and time) and affecting the performance of the imaging system.
Integrate artificial intelligence (AI) circuit systems such as neuromorphic arrays or deep learning accelerators (DLAs) in memory devices to process data from imaging devices directly on memory devices, reduce dependence on host processing devices, and reduce data transfer volume .
By reducing the amount of data transmission and resource consumption, the processing speed and image resolution of the imaging system are improved, and the power consumption and waiting time of the system are reduced.
Smart Images

Figure CN116529818B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to memory devices, and more particularly, to apparatus and methods related to dual-port, dual-function memory devices. Background Art
[0002] Memory devices are typically provided as internal semiconductor integrated circuits in computers or other electronic devices. There are many different types of memory, including volatile and non-volatile memory. Volatile memory may require power to maintain its data and includes random access memory (RAM), dynamic random access memory (DRAM), and synchronous dynamic random access memory (SDRAM), among others. Non-volatile memory may provide permanent data by retaining stored data when power is not supplied, and may include NAND flash memory, NOR flash memory, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), erasable programmable ROM (EPROM), and resistance variable memory, such as phase change random access memory (PCRAM), resistive random access memory (RRAM), and magnetoresistive random access memory (MRAM), among others.
[0003] Memories are also used in a wide range of electronic applications as volatile and nonvolatile data storage devices. For example, nonvolatile memories can be used in personal computers, portable memory sticks, digital cameras, cellular phones, portable music players such as MP3 players, movie players, and other electronic devices. Memory cells can be arranged in arrays, where the arrays are used in memory devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Figure 1 is a block diagram of an apparatus in the form of a system including a dual-port, dual-function memory device of a neuromorphic array according to various embodiments of the present disclosure.
[0005] Figure 2 is a block diagram of a dual-port, dual-function memory device including a neuromorphic array according to various embodiments of the present disclosure.
[0006] Figure 3 is a block diagram of a system including a dual-port, dual-function memory device of a neuromorphic array according to various embodiments of the present disclosure.
[0007] Figure 4 is a block diagram of an apparatus in the form of a system including a dual-port, dual-function memory device of a deep learning accelerator (DLA) according to various embodiments of the present disclosure.
[0008] Figure 5is a block diagram of a dual-port, dual-function memory device including a DLA according to various embodiments of the present disclosure.
[0009] Figure 6 is a block diagram of a system including a dual-port, dual-function memory device including a DLA according to various embodiments of the present disclosure.
[0010] Figure 7 A flowchart of a method of performing operations using a neuromorphic array of dual-port, dual-function memory devices according to various embodiments of the present disclosure.
[0011] Figure 8 Flowchart of a method for performing operations using a DLA of a dual-port, dual-function memory device according to various embodiments of the present disclosure. DETAILED DESCRIPTION
[0012] The present disclosure includes apparatus and methods related to dual-port, dual-function memory devices. In some previous methods, data from an imaging device may be communicated to a host device, such as a central processing unit (CPU), for preprocessing before being written to a memory device. Data from an imaging device may be written to a memory device and then communicated from the memory device to a host device for preprocessing. The preprocessed data may be written to the memory device.
[0013] Some previous methods may include multiple data transfers. As described above, for each set of data communicated from the imaging device, there may be a data transfer from the imaging device to the host device and / or the memory device. For each set of data communicated from the imaging device, there may be a data transfer from the memory device to the host processing device, or vice versa. Each data transfer may include multiple operations. For example, the data transfer may include one or more read operations, one or more decompression operations, and / or one or more compression operations. The data transfers and their associated operations may increase the consumption of resources (e.g., power, time, etc.) of the system, which includes the imaging device, the host processing device, and the memory array.
[0014] At least one embodiment of the present disclosure solves these shortcomings and other problems associated with some previous methods by enabling a memory device to store data communicated from an imaging device and using an artificial intelligence (AI) circuit system of the memory device to perform one or more operations on the data written to the memory device. Examples of AI circuit systems include, but are not limited to, neuromorphic arrays and deep learning accelerators (DLA). As used herein, AI refers to the ability of a device and / or system to make decisions based on the results of past decisions. Devices with AI capabilities may be configured to store patterns and / or instances that may be used later to take action. Deep learning refers to the ability of a device to learn from data provided as an example. Deep learning may be a subset of AI. Neural networks and other types of networks may be classified as deep learning. The low-power, inexpensive design of DLA may be implemented in Internet of Things (IOT) devices and other types of devices.
[0015] An inexpensive and energy-efficient AI accelerator such as a DLA may be implemented at the edge of a memory device. As used herein, the edge of a memory device may refer to an area of the memory device other than the area of the memory device occupied by a memory array and / or sensing circuitry. For example, the edge of a memory device may refer to an output and / or input path of a memory device, wherein the output path and / or input path is used to transmit data to and / or from a memory array of the memory device. A DLA implemented at the edge of a memory device may be referred to as an "edge DLA." A memory device including an edge DLA may be deployed at a remote location without the need for cloud or offload capabilities. Improving the latency, power consumption, and / or throughput of a DLA may improve the latency, power consumption, and / or throughput of a memory device including a DLA.
[0016] Neuromorphic computing architectures may include hardware and / or software configured to simulate neuro-biological architectures present in biological neural systems. Neuromorphic computing architectures may include multi-layered neuron architectures referred to as perceptrons. Neurons in multi-layered neuron architectures may be referred to as artificial neurons. Artificial neurons may combine one or more inputs received to produce an output. For example, an artificial neuron may include circuitry configured to simulate and / or perform one or more functions of a biological neuron. The output of a first artificial neuron may be propagated to a second artificial neuron using an artificial synapse.
[0017] At least one embodiment of the present disclosure improves the performance of an imaging system including a memory device and an imaging device coupled thereto because the memory device includes an AI circuit system that is physically closer to the imaging device and / or the memory array of the memory device, as compared to some previous approaches. By including the AI circuit system on the memory device and coupled to the memory array of the memory device, the amount of data transferred between the memory device and a host processing device and / or between the memory device and the imaging device coupled to the memory device is reduced. For example, as compared to some previous approaches, reducing data transfers can increase the number of frames processed during a time period (increased rate, frames / second). Having the AI circuit system physically closer to the memory array, as compared to some previous approaches, can enable the processing of higher resolution images.
[0018] Performing image processing operations on data from an imaging device by a memory device can offload workload from a host device or a host processing device because, in some previous methods, the host device or the host processing device can instruct preprocessing of the data. At least one embodiment can use the AI circuit system of a memory device to perform image processing operations with minimal (if any) interaction with an external instruction source (e.g., a host processing device). For example, the host processing device can provide instructions to the AI circuit system indicating the image processing operations to be performed. However, the AI circuit system can perform those operations without further instructions from the host processing device. As compared to some previous methods, this can reduce the amount of data transfer because the AI circuit system can receive (e.g., directly retrieve) data from a memory array of a memory device. Therefore, as compared to some previous methods, at least one embodiment of the present disclosure can reduce resource consumption and / or resource requirements of an imaging system because less data transfer and operations associated therewith are required.
[0019] At least one embodiment of the present disclosure may include receiving data corresponding to an image captured by an imaging device from an imaging device via a first port of a memory device. As used herein, a "port" of a memory device refers to different physical and logical interfaces of the memory device. Unless otherwise indicated, the terms "port" and "interface" are used interchangeably herein. A port may include multiple physical connections, such as pins. Examples of ports include serial ports (e.g., camera serial interfaces operating according to a mobile industry processor interface (MIPI) protocol) and memory ports (e.g., double data rate (DDR) interfaces). The first port may be dedicated to communication between a memory cell array of a memory device and a host device. Instructions to perform operations on data may be received from a host device via a second port of the memory device. The second port may be dedicated to communication between a neuromorphic array of a memory device and a host device. Data may be written to a memory array (e.g., a memory cell array) of a memory device. Data may be received from a memory array by a neuromorphic array (e.g., directly retrieved). According to the instructions, the neuromorphic array may be used to perform operations on the data.
[0020] As used herein, the singular forms "a", "an", and "the" include both singular and plural referents unless the context clearly dictates otherwise. Furthermore, throughout this application, the word "may" is used in a permissive sense (i.e., having the potential to, being able to), rather than in a mandatory sense (i.e., must). The term "include" and its derivatives mean "including, but not limited to". The term "coupled" means connected directly or indirectly.
[0021] The figures herein follow a numbering convention in which the first digit or digits correspond to the figure number of the drawing and the remaining digits identify an element or component in the drawing. Similar elements or components between different figures may be identified by using similar numerals. For example, 102 may be Figure 1 Component "102" is referred to in the figure, and similar components may be found in Figure 2 202. As will be appreciated, the elements shown in the various embodiments herein may be added, exchanged and / or eliminated in order to provide multiple additional embodiments of the present disclosure. In addition, as will be appreciated, the proportions and relative scales of the elements provided in the figures are intended to illustrate certain embodiments of the present invention and should not be considered in a limiting sense.
[0022] Figure 1 is a block diagram of an apparatus in the form of a system 100 that includes a dual-port, dual-function memory device 102 (hereinafter referred to as memory device 102) that includes a neuromorphic array 122, according to various embodiments of the present disclosure. As used herein, memory device 102, controller 103, memory array 110, sensing circuitry 106, and / or neuromorphic array 122 may also be individually considered to be an "apparatus."
[0023] System 100 includes a host device 101 coupled to a memory device 102, which includes a memory array 110. Although not specifically described as such, in at least one embodiment, host device 101 may represent a plurality of different devices, such as an imaging device coupled to memory device 102 and a host device coupled to memory device 102. Examples of host device 101 include a personal laptop computer, a desktop computer, a digital camera, a smart phone, a memory card reader, an image sensor, and an ISP, or a vehicle (e.g., an autonomous vehicle), as well as various other types of peripheral devices. Host device 101 may include a system motherboard and / or a backplane, and may include multiple processing devices (e.g., one or more processors, microprocessors, or some other type of control circuitry). System 100 may include separate integrated circuits, or both host device 101 and memory device 102 may be located on the same integrated circuit.
[0024] For clarity, system 100 has been simplified to focus on features that are particularly relevant to the present disclosure. For example, memory array 110 may be a hybrid memory cube (HMC), a DRAM array, a static random access memory (SRAM) array, a spin-transfer torque random access memory (STT RAM) array, a PCRAM array, a thyristor random access memory (TRAM) array, an RRAM array, a NAND flash array, and / or a NOR flash array. In at least one embodiment, memory array 110 may be a solid-state drive (SSD). Memory array 110 may include memory cells arranged in rows coupled by access lines, which may be referred to herein as word lines or select lines, and columns coupled by digital lines, which may be referred to herein as bit lines, data lines, or sense lines. Although Figure 1 A single memory array 110 is shown in FIG. 1 , but the embodiment is not limited thereto. For example, the memory device 102 may include multiple memory arrays 110 (eg, multiple banks of DRAM cells, NAND flash memory cells, etc.).
[0025] Memory device 102 may include address circuitry 104 to latch address signals for data provided on an I / O bus 130 (e.g., a data bus and / or an address bus) by input / output "I / O" circuitry 109 (e.g., provided to external ALU circuitry and DRAM DQ via local I / O lines and global I / O lines). Address signals are received by address circuitry 104 and decoded by row decoder 105 and column decoder 107 to access memory array 110. Data may be read from memory array 110 by sensing voltage and / or current changes on digit lines using sense circuitry 106. Sense circuitry 106 may read and latch pages (e.g., rows) of data from memory array 110. I / O circuitry 109 may be used for bidirectional data communication with host device 101 via I / O bus 130. In various embodiments, sense circuitry 106 may include a plurality of sense amplifiers and a corresponding number of computational components, as will be discussed further herein. Write circuitry 108 may be used to write data to memory array 110. The various components of memory device 102 may be implemented in a single device or package, or they may be implemented as one or more discrete components connected with wires, traces, interposers, wireless interfaces, and the like.
[0026] The controller 103 (e.g., a memory controller) decodes signals provided by the control bus 128 from the host device 101. These signals may include chip enable signals, write enable signals, and address latch signals for controlling operations performed on the memory array 110, including data read, data write, and data erase operations. In various embodiments, the controller 103 is responsible for executing instructions from the host device 101 and sequencing accesses to the memory array 110. The controller 103 may be a state machine, a sequencer, or some other type of controller and include hardware and / or firmware (e.g., microcode instructions) in the form of an application specific integrated circuit (ASIC). The controller 103 may control, for example, the generation of clock signals and the application of clock signals to computing components in the sensing circuitry associated with shifting data.
[0027] The memory device 102 includes a neuromorphic array 122. As indicated by double arrows 111, the neuromorphic array 122 may directly access the memory array 110. The double arrows 111 may represent a high-bandwidth interface between the neuromorphic array 122 and the memory array 110. The high-bandwidth interface may include at least an I / O (data) bus, an address bus, and a control bus, for example, dedicated to communication between the memory device 110 and the neuromorphic array 122. A non-limiting example of an internal high-bandwidth interface may be a DRAM DDR interface. In at least one embodiment, the memory array 110 may be coupled to the neuromorphic array 122 such that the high-bandwidth interface is external, such as a peripheral component interconnect express (PCIe) interface. The neuromorphic array 122 may be configured to perform image processing operations on data received from the memory array 110 according to instructions received by the memory device 102. Non-limiting examples of image processing operations include compression operations, frame buffer operations, color space conversion operations, attention discovery operations, and encoding, decoding, or scaling operations.
[0028] If Figure 1 As illustrated, memory device 102 may be a dual-port memory device. Control bus 128 and I / O bus 130 may be collectively referred to as port 129 of memory device 102. Memory device 102 may include another port, port 112, dedicated to communication between host device 101 and neuromorphic array 122. Although port 112 is illustrated as being more general than port 129, port 112 may include, for example, a connection to I / O circuitry 109 or other I / O circuitry (not specifically illustrated). Port 112 may include at least an I / O bus and a control bus, similar to control bus 128 and I / O bus 130, e.g., dedicated to communication between host device 101 and neuromorphic array 122. For example, port 112 may be used to communicate instructions from host device 101 to neuromorphic array 122 and / or to communicate responses from neuromorphic array 122 to host device 101. Port 112 may be, but is not limited to, a MIPI, PCIe interface, or a compute express link (CXL) interface, or a low power double data rate interface, such as LPDDR59. In multiple embodiments, port 112 may be a separate and distinct physical interface of memory device 102. Host device 101 may communicate an instruction to perform an image processing operation to neuromorphic array 122 via port 112. Neuromorphic array 122 may communicate completion of an operation (e.g., an image processing operation) performed by neuromorphic array 122 to host device 101 via port 112. As described above, in at least one embodiment, host device 101 represents more than one different device, wherein port 129 represents an interface for a first host device (e.g., an imaging device), and port 112 represents an interface for a second host device (e.g., a computing device).
[0029] Figure 2 2 is a block diagram of a dual-port, dual-function memory device 202 (hereinafter referred to as memory device 202) including a neuromorphic array 222 according to various embodiments of the present disclosure. The neuromorphic array 222, the memory array 210, the port 212, and the port 229 may be similar to the combination of Figure 1 The neuromorphic array 122, memory array 110, port 112, and port 129 are described. For clarity and ease of description, Figure 1 In the description of the memory device 102, Figure 2 The description of the memory device 202 in FIG. 2 has been simplified to focus on features that are particularly relevant to the present disclosure.
[0030] Figure 2 The dual functionality and dual ports of the memory device 202 are illustrated. The dual functionality refers to the storage capability of the memory device 202 provided by the memory array 210 and the processing capability of the memory device 202 provided by the neuromorphic array 222. The memory device 202 includes a dedicated host device (not shown) (e.g., in conjunction with Figure 1 1 and 10. The memory device 202 may include a port 212 for communication between the host device 101 and the neuromorphic array 222. The port 212 may be coupled to or include processing logic 224 of the memory device 202. The processing logic 224 may, for example, assist in communicating instructions from the host device to the neuromorphic array 222 and communicating responses from the neuromorphic array 222 to the host device.
[0031] The memory device 202 includes a port 229 for communicating data from a host device to the memory array 210 and / or communicating data from the memory array 210 to the host device. The port 229 may be referred to as a data port of the memory device 202. The port 229 may be coupled to or include storage and decoding logic 226 of the memory device 202. The storage and decoding logic 226 may facilitate communication between the memory array 210 and the host device. The storage and decoding logic 226 may provide an artificial neural network (ANN) model to the neuromorphic array 222. In various embodiments, the storage and decoding logic 226 may represent a combination of Figure 1 Respective functionality of one or more of the following components is described: address circuitry 104, I / O circuitry 109, row decoder 105, write circuitry 108, column decoder 107, controller 103, and sense circuitry 106.
[0032] The neuromorphic array 222 may be coupled to the memory array 210 via a high bandwidth connection, such as a serial interface. The neuromorphic array 222 may include a local memory configured to store data received from the memory array 210 on which operations are to be performed. The neuromorphic array 222 may be used as an AI accelerator, coprocessor, and / or preprocessor. The neuromorphic array 222 may be configured to perform multiplication and addition (MAC) operations on data received from the memory array 210.
[0033] The memory device 202 may be a component of a personal laptop computer, a desktop computer, a digital camera, a mobile phone, an IoT-enabled device or vehicle (e.g., an autonomous vehicle), and various other types of systems. In at least one embodiment, the memory device 202 may be a component of an event recorder of a vehicle, such as an autonomous vehicle. An event recorder may also be referred to as a black box.
[0034] The memory device 202 may provide main memory or be used as a system (e.g., in combination with Figure 1 and 3 The cross-point array of non-volatile memory may be combined with a stackable cross-grid data access array to perform bit storage based on changes in body resistance. In addition, compared to many flash memory-based memories, the cross-point non-volatile memory can perform write-in-place operations, where non-volatile memory cells can be programmed without pre-erasing the non-volatile memory cells.
[0035] Figure 3 3 is a block diagram of a system 340 according to various embodiments of the present disclosure, the system including a dual-port, dual-function memory device 302 (hereinafter referred to as memory device 302) including a neuromorphic array 322. The memory device 302 and the neuromorphic array 322 may be similar to a combination of Figure 2 Memory device 202 and neuromorphic array 222 are depicted.
[0036] System 340 includes an imaging device 342 coupled to memory device 302. Imaging device 342 may include an image sensor. Non-limiting examples of imaging devices include cameras and light detection and ranging (LiDAR) devices. Data from imaging device 342 may be written to memory device 302. Memory device 302 may be connected to memory device 302 via a port of memory device 302, such as by combining Figure 2 Port 229 is depicted coupled to imaging device 342 .
[0037] System 340 includes a host device 301. Host device 301 is coupled to imaging device 342 and memory device 302. Figure 1 The depicted host device 101 may represent the host device 301 and the imaging device 342. For example, the host device 301 may provide instructions to the imaging device 342 regarding an image captured by the imaging device 342. The double arrow between the host device 301 and the memory device 302 may represent the connection between the host device 301 and the memory device 302. Figure 1 The double arrow between imaging device 342 and memory device 302 may represent a combination of Figure 1 The host device 301 can be connected to the port 112. Figure 2 Port 212 is depicted as providing instructions to memory device 302 regarding image processing operations to be performed by neuromorphic array 322. Image processing operations may be performed on data received from imaging device 342 and written to memory device 302. Memory device 302 may be configured to provide signaling to host device 301 via port 112 indicating completion of the image processing operations.
[0038] The system 340 includes a mobile device 348. As used herein, a "mobile device" refers to a component of a system that is configured to alter the movement of the system by affecting at least one of the speed, direction, elevation, depth, orientation, or route of the system. Non-limiting examples of mobile devices include motors, controllers, and actuators. In at least one embodiment, the system 340 may be a robotic system, an autonomous drone, and / or an autonomous vehicle. As used herein, a "drone" refers to a device and / or system that cannot transport humans, while a "vehicle" refers to a device and / or system that can transport humans. The mobile device 348 may be coupled to a host device 301. The host device 301 may be configured to indicate activation of the mobile device 348 based on the results of the image processing operations performed by the neuromorphic array 322 to alter the movement of the system 340.
[0039] Figure 44 is a block diagram of an apparatus in the form of a system 445, which includes a dual-port, dual-function memory device 450 (hereinafter referred to as memory device 450) according to various embodiments of the present disclosure. As used herein, memory device 450, controller 403, memory array 410, sensing circuitry 106, and / or DLA 452 may also be individually referred to as "devices". Host device 401, port 412, port 429, controller 403, address circuitry 404, I / O circuitry 409, row decoder 405, write circuitry 408, memory array 410, sensing circuitry 406, and column decoder 407 may be referred to as "devices" respectively. Figure 1 Depicted are host device 101, port 112, port 129, controller 103, address circuitry 104, I / O circuitry 109, row decoder 105, write circuitry 108, memory array 110, sense circuitry 106, and column decoder 107. Although not specifically illustrated as such, in at least one embodiment, host device 401 may represent a number of different devices, such as an imaging device coupled to memory device 450 and a host device coupled to memory device 450.
[0040] System 445 and combined Figure 1 The described system 100 differs in that the memory device 450 includes a DLA 452 as an AI circuit system instead of a neuromorphic array. The memory device 450 includes a DLA 452. The DLA 452 may be configured to perform matrix operations associated with image processing operations on matrix operands. The matrix operands may be written to the memory array 410. In at least one embodiment, the memory array 410 may be an SSD. The DLA 452 may perform matrix operations on matrix operands received from the memory array 410. As represented by the double arrow 411, the DLA 452 may directly access the memory array 410. The double arrow 411 may represent a high-bandwidth interface between the DLA 452 and the memory array 410. The high-bandwidth interface may include at least an I / O (data) bus, an address bus, and a control bus, for example, dedicated to communication between the memory device 410 and the DLA 452. A non-limiting example of an internal high-bandwidth interface may be a DRAM DDR interface. In at least one embodiment, memory array 410 may be coupled to DLA 452 such that a high bandwidth interface is external, such as a PCIe interface.
[0041] If Figure 4As illustrated, memory device 450 may be a dual-port memory device. Control bus 428 and I / O bus 430 may be collectively referred to as port 429 of memory device 450. Memory device 450 may include port 412 dedicated to communication between host device 401 and DLA 452. Although port 412 is illustrated as being more general than port 429, port 412 may include, for example, a connection to I / O circuitry 409 or other I / O circuitry (not specifically illustrated). Port 412 may include at least an I / O bus and a control bus, similar to control bus 428 and I / O bus 430, for example, dedicated to communication between host device 401 and DLA 452. For example, port 412 may be used to communicate instructions from host device 401 to DLA 452 and to communicate responses from DLA 452 to host device 401. Port 412 may be, but is not limited to, a MIPI, PCIe interface, or CXL interface, or a low power double data rate interface, such as LPDDR5. In various embodiments, port 412 may be a separate and distinct physical interface of memory device 450. Host device 401 may communicate instructions to perform image processing operations to DLA 452 via port 412. DLA 452 may indicate completion of the operation (e.g., image processing operation) performed by DLA 452 to host device 401 via port 412. As described above, in at least one embodiment, host device 401 represents more than one distinct device, with port 429 representing an interface for a first host device (e.g., an imaging device) and port 412 representing an interface for a second host device (e.g., a computing device).
[0042] Figure 5 5 is a block diagram of a dual-port, dual-function memory device 550 (hereinafter referred to as memory device 550) including a DLA 552 according to various embodiments of the present disclosure. DLA 552, memory array 510, port 512, and port 529 may be similar to the combination of Figure 4 DLA 452, memory array 410, port 412, and port 429 are described. For clarity and ease of description, relative to Figure 4 In the description of the memory device 450, Figure 5 The description of the memory device 550 in FIG. 5 has been simplified to focus on features that are particularly relevant to the present disclosure.
[0043] Figure 5 The dual functionality and dual port of memory device 550 are illustrated. The dual functionality refers to the storage capability of memory device 550 provided by memory array 510 and the processing capability of memory device 550 provided by DLA 552. Memory device 550 includes dedicated host devices (not shown) (e.g., in conjunction with Figure 44 and 5. The memory device 550 may include a port 512 for communication between the host device 401 and the DLA 552 as described above. For example, the port 412 may be used for communication between the host device and the DLA 452 and / or for communicating responses from the DLA 452 to the host device. The port 512 may be coupled to or include processing logic 524 of the memory device 550. The processing logic 524 may assist in communicating instructions from the host device to the DLA 552 and communicating responses from the DLA 552 to the host device.
[0044] The memory device 550 includes a port 529 for communicating data from a host device to the memory array 510 and / or communicating data from the memory array 510 to the host device. The port 529 may be coupled to or include storage and decoding logic 526 of the memory device 550. The storage and decoding logic 526 may facilitate communication between the memory array 510 and the host device. The storage and decoding logic 526 may provide an artificial neural network (ANN) model to the DLA 552. In various embodiments, the storage and decoding logic 526 may represent a combination of Figure 4 Respective functionality of one or more of the following components is described: address circuitry 404, I / O circuitry 409, row decoder 405, write circuitry 408, column decoder 407, controller 403, and sense circuitry 406.
[0045] The DLA 552 may be coupled to the memory array 510 via a high bandwidth connection, such as a serial interface. The DLA 552 may include a local memory configured to store matrix operands received from the memory array 510 on which operations are to be performed. The DLA 552 may be used as an AI accelerator, coprocessor, and / or preprocessor. The DLA 552 may be configured to perform MAC operations on matrix operands received from the memory array 510.
[0046] The memory device 550 may be a component of a personal laptop computer, a desktop computer, a digital camera, a mobile phone, an IoT-enabled device or vehicle (e.g., an autonomous vehicle), and various other types of systems. In at least one embodiment, the memory device 550 may be a component of an event recorder of a vehicle, such as an autonomous vehicle. An event recorder may also be referred to as a black box.
[0047] Memory device 550 may provide main memory or be used as a system (e.g., in combination with Figure 4 and 6The cross-point array of non-volatile memory may be combined with a stackable cross-grid data access array to perform bit storage based on changes in body resistance. In addition, compared to many flash-based memories, the cross-point non-volatile memory can perform write-in-place operations, where non-volatile memory cells can be programmed without pre-erasing the non-volatile memory cells.
[0048] Figure 6 FIG. 6 is a block diagram of a system 660 including a dual-port, dual-function memory device 650 including a DLA 652 according to various embodiments of the present disclosure. The memory device 650 and the DLA 652 may be similar to a system 660 incorporating a dual-port, dual-function memory device 650. Figure 5 Memory device 550 and DLA 552 are depicted.
[0049] The system 660 includes an imaging device 642 coupled to a memory device 650. The imaging device 642 may include an image sensor. Non-limiting examples of imaging devices include cameras and LiDAR devices. Data from the imaging device 642 may be written to the memory device 650.
[0050] System 660 includes host device 601. Host device 601 is coupled to imaging device 642 and memory device 650. Host device 646 and imaging device 642 are combined by Figure 4 The depicted host device 401 indicates that, for example, the host device 601 may provide instructions to the imaging device 642 regarding images captured by the imaging device 642. The double arrow between the host device 601 and the memory device 650 may indicate that the host device 601 is connected to the memory device 650. Figure 4 The double arrow between imaging device 642 and memory device 650 may represent a combination of Figure 1 The host device 601 can be connected to the port 112. Figure 5 Port 512 is depicted as providing instructions to memory device 650 regarding image processing operations performed by DLA 652 on matrix operands from imaging device 642 written to memory device 650. Memory device 650 may be configured to provide signaling to host device 601 via port 512 indicating completion of the image processing operation.
[0051] System 660 includes mobile device 648. Non-limiting examples of mobile devices include motors, controllers, and actuators. In at least one embodiment, system 660 can be a robotic system, an autonomous drone, and / or an autonomous vehicle. Mobile device 648 can be coupled to host device 601. Host device 601 can be configured to indicate activation of mobile device 648 based on the results of image processing operations performed by DLA 652 to alter the movement of system 660.
[0052] Figure 7 A flowchart of a method for performing operations using a neuromorphic array of dual-port, dual-function memory devices according to various embodiments of the present disclosure. Figure 7 The described method can be combined, for example, with Figure 1 The described memory device 102 is implemented.
[0053] At block 770, the method may include receiving first signaling from the imaging device via a first port of the memory device indicating data corresponding to an image captured by the imaging device. The first port may be dedicated for communication between a memory cell array of the memory device and a host device.
[0054] At block 771, the method may include receiving second signaling from a host device via a second port of the memory device indicating an instruction to perform an operation on the data. The second port may be dedicated to communication between the neuromorphic array of the memory device and the host device. The imaging device may be a LiDAR device. The first signaling may be received from the LiDAR device via the first port.
[0055] At block 772, the method may include writing data to the memory cell array. At block 773, the method may include receiving (e.g., directly retrieving) data from the memory cell array by the neuromorphic array. At block 774, the method may include performing an operation on the data according to the instruction by the neuromorphic array. Performing the operation may include performing a MAC operation on the data by the neuromorphic array, and communicating the result of the operation from the neuromorphic array to the memory cell array by the neuromorphic array. A third signaling indicating completion of the operation may be communicated to the host device via the second port. Performing the operation may include performing at least one of: a compression operation; a frame buffer operation; an encoding, decoding, or scaling operation; a color space conversion operation; and an attention discovery operation.
[0056] Figure 8 Flowchart of a method for performing matrix operations using a dual-port, dual-function memory device DLA according to various embodiments of the present disclosure. Figure 8 The described method can be combined, for example, with Figure 4 The described memory device 450 is implemented.
[0057] At block 880, the method may include receiving, from an imaging device via a first port of a memory device, first signaling indicating a matrix operand corresponding to an image captured by the imaging device. The first port may be dedicated to communication between a memory cell array of the memory device and a host device. The imaging device may be a LiDAR device. The LiDAR device, the memory device, and the host device may be components of an autonomous vehicle. The first signaling may be received from the LiDAR device via the first port.
[0058] At block 881, the method may include receiving second signaling from a host device via a second port of the memory device indicating an instruction to perform a matrix operation on a matrix operand. The second port may be dedicated for communication between a DLA of the memory device and the host device. The first signaling, the second signaling, and performing the matrix operation may be associated with navigating an autonomous vehicle.
[0059] At block 882, the method may include writing a matrix operand to a memory cell array. At block 883, the method may include receiving (e.g., directly retrieving) the matrix operand from the memory cell array by the DLA. At block 884, the method may include performing, by the DLA, the matrix operation on the matrix operand according to the instruction. A third signaling indicating completion of the matrix operation may be communicated to the host device via the second port. Performing the matrix operation may include performing at least one of: a compression operation; a frame buffer operation; an encoding, decoding, or scaling operation; a color space conversion operation; and an attention discovery operation.
[0060] Embodiments may include a tangible machine-readable storage medium (also referred to as a computer-readable medium) on which is stored one or more sets of instructions or software embodying any one or more of the methods or functions described herein. In some embodiments, a memory device or a processing device constitutes a machine-readable medium. The term "machine-readable storage medium" includes a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" includes any medium capable of storing or encoding a set of instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present disclosure. Thus, the term "machine-readable storage medium" includes, but is not limited to, solid-state memory, optical media, and magnetic media.
[0061] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that arrangements intended to achieve the same results may replace the specific embodiments shown. The present disclosure is intended to cover adaptations or variations of the various embodiments of the present disclosure. It should be understood that the above description has been made in an illustrative manner rather than a restrictive manner. After reviewing the above description, it will be immediately clear to those of ordinary skill in the art that the combination of the above-described embodiments and other embodiments not specifically described herein. The scope of the various embodiments of the present disclosure includes other applications in which the above-described structures and methods are used. Therefore, the scope of the various embodiments of the present disclosure should be determined with reference to the attached claims together with the entire scope of equivalents authorized by such claims.
[0062] In the foregoing Detailed Description, various features are grouped together in a single embodiment for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the disclosed embodiments of the disclosure must use more features than are expressly recited in each claim. Rather, as reflected in the appended claims, inventive subject matter lies in less than all features of a single disclosed embodiment. Therefore, the appended claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.
Claims
1. A method comprising: receiving, via a first port of a memory device, first signaling indicating data from an imaging device external to the memory device; wherein the first port is dedicated to communication between a memory cell array inside the memory device and a host device outside the memory device; receiving second signaling via a second port of the memory device, the second signaling indicating an instruction from the host device to perform an operation on the data using a neuromorphic array within the memory device, wherein the second port is dedicated to communication between the neuromorphic array and the host device; The data is written to the memory cell array.
2. The method according to claim 1, further comprising: performing multiplication and addition MAC operations on the data via the neuromorphic array; and Results of the MAC operation are communicated from the neuromorphic array to the memory cell array through the neuromorphic array.
3. The method of claim 1, wherein the imaging device comprises a Light Detection and Ranging (LiDAR) device, and Wherein receiving the first signaling includes receiving the first signaling from the LiDAR device via the first port. 4 . The method of claim 3 , further comprising navigating a vehicle including the LiDAR device and the memory device based on the first and second signaling and a result of the operation.
5. The method of claim 1, wherein performing the operation comprises performing at least one of the following: Compression operation; Frame buffer operations; encoding, decoding or scaling operations; Color space conversion operations; and Attention finding operation.
6. A method comprising: receiving, via a first port of a memory device, first signaling indicating a matrix operand from an imaging device external to the memory device, wherein the first port is dedicated to communication between a memory cell array inside the memory device and a host device outside the memory device; receiving, via a second port of the memory device, second signaling indicating an instruction from the host device to perform a matrix operation on the matrix operand, wherein the second port is dedicated to communication between the deep learning accelerator DLA inside the memory device and the host device; and The matrix operation is performed on the matrix operands according to the instruction by the DLA.
7. The method of claim 6, wherein the memory device and the host device are components of an autonomous vehicle.
8. The method of claim 7, wherein the first signaling, the second signaling, and performing the matrix operation are associated with navigating the autonomous vehicle.
9. The method of claim 6, further comprising communicating third signaling to the host device via the second port indicating completion of the matrix operation.
10. A memory device comprising: a first port coupled to the memory cell array and configured to communicate first signaling indicative of data from an imaging device external to the memory device to the memory cell array internal to the memory device, wherein the first port is dedicated to communication between the imaging device and the memory cell array; as well as a second port coupled to the neuromorphic array and configured to communicate second signaling indicative of an instruction to perform an operation from a host device external to the memory device to the neuromorphic array internal to the memory device, wherein the second port is dedicated for communication between the neuromorphic array and the host device.
11. The memory device of claim 10, further comprising the neuromorphic array coupled to the memory cell array and configured to perform multiply and add MAC operations according to the instruction.
12. The memory device of claim 11, wherein the neuromorphic array is further configured to function as an artificial intelligence (AI) accelerator.
13. The memory device of claim 11, wherein the neuromorphic array is further configured to function as a co-processor or a pre-processor.
14. The memory device of claim 10, further comprising: storage and decoding logic coupled to said first port; and Processing logic is coupled to the second port.
15. The memory device of claim 10, wherein the imaging device comprises a Light Detection and Ranging (LiDAR) device.
16. The memory device of claim 10, further comprising a movement device coupled to the memory device and configured to alter movement of a vehicle comprising the imaging device, the memory device, and the neuromorphic array.
17. The memory device of claim 16, wherein the moving device comprises at least one of a motor, a controller, and an actuator.
18. The memory device of claim 16, wherein activation of the mobile device is based at least in part on a result of the operation.
19. The memory device of claim 18, wherein navigation of the vehicle is based at least in part on the activation of the mobile device.
20. The memory device of claim 10, further comprising the memory cell array, wherein the memory cell array comprises a solid state drive (SSD).
Citation Information
Patent Citations
Methods and arrangements to manage memory in cascaded neural networks
CN110389910A
Device structure for neuromorphic computing system
US10242737B1
Neural Network Processor
US20160342891A1
Neural network apparatus, neural network processor, and method of operating neural network processor
US20180253636A1
Apparatuses and methods for in-memory operations
US20190065111A1
