Method for processing an image and programmable data storage system

Abstract

A method and programmable data storage system for processing an image are disclosed. A processor identifies luminance data and chrominance data of a received image and retrieves a first machine learning model stored in a storage device. The first model is applied to make a first prediction about the image based on the luminance data and make a first determination about a standard. In response to making the first determination, a first label associated with the first prediction is returned. A second determination is also made about the standard. In response to making the second determination, a second machine learning model stored in the storage device is retrieved. The second machine learning model is applied to make a second prediction about the image based on color data associated with the image and determine a second label associated with the second prediction.

Description

[0001] This application claims priority and benefit to U.S. Provisional Application No. 62 / 977,111, filed February 14, 2020, entitled “In-Storage-Based Image Processing Based on Machine Learning,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to data processing, and more specifically, to methods for processing images and programmable data storage systems. Background Technology

[0003] Machine learning can be used for data processing tasks, such as image processing tasks including computer vision. Image processing typically requires analyzing digital images for pattern recognition. Some machine learning algorithms can perform this analysis using color information from digital images. Color information can be the RGB (red, green, blue) data values ​​of the pixels that make up the digital image. RGB input can be represented via three data channels, which leads to more computation for image processing.

[0004] Therefore, there is a desire for a system and method for image processing that utilizes machine learning, which can improve the efficiency of image processing tasks compared to those using traditional image processing methods. Summary of the Invention

[0005] According to one embodiment, a method for processing an image includes: receiving an image from a source by a processor embedded in a storage device. The processor identifies luminance data and chrominance data of the received image and retrieves a first machine learning model stored in the storage device. The processor applies the first machine learning model to make a first prediction about the image based on the luminance data and makes a first determination regarding a criterion. In response to making the first determination regarding the criterion, the processor returns a first label associated with the first prediction. The processor also makes a second determination regarding the criterion. In response to making the second determination regarding the criterion: the processor retrieves a second machine learning model stored in the storage device, applies the second machine learning model to make a second prediction about the image based on color data associated with the image, and determines a second label associated with the second prediction.

[0006] According to one embodiment, the storage device includes a solid-state drive, and the embedded processor includes a field-programmable gate array (FPGA).

[0007] According to one embodiment, the first machine learning model and the second machine learning model include deep neural networks.

[0008] According to one embodiment, a first prediction or a second prediction determines that an object is depicted in the image, and a first label or a second label identifies the object.

[0009] According to one embodiment, a first label identifies the object, and a second label provides color information of the identified object.

[0010] According to one embodiment, a confidence value for a first prediction is determined, wherein a second determination is made in response to determining that the confidence value for the first prediction is below a threshold.

[0011] According to one embodiment, an image processing task is identified, wherein a second determination is made in response to determining that the image processing task includes color prediction.

[0012] According to one embodiment, the first machine learning model is trained using an image containing brightness data.

[0013] According to one embodiment, the second machine learning model is trained using an image containing color data.

[0014] According to one embodiment, the storage device is hosted in a computing device of at least one of an autonomous vehicle, an edge data center, a smart store, and a smartphone.

[0015] Embodiments of this disclosure also relate to a programmable data storage system, the programmable data storage system comprising: a non-volatile memory; a storage controller configured to control the non-volatile memory; and a processor configured to: receive an image from a source; identify luminance data and chrominance data of the received image; retrieve a first machine learning model stored in the non-volatile memory via the storage controller; apply the first machine learning model to make a first prediction about the image based on the luminance data but ignoring the color data; make a first determination associated with a criterion; in response to making the first determination associated with the criterion, return a first label associated with the first prediction; make a second determination associated with the criterion; and in response to making the second determination associated with the criterion: retrieve a second machine learning model stored in the non-volatile memory via the storage controller; apply the second machine learning model to make a second prediction about the image based on color data associated with the image; and determine a second label associated with the second prediction.

[0016] As those skilled in the art will recognize, embodiments of this disclosure result in less computation time and / or energy consumption of the processing elements, at least in part, due to the smaller number of pixels processed when using the image's luminance information (as opposed to RGB or additional color information). Computing a smaller number of pixels can generally lead to faster image searches with lower power consumption. Furthermore, when used in applications involving autonomous vehicles, the disclosed embodiments can enable moving vehicles to identify objects more quickly than conventional methods that take full-color information into account, or enable smartphones to identify owners more quickly during login.

[0017] These and other features, aspects, and advantages of embodiments of the present disclosure will be more fully understood when considered in conjunction with the following detailed description, the appended claims, and the accompanying drawings. Of course, the actual scope of the invention is defined by the appended claims. Attached Figure Description

[0018] The following figures describe non-limiting and non-exhaustive embodiments of this example, wherein, unless otherwise stated, the same reference numerals refer to the same parts (components) in the various figures.

[0019] Figure 1 This is a block diagram of a system for in-memory data processing according to an exemplary embodiment;

[0020] Figure 2 This is a more detailed block diagram of a storage device according to an exemplary embodiment; and

[0021] Figure 3 This is a flowchart of a process for in-store data processing according to an exemplary embodiment. Detailed Implementation

[0022] The detailed description set forth below with reference to the accompanying drawings is intended as a description of exemplary embodiments of a method for in-memory image processing via machine learning provided in this disclosure, and is not intended to represent the only form in which this disclosure can be constructed or utilized. This description, in conjunction with the illustrated embodiments, illustrates the features of this disclosure. However, it should be understood that the same or equivalent functions and structures can be implemented by different embodiments that are also intended to be included within the scope of the disclosure. As indicated elsewhere herein, the same reference numerals are intended to indicate the same elements or features.

[0023] Some computational tasks (such as image recognition in autonomous vehicles during autonomous driving) may require completion as quickly as possible. However, a certain level of accuracy may be necessary because moving vehicles need to quickly identify objects in their surroundings to maneuver the car appropriately. Other types of image processing tasks (such as certain image processing tasks in smart stores (e.g., cashierless stores, retail stores incorporating artificial intelligence and / or virtual reality, etc.) and / or edge computing devices and / or smartphones) may also benefit from faster computation of images. Performing image recognition using RGB information across the three color channels may result in increased computation, potentially increasing the time required to identify objects and possibly consuming more power. For some tasks or certain objects, image recognition may not require analysis using RGB (color) information. Instead, grayscale and / or brightness information may be sufficient.

[0024] For example, if image processing is performed for autonomous driving applications on autonomous vehicles, it may not be necessary to identify certain features and / or elements of the environment surrounding the vehicle. For instance, while it might be desirable to computationally determine, for example, a pedestrian in front of the vehicle, it might not be important to computationally determine, for example, the specific clothing of a given pedestrian or the color of such clothing. Doing so would slow down image processing operations and could lead to reduced performance and / or reliability in situations where decisions might need to be made based on the results of image processing with a predetermined waiting time.

[0025] In one embodiment, image processing computations can be accelerated by running machine learning algorithms within the data storage device itself, using grayscale / luminance data in the first stage of image processing without considering color information. Processing luminance information as a first step, compared to processing color information, results in processing a smaller number of pixels. This can lead to relatively low power consumption while allowing for faster detection, recognition, identification, and / or classification of objects (collectively, classification). Based on the objects identified during the first step and their attributes, color information can be used as a second step in the image processing process to make more accurate predictions about the image, as needed. For example, in some cases, autonomous vehicles may need not only to recognize the presence of lane markings or traffic lights in an image but also the color of such lane markings or traffic lights. Therefore, if the task at hand explicitly or inherently requires color consideration, color information can be considered during the second stage of image processing.

[0026] When compared to image processing on the host device's central processing unit (CPU), running the first and second stages of image processing within a storage device (e.g., via a computation-based storage approach) can accelerate object recognition in an image. For example, moving large amounts of image data or machine learning models to the CPU for processing and analysis can be expensive in terms of energy consumption and the deployed computing and network resources. This data movement can increase the burden on resources, including but not limited to network bandwidth, CPU cycles, and CPU memory (both capacity and bandwidth). By implementing image processing on the same storage device storing image data and machine learning models, the exposed system can utilize higher available bandwidth within the device (e.g., a solid-state drive (SSD)) and preserve the limited bandwidth between the storage device and the host CPU interface (such as a Peripheral Component Interconnect Fast (PCIe) interface or network infrastructure interface in the case of Remote Direct Attached Storage (RDAS)) or CPU interconnect (such as Gen Z, CCIX (Chip-to-Chip Interconnect Architecture), and OpenCAPI (Open Coherent Accelerator Processor Interface)).

[0027] Therefore, technologies for processing data closer to or inside the storage device are expected to offer time savings by consuming less power, resulting in more cost-effective image data processing compared to processing images in a conventional central processing unit (CPU) outside the storage device.

[0028] Figure 1 This is a block diagram of a system for in-memory image processing according to an exemplary embodiment. The system includes an application module 100 coupled to a programmable storage device 102 (e.g., the programmable storage device 102 may be represented as a data storage device 102) via a data communication link 104. The data storage device 102 may also be coupled to one or more sensors 106 and a computing device 108 via data communication links 110, 112, respectively. Data communication links 104, 110, 112 may take the form of a PCIe (Peripheral Component Interconnect) connection, an Ethernet connection, etc.

[0029] In one embodiment, one or more sensors 106 include one or more cameras for capturing images and transmitting them to a data storage device 102. If image processing is to be performed to enable or assist autonomous driving, the sensors 106 may also include sensors typical of autonomous vehicles (such as LiDAR, 360° cameras, far-infrared ground-penetrating radar, ultrasonic sensors, etc.). One or more such sensors can be used together to detect, for example, that a vehicle is moving or to detect, for example, specific products in a smart store.

[0030] In one embodiment, data storage device 102 may include a solid-state drive (SSD) with an embedded processor. The SSD may be, for example, a non-volatile memory fast (NVMe) (NVMe-oF) compatible SSD (such as an Ethernet SSD (eSSD), an NVMe SSD, or other persistent (non-volatile) memory device).

[0031] In one embodiment, data storage device 102 is hosted in an edge computing system (such as a micro data center, cell tower, autonomous vehicle, smart store, smartphone, IoT (Internet of Things) device, etc.). One or more of application module 100 and sensor 106 may also be hosted in the edge computing system.

[0032] Application module 100 may include any software application configured to transmit image processing tasks and / or queries to data storage device 102. Application module 100 may also be configured to take actions based on image predictions returned by data storage device 102. When integrated into autonomous vehicles or smart stores, application module 100 may be configured to send a request to data storage device 102 to identify objects in images captured by sensors (e.g., cameras) 106. For autonomous vehicles, the identified objects may be pedestrians, other vehicles, lane markings, traffic lights, and / or objects around the vehicle. In response to the identification of objects in the image, application module 100 may be configured to take specific actions (such as controlling the vehicle to stop, decelerate, or move in a specific direction).

[0033] Within a smart store, the objects identified by data storage device 102 can be, for example, products selected by customers, SKU codes of such products, or customers in the store. In response to, for example, a customer leaving the store with products, application module 100 can be configured to retrieve product information, including, for example, the price of the products, for automatically charging the customer for those products. Quickly identifying products selected by customers within the smart store also enables the determination of customer trends and interests within the store.

[0034] In one embodiment, application module 100 may include a search engine residing on any host device accessible to the user. The user can enter a search query and send it to a data storage device 102, for example, hosted in a microdata center, for processing. For example, a user can request images of a brown dog or a yellow cat. The resulting images that satisfy the user's query can then be provided to application module 100 for display on the host device.

[0035] In one embodiment, data storage device 102 can interact with computing device 108 to exchange data. Computing device 108 can be a server, cloud device, another storage device, etc. For example, computing device 108 can be configured to store various pre-trained machine learning models and, appropriately, push one or more of these models to data storage device 102. This can occur, for example, during the initialization of data storage device 102, and / or when data storage device 102 needs to switch between a machine learning model already present in data storage device 102 and another model retrieved from computing device 108. Instead of pushing models to data storage device 102, data storage device 102 can also be configured to, appropriately, pull certain machine learning models. Computing device 108 can select and download different machine learning models based on the image processing task or based on the type of request from application module 100. For example, in an autonomous vehicle, if the vehicle is driving in foggy weather, computing device 108 can select a machine learning model specifically trained to detect objects in foggy weather.

[0036] Figure 2 This is a more detailed block diagram of a data storage device 102 according to an exemplary embodiment. The data storage device 102 includes an embedded CPU 200 and other embedded processors including a field-programmable gate array (FPGA) 202 and a storage controller (e.g., an SSD controller) 204. The data storage device 102 may also include non-volatile memory 206 (e.g., NAND flash memory). In some embodiments, the data storage device 102 may be housed within a bracket (or chassis, not shown) that includes an Ethernet switch and a baseboard management controller (BMC) connected via a directly connected PCIe interface or PCIe switch. The Ethernet switch provides Ethernet connectivity to the data storage device 102 via a midplane, and the PCIe switch and / or SMBus switch provides a management interface to the data storage device 102 via the midplane. In some embodiments, the midplane may be decomposed into multiple smaller portions that provide similar or identical connections to the larger midplane, and these smaller portions may be functionally (e.g., logically or from a circuit perspective) similar or identical. Various small PCBs (printed circuit boards) can be used to implement decoupled portions of the midplane.

[0037] In one embodiment, the BMC can be configured to program the data storage device 102 according to instructions given by a system administrator. The BMC can also be configured to manage the internal components of the rack (including an Ethernet switch, a PCIe switch, and the data storage device 102). In some embodiments, the Ethernet switch can provide network connectivity to the data storage device 102, such that in some embodiments, the Ethernet switch can communicate with the application module 100 and / or the computing device 108 via a data communication network.

[0038] In one embodiment, FPGA 202 uses machine learning algorithms to process one or more modules for in-memory image processing. For example, FPGA 202 can be connected to the one or more modules via a Serial Peripheral Interface (SPI). The one or more modules include an image processing unit 208, a luminance / chrominance separator 210, and one or more machine learning models 212. Although FPGA 202 is used as an example of an embedded processor including modules for in-memory image processing, those skilled in the art will recognize that other commercially available embedded processors, including application-specific integrated circuits (ASICs) or custom-designed dedicated processors (such as TensorFlow CPUs), can be used. Furthermore, although the one or more modules 208, 210, and 212 are assumed to be separate functional units, those skilled in the art will recognize that the functionality of the modules can be combined or integrated into a single module, or further subdivided into more sub-modules, without departing from the spirit and scope of the inventive concept.

[0039] In one embodiment, the luminance / chrominance separator 210 is configured to receive an image provided by the sensor 106 and separate luminance information from chrominance information. This can be achieved, for example, by mathematically mapping the RGB color information of the received image to the YCbCr color space, where Y represents the luminance component of a pixel, and Cb and Cr represent the chrominance components of a pixel. The mapping of RGB color information to the YCbCr color space can be implemented, for example, as described below.

[0040] Assume E R E G and E B It is an analog value between 0 and 1, which describes how much red, green, and blue are present in a pixel (given octet quantization has E). R =R / 255, E G =G / 255 and E B =B / 255). Then, the typical luminance-chrominance conversion (e.g., ITU-R Recommendation 624-4 System B,G) is given by the following formula:

[0041] E Y =0.299-ER +0.587-E G +0.114-E B

[0042] E Cb = -0.169-E R -0.331-E G +0.500-E B

[0043] E Cr =0.500-E R -0.419-E G -0.081-E B

[0044] Among them, E Y Between 0 and 1, and E Cb and E Cr Between -0.5 and 0.5.

[0045] Then, the conversion to an 8-bit value is completed using the following formula:

[0046] Y = 219 - E Y +16

[0047] Cb = 224 - E Cb +128

[0048] Cr = 224-E Cr +128

[0049] The converted pixel values ​​can then be compressed via chroma subsampling to efficiently store the image data in the data storage device 102. Various chroma subsampling methods can be used, each expressed as a three-part ratio corresponding to the Y, Cb, and Cr values ​​(e.g., 4:0:0, 4:2:0, 4:2:2, etc.). Subsampling methods can algorithmically discard chroma values ​​at specified pixel locations. The original image and / or compressed image are then stored in the data storage device 102. Subsequent applications using the compressed image (e.g., image processing unit 208) can simply replace the remaining chroma values ​​at their locations.

[0050] Image processing unit 208 can classify an image generated from pixel values ​​using converted pixel values ​​in the YCbCr color space. In one embodiment, image processing unit 208 uses a two-stage approach to perform classification quickly. During the first stage, image processing unit 208 performs image processing using only luminance information. An appropriate machine learning model 212 is retrieved from non-volatile memory 206 via storage controller 204 for the first stage processing. Depending on the identified object and its attributes, chromaticity information can be used as needed during the second stage of processing for more accurate predictions.

[0051] In one embodiment, chromaticity information is used during the second stage when the prediction confidence is below a threshold, or when a particular image processing task requires consideration of color. However, in most cases, luminance information may be sufficient for object recognition. In such cases, image processing can be completed faster by performing calculations based on luminance information without adversely compromising accuracy. When a two-stage approach is used for moving vehicles, objects can be recognized in less time than with conventional mechanisms. Experiments show that calling the machine learning model using only luminance information results in a 21% to 32% improvement in prediction time compared to using the same model with RGB data, while accuracy decreases by only 1.4% to 1.8%. Furthermore, power consumption is also reduced when the second stage processing is not performed because the number of pixels processed using only luminance information during the first stage is reduced.

[0052] In one embodiment, various pre-trained machine learning models 212 are stored in the non-volatile memory 206 of the data storage device 102. The machine learning models 212 can all be deep neural networks (including, but not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTMs), and / or combinations thereof), wherein each machine learning model can include various neural network architectures with different numbers of layers and different numbers of nodes within each layer. Different types of machine learning models can be trained for different types of tasks to be performed. For example, one model can be trained for autonomous driving, another for recognizing household items, and yet another for recognizing handwritten characters. In some embodiments, a single general-purpose machine learning model can be trained for multiple types of different image processing tasks.

[0053] Various models can also be trained and maintained for specific types of image processing tasks to be performed (e.g., image processing for autonomous driving). In one embodiment, the model is trained using luminance information, chrominance information, both luminance and chrominance information, and / or RGB data. In one embodiment, the model can be trained or retrained at computing device 108, for example, during nighttime. During the initialization or reconfiguration of data storage device 102, one or more appropriate models can be retrieved from computing device 108 and stored in the non-volatile memory 206 of data storage device 102.

[0054] The type of component invoked by the image processing unit 208 for the image processing task depends on whether a first or second prediction phase is being performed. In one embodiment, a component trained using only luminance data is retrieved during the first prediction phase. If the prediction proceeds to the second phase, a component trained based on only chroma, luminance, and chroma data, or RGB data, is retrieved. For example, if the goal is to predict the color of an object to be identified, a component trained using only chroma data can be retrieved during the second phase. In another example, if a prediction using only luminance information is not satisfied because the identified object is below a predetermined threshold representing a confidence value associated with the prediction, a component trained on both luminance and chroma data, or a component trained on RGB data, can be retrieved during the second phase.

[0055] Figure 3 This is a flowchart of a process for in-memory image processing according to an exemplary embodiment. In action 300, image processing unit 208 receives an image from, for example, sensor 106. If image processing is to be performed for autonomous driving, it is determined in action 302 whether the autonomous vehicle is moving. Such information may be provided, for example, by one or more sensors 106 integrated into the autonomous vehicle.

[0056] In action 304, the image processing unit 208 invokes the luminance / chrominance separator 210 to convert the RGB data in the received image into luminance and chrominance data. If the received image is already in the YCbCr color space, such conversion is not necessary.

[0057] In action 306, luminance data and chrominance data are stored separately in the non-volatile memory 206 of data storage device 102. In one embodiment, the luminance data and / or chrominance data are retained in data storage device 102 even after the prediction process is complete. This can be used, for example, to retrain a machine learning model based on feedback received by image processing unit 208 regarding the accuracy of the prediction. The disclosed system can retain luminance data and chrominance data for at least a period of time to reconstruct the received image and provide it to a requesting user. In some embodiments, the luminance data and / or chrominance data are deleted from the non-volatile memory 206 once the prediction process is complete. This can help free up memory when the data is no longer needed.

[0058] In action 308, image processing unit 208 (via storage controller 204) selects one of the machine learning models 212 stored in non-volatile memory 206 of data storage device 102 and applies the selected machine learning model to perform a first-stage prediction about the received image. In one embodiment, the first-stage prediction is performed based solely on luminance data without considering chrominance data to obtain faster prediction results. The machine learning model selected for the first-stage prediction may include a convolutional neural network (CNN) already trained on luminance data. The CNN model can be invoked to apply filters to detect certain features of the image (e.g., edges). CNNs allow images to be transformed into a more processable form without losing features that could contribute to accurate prediction. Generally, different convolutional layers of a CNN are applied to capture features with different levels of abstraction. For example, the first convolutional layer may be responsible for capturing low-level features (such as edges, gradient orientation, etc.). Pooling layers can further reduce the spatial size of the convolutional features. The final output of the pooling layers can then be flattened and fed into a regular neural network (e.g., a fully connected simple neural network) for classification purposes. In this respect, the prediction based on the machine learning model returns a label or kind or category for objects in the image. For example, based on the prediction that the image contains pedestrians, the image processing unit 208 returns a label indicating the pedestrian.

[0059] In action 310, it is determined whether a specific criterion has been met to decide whether image processing can end with the first-stage prediction or needs to proceed to the second-stage prediction. In one embodiment, the specific criterion is the confidence level of the prediction. In this regard, the classification in action 308 returns a confidence value indicating the probability that the predicted label is correct.

[0060] Referring again to action 310, if the confidence value is higher than a preset threshold, then in action 312 it is determined whether color information should be considered to provide a more precise or accurate identification of the object. In one embodiment, the determination of whether color information should be considered may depend on the object identified during the first step and the object's attributes. For example, in an autonomous vehicle, if the object identified in the image is a pedestrian, it may not be necessary to use chromaticity information to detect the color of the pedestrian's clothing. However, if the object identified in the image is an ambulance, it may be necessary to use chromaticity information to detect whether the ambulance's lights are flashing red and blue, which could lead to an action of yielding right-of-way to the ambulance. As yet another example, in a smart store, if the identified products have various colors, it may be necessary to use chromaticity information to detect the product's color to determine the price of the product of that particular color.

[0061] The determination of whether color information should be considered can also depend on the query provided by the application module 100 that triggers the prediction process. For example, if the query that triggers the prediction process explicitly or implicitly queries for color, then the second-stage prediction is triggered. Such a query could be a user's search request for "yellow cat" or "pink flower". Some image processing tasks may also inherently require color information. For example, predicting colors (e.g., white and yellow) may be necessary to identify lane markings to determine traffic direction, while pedestrian identification may not require this color information.

[0062] Referring again to actions 310 and 312, if it is determined in action 310 that the confidence value is lower than a preset threshold, or in action 312 that color information should be considered, then the image processing unit 208 performs a second-stage prediction in action 314. In this regard, the image processing unit 208 invokes an appropriate machine learning model from various machine learning models 212 stored in the non-volatile memory 206 of the data storage device 102. For example, if the second stage is triggered due to a low confidence value, the machine learning model retrieved from the non-volatile memory 206 via the storage controller 204 can be a model trained using RGB information, or a model trained using both luminance and chrominance information. The RGB data or luminance and chrominance data of the image can then be used to reclassify the image based on the newly retrieved machine learning model. Because more pixel data is analyzed during the second stage, a more accurate prediction of objects in the image can be obtained compared to predictions based solely on luminance data.

[0063] In some cases, the machine learning model invoked in the second stage is a model trained using only chromaticity data. This model can be adapted to classify the colors of objects detected during the first stage. For example, if the query / task provided to the image processing unit 208 is whether the image contains a yellow cat, the first-stage machine learning model using only luminance data can predict that the image contains a cat. The second-stage machine learning model can then predict the cat's color as yellow.

[0064] Then, in action 316, labels from the first prediction stage and / or the second prediction stage are returned for the classified object. In one embodiment, the label is a combination of labels generated from the first and second predictions. For example, the label from the first stage could be "cat," and the label from the second stage could be "yellow," resulting in a combined label of "yellow cat" being returned in action 316. In another embodiment, the label from the first-stage prediction is replaced with a label from the second-stage prediction, which is considered more accurate. For example, if the first-stage prediction generates the label of the object as a bicycle with a confidence value below a threshold level, and the second-stage prediction generates the label of the object as a motorcycle with a confidence value above a threshold level, then the output label in action 316 is motorcycle.

[0065] A two-stage approach that separates luminance and chrominance information and uses machine learning, as described herein, for image classification helps provide faster image search for image inspection and transcoding without compromising the search accuracy desired for a particular image processing task. Furthermore, running machine learning algorithms within the storage device itself results in more efficient and cost-effective image classification compared to traditional mechanisms that can be processed via the CPU.

[0066] It will be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or portion from another. Therefore, without departing from the spirit and scope of the inventive concept, the first element, component, region, layer, or portion discussed herein may be referred to as the second element, component, region, layer, or portion.

[0067] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the inventive concept. As used herein, the terms “substantially,” “about,” and similar terms are used as approximate terms rather than terms of degree and are intended to take into account the inherent biases of measurements or calculations that will be recognized by one of ordinary skill in the art.

[0068] As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “an” are intended to include the plural forms as well. It will also be understood that the terms “comprising” and / or “including” as used in this specification indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of…” when following a list of elements modify the entire list of elements without modifying any individual element in the list. Furthermore, in describing embodiments of the inventive concept, the use of “may” means “one or more embodiments of this disclosure.” Additionally, the term “exemplary” is intended to refer to an example or illustration. As used herein, the term “use” and its variations may be considered synonymous with the term “utilize” and its variations, respectively.

[0069] It will be understood that when an element or layer is referred to as being "on" another element or layer, "connected to", "bonded to", or "adjacent to" another element or layer, the element or layer may be directly on, directly connected to, bonded to, or adjacent to the other element or layer, or one or more intermediate elements or layers may be present. Conversely, when an element or layer is referred to as being "directly on" another element or layer, "directly connected to", "directly bonded to", or "immediately adjacent to" another element or layer, no intermediate elements or layers are present.

[0070] Any numerical range described herein is intended to include all subranges of the same numerical precision contained within the described range. For example, the range “1.0 to 10.0” is intended to include all subranges between the described minimum value 1.0 and the described maximum value 10.0 (and including both the described minimum value 1.0 and the described maximum value 10.0) (i.e., a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0 (e.g., 2.4 to 7.6)). Any maximum numerical limit described herein is intended to include all lower numerical limits contained therein, and any minimum numerical limit described in this specification is intended to include all higher numerical limits contained therein.

[0071] Although exemplary embodiments of systems and methods for in-memory image processing have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that systems and methods for knowledge distillation constructed in accordance with the principles of this disclosure may be implemented in ways different from those specifically described herein. This disclosure is also defined by the claims and their equivalents.

Claims

1. A method for processing an image, comprising: The image is received from the source by a processor embedded in the storage device; The processor identifies the brightness and chromaticity data of the received image; The processor retrieves the first machine learning model stored in the storage device; The processor applies a first machine learning model to make an initial prediction about the image based on brightness data; In response to a first determination made by the processor in association with a standard, the processor returns a first label associated with the first prediction; and In response to the processor making a second determination regarding the standard: The processor retrieves the second machine learning model stored in the storage device; The processor applies a second machine learning model to make a second prediction about the image based on color data associated with the image; as well as The processor determines the second label associated with the second prediction. The method further includes: determining a confidence value for the first prediction. Specifically, a second determination is made in response to determining that the confidence value of the first prediction is lower than a threshold, or in response to determining that the confidence value of the first prediction is higher than a threshold and determining that color information should be considered.

2. The method according to claim 1, wherein, The storage device includes a solid-state drive, and the embedded processor includes a field-programmable gate array (FPGA).

3. The method according to claim 1, wherein, The first and second machine learning models include deep neural networks.

4. The method according to claim 1, wherein, The first or second prediction determines that an object is depicted in the image, and the first or second label identifies the object.

5. The method according to claim 1, wherein, The first label identifies the object, and the second label provides color information for the identified object.

6. The method according to claim 1, further comprising: Identify an image processing task, wherein a second determination is made in response to determining that the image processing task includes color prediction.

7. The method according to claim 1, wherein, The first machine learning model was trained using images containing brightness data.

8. The method according to claim 1, wherein, The second machine learning model is trained using images containing color data.

9. The method according to claim 1, wherein, The storage device is hosted in at least one computing device in autonomous vehicles, edge data centers, and smart stores.

10. A programmable data storage system, comprising: Non-volatile memory; The memory controller is configured to control the non-volatile memory; as well as The processor is configured as follows: Receive images from the source; Identify the brightness and chromaticity data of the received image; The first machine learning model stored in the non-volatile memory is retrieved from the non-volatile memory via the memory controller; The first machine learning model is applied to make an initial prediction about the image based on brightness data without considering color data. In response to making an initial determination regarding the standard, return a first label associated with the first prediction; and In response to making a second determination regarding the standard: The second machine learning model stored in the non-volatile memory is retrieved from the non-volatile memory via the memory controller; A second machine learning model is applied to make a second prediction about the image based on the color data associated with the image; as well as Identify the second label associated with the second prediction. The processor is also configured to: determine the confidence value of the first prediction. Specifically, a second determination is made in response to determining that the confidence value of the first prediction is lower than a threshold, or in response to determining that the confidence value of the first prediction is higher than a threshold and determining that color information should be considered.

11. The system according to claim 10, wherein, The first and second machine learning models include deep neural networks.

12. The system according to claim 10, wherein, The first or second prediction determines that an object is depicted in the image, and the first or second label identifies the object.

13. The system according to claim 10, wherein, The first label identifies the object, and the second label provides color information for the identified object.

14. The system according to claim 10, wherein, The processor is also configured as follows: Identify an image processing task, wherein a second determination is made in response to determining that the image processing task includes color prediction.

15. The system according to claim 10, wherein, The first machine learning model was trained using images containing brightness data.

16. The system according to claim 10, wherein, The second machine learning model is trained using images containing color data.

17. The system according to claim 10, wherein, The programmable data storage system is hosted on a computing device in at least one of autonomous vehicles, edge data centers, and smart stores.

18. The system according to claim 10, wherein, The processor includes a field-programmable gate array (FPGA).

Images