Hyperspectral edge computing system and edge computing device

Abstract

This disclosure relates to an edge computing device, comprising: an input / output interface for receiving a hyperspectral image array, the hyperspectral image array comprising multiple hyperspectral images corresponding to multiple wavelengths; a processor coupled to the input / output interface; a memory coupled to the processor and the input / output interface for storing the hyperspectral image array, wherein the hyperspectral image array has a first data amount in the memory; and a deep learning accelerator coupled to the processor and the memory for executing at least one machine learning algorithm. The processor controls the deep learning accelerator to access the hyperspectral image array in the memory and uses at least one machine learning algorithm to extract features from the multiple hyperspectral images in the hyperspectral image array to form dimensionality-reduced hyperspectral image data, which is then stored in the memory. The dimensionality-reduced hyperspectral image data has a second data amount in the memory, and the second data amount is less than the first data amount.

Description

Technical Field

[0001] This disclosure relates to a hyperspectral edge computing system for processing hyperspectral image arrays, and to an edge computing device for processing hyperspectral image arrays. Prior Technology

[0002] Hyperspectral imagery has a wide range of applications. It can reveal the different intensities of light reflected from substances to analyze the environment. However, due to the massive amounts of data in hyperspectral images, traditional image analysis requires cloud servers or high-performance computers to run artificial intelligence (AI) models or algorithms. Therefore, in environments without network coverage or without high-performance computing devices, such as on drone platforms, hyperspectral image analysis may not be applicable, for tasks like maritime search and coastal monitoring, or deep-mountain search and rescue. Therefore, there is a need for technologies that can process hyperspectral images on lightweight computing devices while possessing the computational power and models required for hyperspectral image processing. Summary of the Invention

[0003] According to the hyperspectral edge computing technology provided by various embodiments of this disclosure, a deep learning accelerator (DLA) can be used on an edge computing device, a machine learning (ML) model can be deployed on the DLA, and the DLA can be used to accelerate the computation of the machine learning model, so that the edge computing device can have the computing power and speed to process hyperspectral image arrays with massive amounts of data.

[0004] According to a first aspect of this disclosure, a hyperspectral image edge computing system is proposed. The hyperspectral image edge computing system includes a hyperspectral sensor for capturing a hyperspectral image array. The hyperspectral image array comprises multiple hyperspectral images corresponding to multiple wavelengths. The hyperspectral image edge computing system also includes an edge computing device. The edge computing device includes an input / output interface coupled to the hyperspectral sensor and for receiving the hyperspectral image array. The edge computing device also includes a processor coupled to the input / output interface. The edge computing device also includes memory coupled to the processor and the input / output interface. The memory is used to store the hyperspectral image array, and the hyperspectral image array has a first data amount in the memory. The edge computing device also includes a deep learning accelerator coupled to the processor and the memory. The deep learning accelerator is used to execute at least one machine learning algorithm. The processor controls the deep learning accelerator to access the hyperspectral image array in the memory and uses at least one machine learning algorithm to perform feature extraction on multiple hyperspectral images in the hyperspectral image array to obtain dimensionality-reduced hyperspectral image data and store it in the memory. The reduced-dimensional hyperspectral image data has a second data volume in memory, and the second data volume is smaller than the first data volume.

[0005] According to a second aspect of this disclosure, a hyperspectral edge computing device is proposed. The hyperspectral edge computing device includes an input / output interface for receiving a hyperspectral image array. The hyperspectral image array comprises multiple hyperspectral images corresponding to multiple wavelengths. The hyperspectral edge computing device also includes a processor coupled to the input / output interface. The hyperspectral edge computing device also includes a memory coupled to the processor and the input / output interface. The memory is used to store the hyperspectral image array, and the hyperspectral image array has a first data amount in the memory. The hyperspectral edge computing device also includes a deep learning accelerator coupled to the processor and the memory. The deep learning accelerator is used to execute at least one machine learning algorithm. The processor controls the deep learning accelerator to access the hyperspectral image array in the memory and uses at least one machine learning algorithm to extract features from the multiple hyperspectral images in the hyperspectral image array to obtain dimensionality-reduced hyperspectral image data, which is then stored in the memory. The dimensionality-reduced hyperspectral image data has a second data amount in the memory, and the second data amount is less than the first data amount.

[0006] To provide a better understanding of the above and other aspects of this disclosure, specific embodiments are described below in conjunction with the accompanying drawings: Simple Explanation of the Diagram

[0007] Figure 1 illustrates a schematic diagram of an example hyperspectral edge computing system for processing hyperspectral image arrays according to various embodiments of the present disclosure. Figure 2 illustrates a functional block diagram of an example edge computing device for processing hyperspectral image arrays according to various embodiments of the present disclosure. Figure 3 illustrates a schematic diagram of a machine learning model according to various embodiments of the present disclosure, in conjunction with the edge computing device of Figure 2, for processing a hyperspectral image array. Figure 4 illustrates a flowchart of a process for deploying a machine learning model on a deep learning accelerator according to various embodiments of the present disclosure. Implementation

[0008] Figure 1 illustrates a schematic diagram of an example hyperspectral edge computing system 100 for processing a hyperspectral image array 200 according to various embodiments of the present disclosure. To quickly and instantly grasp the results generated by various applications of hyperspectral images, as shown in Figure 1, the hyperspectral edge computing system 100 provided in various embodiments of the present disclosure can process the hyperspectral image array 200 into dimensionality-reduced hyperspectral image data 300. That is, it compresses and reduces the dimensionality of the hyperspectral image array 200, which has a massive amount of data, while retaining feature values, to obtain dimensionality-reduced hyperspectral image data 300 with a smaller data volume. The dimensionality-reduced hyperspectral image data 300 with a smaller data volume can be quickly accessed on various terminal devices, such as other devices connected to the hyperspectral edge computing system 100 via wired or wireless means, such as displays or wireless network interfaces, to quickly perform subsequent interpretation of the acquired hyperspectral images. The hyperspectral image array 200 can, for example, consist of 25 hyperspectral images corresponding to 25 different wavelengths, with each hyperspectral image having a data capacity of at least 100 Mb.

[0009] Figure 2 illustrates a functional block diagram of an example edge computing device 110 for processing a hyperspectral image array 200 according to various embodiments of the present disclosure. The edge computing device 110 includes a processor 111, memory 112, a DLA (deep learning accelerator) 113, a GPU 114, a storage device 115, and an input / output interface 116, all of which are coupled via a bus / interface 117.

[0010] Processor 111 may include one or more cores, and processor 111 may be any combination of hardware units capable of executing programmable instructions, microprocessors, signal processors, AI processors, and other similar devices. Processor 111 may optionally include one or more internal registers, one or more caches, and / or one or more internal memories.

[0011] GPU 114 can be a processing unit capable of accelerating highly parallel processing, such as graphics processing, signal processing, and / or AI processing. Similar to processor 111, GPU 114 may selectively include one or more internal registers, one or more caches, and / or one or more internal memories. In some embodiments, GPU 114 may also support accelerating ML model computations for processing hyperspectral image array 200.

[0012] Memory 112 may include one or more memory devices or memory arrays for storing instructions and / or data, such as hyperspectral image array 200 or dimensionality-reduced hyperspectral image data 300. The amount of data that memory 112 can store is greater than the memory of processor 111 and / or GPU 114.

[0013] Storage device 115 may include one or more storage elements, such as an SSD based on a flash-based storage element or an HHD based on a rotating magnetic and / or optical non-volatile storage element (e.g., a disk), which can be used for storing instructions and / or data, such as a hyperspectral image array 200 or reduced-dimensional hyperspectral image data 300. Compared to HDD, SSDs have lower access latency but can store less data.

[0014] Input / output interface 116 includes components that can connect any combination of processor 111, memory 112, DLA 113, GPU 114, and / or storage device 115 to external components of edge computing device 110, such as hyperspectral sensor 120. Examples of external components include a large number of storage devices, local or wide area networks (e.g., the Internet), human-machine interface components (e.g., keyboard, mouse, and / or display), and other components that provide the ability to extend and / or enhance capabilities not provided by edge computing device 110, such as the acquisition of hyperspectral sensor 120 for hyperspectral image array 200 outside of edge computing device 110.

[0015] Bus / interface 117 enables communication between components coupled thereto (e.g., processor 111, memory 112, DLA 113, GPU 114, storage device 115, and / or input / output interface 116). Bus / interface 117 includes one or more serial and / or parallel communication channels in various ways, as well as selective protocol conversion and / or adaptation capabilities to enhance communication between components coupled thereto.

[0016] DLA 113 may have a combination of circuits or processing units that provide dedicated or general-purpose acceleration functions for specific algorithms or AI models, such as deep learning algorithms / models or machine learning (ML) algorithms / models. In some embodiments, DLA 113 may include adder-multiplier digital circuitry 118 and comparator circuitry 119 to deploy and accelerate the ML algorithms / models described herein for processing the hyperspectral image array 200, enabling the edge computing device 110 to process the hyperspectral image array 200.

[0017] Depending on the application requirements, additional partitioning of the components shown in the diagram, coupling between components, component capabilities and / or capacity, and other additional components may be considered.

[0018] As shown in Figure 2, the edge computing device 110 can receive a hyperspectral image array 200 containing multiple hyperspectral images corresponding to multiple wavelengths from the hyperspectral sensor 120 through the input / output interface 116, and can store the hyperspectral image array 200 in memory 112 and / or storage device 115, for example, through instructions from the processor 111. The edge computing device 110 can also control the DLA 113 to access the hyperspectral image array 200 through the processor 111, and use an ML model to extract features from the multiple hyperspectral images in the hyperspectral image array 200, so as to compress and reduce the dimensionality of the hyperspectral image array 200 with a large amount of data, so as to obtain dimensionality-reduced hyperspectral image data 300 with a smaller amount of data. The reduced-dimensional hyperspectral image data 300 can be stored in memory 112 and / or storage device 115, or output to other external devices coupled to input / output interface 116 via input / output interface 116. For example, the reduced-dimensional hyperspectral image data 300 can be displayed on an external display (in conjunction with GPU 114) coupled to input / output interface 116, or transmitted to other terminal devices via a wired / wireless network terminal coupled to input / output interface 116. The reduced-dimensional hyperspectral image data 300, with its smaller data size, is beneficial for backend processing and / or transmission to other external devices.

[0019] Figure 3 illustrates a schematic diagram of an ML model 130 according to various embodiments of the present disclosure, in conjunction with the edge computing device 110 of Figure 2, to process the hyperspectral image array 200. As discussed above, the edge computing device 110 can receive and store the hyperspectral image array 200 having multiple hyperspectral images corresponding to multiple wavelengths through an input / output interface 116, for example from the hyperspectral sensor 120 of Figures 2 and 1. In this example, the hyperspectral image array 200 is stored in memory 112.

[0020] Next, the processor 111 can control the DLA 113 to execute the ML model 130. In some embodiments, the ML model 130 may be deployed on the DLA 113. When the ML model 130 is executed on the DLA 113, it can access hyperspectral image array 200 data stored in memory 112 as feature input. Then, the addition and multiplication digital circuitry 118 of the DLA 113 can support and accelerate the ML model 130 to perform distance calculations, such as Euclidean distance calculations, on each hyperspectral image in the hyperspectral image array 200, which serves as feature input data.

[0021] Next, the comparator circuit 119 of DLA 113 can support and accelerate the data sampling of the distance calculation results of the features in each hyperspectral image by the ML model 130 to obtain the feature output result, that is, the dimension-reduced hyperspectral image data 300 with a smaller amount of data.

[0022] After obtaining the reduced-dimensional hyperspectral image data 300, as discussed above, it can be stored in memory 112 and / or storage device 115, or output to other external devices coupled to the input / output interface 116 via the input / output interface 116. For example, the reduced-dimensional hyperspectral image data 300 can be displayed on an external display coupled to the input / output interface 116 (e.g., in conjunction with the GPU 114 in Figure 2), or the reduced-dimensional hyperspectral image data 300 can be sent to other terminal devices via a wired / wireless network terminal coupled to the input / output interface 116. In some embodiments, the ML model 130 can be adjusted so that the data volume of the reduced-dimensional hyperspectral image data 300 matches the bandwidth of the wireless network terminal coupled to the input / output interface 116, thereby facilitating wireless data transmission. In some implementations, DLA 113 may be Mediatek's MDLA product, and the ML model 130 deployed on DLA 113 may be in an MDLA-compliant format.

[0023] In some embodiments, DLA 113 may further include filter circuitry (not shown) and analog-to-digital converter / digital-to-analog converter (ADC / DAC) circuitry (not shown) for deploying other deep learning models. For example, the addition / multiplication digit circuitry 118 of DLA 113 can be used for synaptic functions in convolutional layers or fully connected layers of the deep learning model; the filter circuitry can be used for normalization functions in the deep learning model; the ADC / DAC circuitry can be used for activation functions in the deep learning model; and the comparator circuitry 119 of DLA 113 can be used for sampling data from the results of the deep learning model.

[0024] From the above description, it can be understood that the edge computing device and DLA technology provided according to the various embodiments disclosed herein can accelerate the computation of ML models / algorithms using underlying DLA-related circuitry. The following will describe, with reference to Figure 4, the procedure for deploying the ML model on the DLA.

[0025] Figure 4 illustrates a flowchart of procedure 400 for deploying a machine learning (ML) model on a deep learning accelerator (DLA) according to various embodiments of this disclosure. In step S410, model weights in PyTorch format are input. PyTorch is a machine learning library based on the Torch library.

[0026] In step S420, the model weights are converted to ONNX format. ONNX (Open Neural Network Exchange) is an open-source artificial intelligence ecosystem.

[0027] In step S430, the model weights in ONNX format are converted to TF format. TF (TensorFlow) is a software library for machine learning and artificial intelligence, mainly used for training and inference of neural networks.

[0028] In step S440, the ML model is quantized using model weights in TF format.

[0029] In step S450, the quantized ML model in TF Lite format is output. TF (TensorFlow) Lite is a tool that helps developers run AI models on mobile devices, embedded devices, and IoT devices, and to run ML models on terminal devices, such as edge computing devices, such as ML model 130 in Figure 3.

[0030] In step S460, the quantized ML model in TF lite format is converted into DLA format for deployment on DLA (e.g., DLA 113 in Figures 2 and 3), and DLA acceleration is used, for example, converting the ML model into MDLA format that conforms to the aforementioned MDLA product, but this is not a limitation.

[0031] In step S470, a quantized ML model in DLA format is used, for example, in the edge computing device 110 of the first to third images, to infer (e.g., compress or reduce the dimension) the hyperspectral image array, for example, the hyperspectral image array 200 of the first to third images, to generate dimension-reduced hyperspectral image data, for example, dimension-reduced hyperspectral image data 300 of the first to third images.

[0032] As explained above, the edge computing techniques for processing hyperspectral images provided by various embodiments of this disclosure allow the use of lightweight edge computing devices to replace conventional mainframes or cloud servers, facilitating applications in offline environments. This means data processing is performed directly on the local device (edge ​​computing device), reducing network dependence. For example, a hyperspectral image edge computing system can be mounted on a gimbal mounted on an unmanned vehicle to directly process hyperspectral image arrays acquired in the environment. Similarly, an ML model architecture can be integrated into the edge computing device, and DLA can be used to accelerate the ML model to infer (e.g., compress or reduce) massive amounts of data from processed hyperspectral images, and the results can yield dimensionality-reduced hyperspectral image data with a smaller data volume.

[0033] The technical terms used in this specification are based on common terminology in the field. Where this specification provides explanations or definitions for certain terms, the interpretation of those terms shall be based on the explanations or definitions provided in this specification. Each of the embodiments disclosed herein has one or more technical features. Where feasible, those skilled in the art may selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

[0034] The above disclosure provides different features for implementing some embodiments or examples of this disclosure. Specific examples of components and configurations described above (e.g., mentioned values ​​or names) are used to simplify / illustrate some embodiments of this disclosure. Of course, these components and configurations are merely examples and are not intended to be limiting. Furthermore, reference numerals and / or letters may be repeated in various instances of some embodiments of this disclosure. This repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.

[0035] Although this disclosure has been presented above with reference to embodiments, it is not intended to limit the scope of this disclosure. Those skilled in the art to which this disclosure pertains can make various modifications and refinements without departing from the spirit and scope of this disclosure. Therefore, the scope of protection of this disclosure shall be determined by the appended claims.

[0036] 100: Hyperspectral Edge Computing System 110: Edge computing device 111: Processor 112: Memory 113:DLA 114: GPU 115: Storage device 116: Input / Output Interface 116 117: Bus / Interface 118: Addition and Multiplication Digital Circuits 119: Comparator Circuit 120: Hyperspectral Sensor 130: ML model 200: Hyperspectral Image Array 300: Dimensionally Reduced Hyperspectral Image Data 400: Program S410~S470: Steps

Claims

1. A hyperspectral image edge computing system, comprising: a hyperspectral sensor for capturing a hyperspectral image array, the hyperspectral image array comprising a plurality of hyperspectral images corresponding to a plurality of wavelengths; and an edge computing device, comprising: An input / output interface is coupled to the hyperspectral sensor, which is used to receive the hyperspectral image array; A processor coupled to the input / output interface; a memory coupled to the processor and the input / output interface, the memory storing the hyperspectral image array, the hyperspectral image array having a first data amount in the memory; and a deep learning accelerator (DLA) coupled to the processor and the memory, the deep learning accelerator executing at least one machine learning algorithm, wherein the processor controls the deep learning accelerator to access the hyperspectral image array in the memory, and uses the at least one machine learning algorithm to extract features from the hyperspectral images in the hyperspectral image array to obtain a dimension-reduced hyperspectral image data and store it in the memory, wherein the dimension-reduced hyperspectral image data has a second data amount in the memory, the second data amount being less than the first data amount, wherein the deep learning accelerator includes an addition / multiplication digital circuit and a comparator circuit, the addition / multiplication digital circuit and the comparator circuit being used to accelerate and support the at least one machine learning algorithm.

2. The hyperspectral image edge computing system as described in claim 1, wherein the at least one machine learning algorithm accesses the hyperspectral image array in the memory using the deep learning accelerator as a feature input, wherein the at least one machine learning algorithm uses the addition and multiplication digital circuitry of the deep learning accelerator to perform distance calculations on a plurality of features in each of the hyperspectral images in the hyperspectral image array as the feature input, wherein the at least one machine learning algorithm uses the comparator circuitry of the deep learning accelerator to sample the result of the distance calculation on the features in each of the hyperspectral images to obtain the dimension-reduced hyperspectral image data.

3. The hyperspectral image edge computing system as described in claim 1, wherein the input / output interface is coupled to a wireless network interface having a bandwidth, wherein the at least one machine learning algorithm is adjustable to make the second data amount of the dimensionality-reduced hyperspectral image data conform to the bandwidth capability of the wireless network interface.

4. The hyperspectral image edge computing system as described in claim 1 further includes a GPU coupled to the processor and the input / output interface, wherein the input / output interface is coupled to a display, wherein the dimension-reduced hyperspectral image data can be processed by the GPU and output to the display for display of the dimension-reduced hyperspectral image data.

5. The hyperspectral image edge computing system as described in claim 1, wherein the deep learning accelerator executes the at least one machine learning algorithm by: inputting a PyTorch format model weight of at least one machine learning model of the at least one machine learning algorithm; converting the PyTorch format model weight to an ONNX format model weight; converting the ONNX format model weight to a TF format model weight; quantizing the at least one machine learning model using the TF format model weight to output at least one TF lite format quantized machine learning model; and converting the TF lite format quantized machine learning model into the format of the deep learning accelerator to execute the at least one machine learning algorithm on the deep learning accelerator.

6. An edge computing device, comprising: An input / output interface is provided for receiving a hyperspectral image array, the hyperspectral image array comprising a plurality of hyperspectral images corresponding to a plurality of wavelengths; a processor is coupled to the input / output interface; a memory is coupled to the processor and the input / output interface, the memory being used to store the hyperspectral image array, and the hyperspectral image array having a first data amount in the memory; and a deep learning accelerator is coupled to the processor and the memory, the deep learning accelerator being used to execute at least one machine learning algorithm, wherein the processor controls the deep learning accelerator to access the hyperspectral image array in the memory, and uses the at least one machine learning algorithm to extract features from the hyperspectral images in the hyperspectral image array to obtain dimensionality-reduced hyperspectral image data and store it in the memory, wherein the dimensionality-reduced hyperspectral image data has a second data amount in the memory, and the second data amount is less than the first data amount. The deep learning accelerator includes an addition and multiplication digital circuit and a comparator circuit, which can be used to accelerate and support the at least one machine learning algorithm.

7. The edge computing apparatus as claimed in claim 6, wherein the at least one machine learning algorithm accesses the hyperspectral image array in the memory as a feature input via the deep learning accelerator, wherein the at least one machine learning algorithm performs distance calculations on a plurality of features in each of the hyperspectral images in the hyperspectral image array as the feature input via the addition and multiplication digital circuitry of the deep learning accelerator, wherein the at least one machine learning algorithm samples the result of the distance calculation on the features in each of the hyperspectral images via the comparator circuitry of the deep learning accelerator to obtain the dimension-reduced hyperspectral image data.

8. The edge computing device as claimed in claim 6, wherein the input / output interface is coupled to a wireless network interface having a bandwidth, wherein the at least one machine learning algorithm is adjustable to make the second data amount of the dimensionality-reduced hyperspectral image data conform to the bandwidth capability of the wireless network interface.

9. The edge computing device as described in claim 6 further includes a GPU coupled to the processor and the input / output interface, wherein the input / output interface is coupled to a display, wherein the dimension-reduced hyperspectral image data can be processed by the GPU and output to the display for display of the dimension-reduced hyperspectral image data.

10. The edge computing apparatus as claimed in claim 6, wherein the deep learning accelerator executes the at least one machine learning algorithm by: inputting a PyTorch format model weight of at least one machine learning model of the at least one machine learning algorithm; converting the PyTorch format model weight to an ONNX format model weight; converting the ONNX format model weight to a TF format model weight; quantizing the at least one machine learning model using the TF format model weight to output at least one TF lite format quantized machine learning model; and converting the TF lite format quantized machine learning model into the format of the deep learning accelerator to execute the at least one machine learning algorithm on the deep learning accelerator.

Images