A hyperspectral sensing and processing system

By adopting a multiplexed reconstructible network acceleration framework in the hyperspectral perception processing system, the multiplexing of full pipeline parallel acceleration and convolution operations is achieved, and the problems of expensive equipment and high computing resources are solved, and the problems of traditional hyperspectral imaging technology are improved. Processing speed and portability are suitable for multi-field applications.

CN118350982BActive Publication Date: 2024-11-19BEIJING INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410781451.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-19
Estimated Expiration
2044-06-18

AI Technical Summary

Technical Problem

Traditional hyperspectral imaging technology equipment is expensive and complex to operate. Deep learning models consume high computing resources in hyperspectral image processing, making it difficult to achieve portable deployment.

Method used

Design a hyperspectral perception processing system, use programmable logic to build a multiplexed reconfigurable network acceleration framework, realize parallel acceleration operations of all pipelines, and map convolution operations of different dimensions and matrix multiplication operations to multiplex , reduce resource overhead.

Benefits of technology

While reducing the power consumption and volume of hyperspectral sensing processing systems, it improves processing speed, which is suitable for portable deployment and realizes efficient spectral image reconstruction and recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118350982B_ABST
    Figure CN118350982B_ABST
Patent Text Reader

Abstract

The present disclosure relates to the field of computer vision technology, and in particular to a hyperspectral perception processing system. The system includes: a programmable logic end, in which a multiplexed reconfigurable network acceleration framework is built; the multiplexed reconfigurable network acceleration framework is used to deploy a neural network, and implement full-pipeline parallel acceleration operations in the process of deploying the neural network, and uniformly map convolution operations and matrix multiplication operations of different dimensions to implement convolution operations and matrix multiplication operations in a multiplexed manner. The present disclosure using the above scheme can reduce the power consumption and volume of the hyperspectral perception processing system while improving the processing speed of the hyperspectral perception processing system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a hyperspectral perception processing system. Background Art

[0002] Hyperspectral imaging technology, as a powerful scientific tool, has attracted widespread attention from the scientific research and industrial communities. Its uniqueness lies in its ability to simultaneously capture the spatial information and detailed spectral information of the target object. Each pixel not only records the color information, but also the spectral information of the object. This makes hyperspectral imaging show an unparalleled advantage in accurately identifying and analyzing the composition of materials, providing important support for many fields such as biomedicine, environmental monitoring, cultural heritage protection, intelligent transportation system development, and military reconnaissance.

[0003] Nevertheless, the equipment of traditional hyperspectral imaging technology is usually expensive and complicated to operate, which is not conducive to the popularization and daily application of technology. In order to overcome these problems, hyperspectral image reconstruction technology based on deep learning has emerged. It optimizes the acquisition and processing process of hyperspectral images through algorithms, greatly reducing costs and improving efficiency. This technology can reconstruct complete hyperspectral data from limited spectral information, which is of great significance for the rapid identification of substances.

[0004] However, deep learning models, especially in the field of hyperspectral image processing, often require a lot of computing resources. Although the Graphics Processing Unit (GPU) has powerful parallel processing capabilities and can greatly speed up the training and reasoning of deep learning models, its high power consumption and large size make GPU unsuitable for portable deployment. On the other hand, although the Central Processing Unit (CPU) is suitable for portable deployment, it takes a long time to process these complex models and has a slow processing speed. Summary of the invention

[0005] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, a first objective of the present disclosure is to propose a hyperspectral perception processing system to improve the processing speed of the hyperspectral perception processing system while reducing the power consumption and volume of the hyperspectral perception processing system.

[0007] The second objective of the present disclosure is to provide a hyperspectral sensing processing chip.

[0008] To achieve the above objectives, the first aspect of the present disclosure provides a hyperspectral sensing and processing system, including:

[0009] A programmable logic end, wherein a multiplexed reconfigurable network acceleration framework is built in the programmable logic end;

[0010] The multiplexed reconfigurable network acceleration framework is used to deploy a neural network, and implement full-pipeline parallel acceleration operations during the deployment of the neural network, and uniformly map convolution operations and matrix multiplication operations of different dimensions to multiplex and implement convolution operations and matrix multiplication operations.

[0011] Optionally, the hyperspectral perception processing system further includes:

[0012] An image sensor is used to encode an input image to obtain an image encoding signal;

[0013] External memory, used to store network parameters and output results;

[0014] Internal memory, used to implement buffer data interaction;

[0015] The core processor end is used to implement floating point operations and thread scheduling during data interaction between the image sensor, the external memory, the core processor end and the programmable logic end.

[0016] Optionally, the programmable logic end further includes a global control module, and the multiplexed reconfigurable network acceleration framework includes a transposition module, an input cache, a parameter cache, a result cache, an output cache, a convolution and full connection module, and a neural network calculation module; wherein,

[0017] The global control module is used to receive the instruction operation and input data input by the core processor end, transmit the input data to the input buffer and the parameter buffer, and read the output result from the output buffer;

[0018] The transposition module is used to perform a matrix transposition operation on the cache data input by the input cache, obtain the matrix-transposed data and input it into the parameter cache;

[0019] The convolution and fully connected module is used to uniformly map convolution operations and matrix multiplication operations of different dimensions based on the cache data input by the input cache and the cache data input by the parameter cache to multiplex the convolution operations and the matrix multiplication operations, and obtain and input the calculation results into the result cache;

[0020] The neural network calculation module is used to perform neural network calculation based on the cache data input by the result cache, obtain and input the output result into the output cache.

[0021] Optionally, the neural network calculation module includes at least one of the following sub-modules:

[0022] Pooling module, used to implement 2D pooling operations in neural networks;

[0023] Upsampling module, used to implement image upsampling operations in neural networks;

[0024] Downsampling module, used to implement image downsampling operations in neural networks;

[0025] GELU module, used to implement nonlinear GELU activation operations in neural networks;

[0026] Softmax module, used to implement nonlinear Softmax activation operation in neural networks;

[0027] The layer normalization module is used to implement the layer normalization operation in the neural network.

[0028] Optionally, the calculation array in the convolution and fully connected module is composed of a binary tree, the convolution block parameters corresponding to the convolution operation include a channel input parameter and a channel output parameter, and the matrix block parameters corresponding to the matrix multiplication operation include a width input parameter and a width output parameter;

[0029] The calculation array input channels corresponding to the calculation array correspond to the channel input parameters and the width input parameters respectively, and the calculation array output channels corresponding to the calculation array correspond to the channel output parameters and the width output parameters respectively, so as to multiplex the convolution operation and the matrix multiplication operation.

[0030] Optionally, the core processor side includes:

[0031] An image sensor control module, used to control data interaction between the image processing module and the image sensor;

[0032] The image processing module is used to perform image processing on the image coding signal to obtain and input the processed image coding signal into the data and thread scheduling module;

[0033] An external memory control module, used for controlling data interaction between the external memory and the data and thread scheduling module;

[0034] An internal memory control module, used for controlling data interaction between the internal memory and the data and thread scheduling module;

[0035] The Contact module and the Add module are used to perform data interaction with the data and thread scheduling module to realize the residual operation of the neural network;

[0036] The data and thread scheduling module is used to implement data transmission management and thread scheduling during the process of data interaction between the image processing module, the external memory, the internal memory, the Contact module, the Add module and the programmable logic end.

[0037] Optionally, the neural network comprises a spectral reconstruction network, wherein,

[0038] The image processing module is further used to perform spectral reconstruction image preprocessing on the image coding signal to obtain the image coding signal after spectral reconstruction image preprocessing;

[0039] The multiplexed reconfigurable network acceleration framework is also used to input the image encoding signal after the spectral reconstruction image preprocessing into the spectral reconstruction network, read the network parameters, control the spectral reconstruction network to perform forward reasoning to obtain a high-channel hyperspectral image result, and input the high-channel hyperspectral image result as an output result into the external memory.

[0040] Optionally, the neural network includes a spectral reconstruction network and a target recognition network, wherein:

[0041] The image processing module is further used to perform target recognition image preprocessing on the image coding signal to obtain an image coding signal after target recognition image preprocessing;

[0042] The multiplexed reconfigurable network acceleration framework is also used to input the image encoding signal after the target recognition image preprocessing into the target recognition network, read the network parameters, and control the target recognition network to perform forward reasoning to obtain a recognition result;

[0043] The image processing module is further used to segment the recognition result into a main object and determine the pixel size of the main object obtained after segmentation;

[0044] The multiplexed reconfigurable network acceleration framework is also used to input the main target and the pixel size into the spectral reconstruction network, read the network parameters, control the spectral reconstruction network to perform forward reasoning to obtain high-channel key target hyperspectral information results, and input the high-channel key target hyperspectral information results as output results into the external memory.

[0045] Optionally, the multiplexed reconfigurable network acceleration framework is used to control the spectral reconstruction network to perform forward reasoning, and when a fully connected operation is used to implement a multi-head self-attention operation in the spectral reconstruction network, sparse matrix multiplication is used for optimization.

[0046] Optionally, the hyperspectral sensing and processing system adopts a double buffer optimization strategy during operation.

[0047] To achieve the above-mentioned purpose, a second aspect of the present disclosure provides a hyperspectral sensing and processing chip, including: a hyperspectral sensing and processing system as shown in any one of the above-mentioned first aspects.

[0048] In summary, the hyperspectral perception processing system and chip provided by the present disclosure can realize full-pipeline parallel acceleration operations and complete the accelerated calculation of neural networks by utilizing the powerful parallel processing capabilities provided by the programmable logic array. Secondly, the multiplexed reconfigurable network acceleration framework can reduce the resource overhead of the programmable logic end by uniformly mapping convolution operations and matrix multiplication operations of different dimensions to multiplex convolution operations and matrix multiplication operations. Secondly, the multiplexed reconfigurable network acceleration framework is small in size and suitable for portable deployment. Therefore, the present disclosure can improve the processing speed of the hyperspectral perception processing system while reducing the power consumption and volume of the hyperspectral perception processing system.

[0049] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0051] Figure 1 A schematic diagram of the structure of a hyperspectral perception processing system provided by an embodiment of the present disclosure;

[0052] Figure 2 A workflow diagram of a hyperspectral perception processing system provided by an embodiment of the present disclosure;

[0053] Figure 3 A schematic diagram of the structure of a spectrum reconstruction network provided by an embodiment of the present disclosure;

[0054] Figure 4 A schematic diagram of the principle of a multiplexed reconfigurable network acceleration framework provided by an embodiment of the present disclosure;

[0055] Figure 5 A schematic diagram of the principle of a double buffering strategy provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0057] The present disclosure is described in detail below with reference to specific embodiments.

[0058] Figure 1 The structure diagram of a hyperspectral sensing and processing system provided by the embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the hyperspectral sensing and processing system comprises:

[0059] Programmable Logic (PL) side, where a reusable and reconfigurable network acceleration framework is built;

[0060] A reusable and reconfigurable network acceleration framework is used to deploy neural networks and implement full-pipeline parallel acceleration operations during the deployment of neural networks. Convolution operations and matrix multiplication operations of different dimensions are uniformly mapped to implement convolution operations and matrix multiplication operations in a multiplexed manner.

[0061] According to some embodiments, the module parameters of the multiplexed reconfigurable network acceleration framework can be flexibly adjusted, for example, it can support flexible reconstruction of convolution block dimension, convolution kernel size, number of input and output channels, matrix scale and parallel dimension. It has good applicability to different processors and neural network models such as YOLO and Transformer.

[0062] It is easy to understand that by utilizing the powerful parallel processing capabilities provided by the programmable logic array, full-pipeline parallel acceleration operations can be implemented to complete the accelerated calculation of the neural network. Secondly, the reusable reconfigurable network acceleration framework can reduce the resource overhead of the programmable logic end by uniformly mapping convolution operations and matrix multiplication operations of different dimensions to multiplex convolution operations and matrix multiplication operations. Secondly, the reusable reconfigurable network acceleration framework is small in size and suitable for portable deployment.

[0063] Alternatively, if Figure 1 As shown, the hyperspectral sensing and processing system also includes an image sensor, a core processor (Processing System, PS) end, an external memory and an internal memory; wherein,

[0064] The image sensor is connected to a first terminal of the core processor, the external memory is connected to a second terminal of the core processor, the internal memory is connected to a third terminal of the core processor, and the fourth terminal of the core processor is connected to the programmable logic terminal.

[0065] According to some embodiments, the image sensor may be used to encode an input image to obtain an image encoding signal. The pixels of the input image may be set according to the actual application scenario. For example, the pixels of the input image may be 512×512.

[0066] In some embodiments, a broadband mask array may be integrated in front of the photosensitive array of the image sensor, and the image sensor may encode the input image through the broadband mask array.

[0067] According to some embodiments, external memory may be used to store network parameters and output results.

[0068] In some embodiments, the network parameters include parameters related to network design and parameters related to the training process.

[0069] Among them, parameters related to network design include but are not limited to the number of network layers, the types and construction order of different layers, parameter settings of hidden layer neurons, selection of LOSS, regularization parameters, etc.

[0070] Among them, parameters related to the training process include but are not limited to network weight initialization method, learning rate, number of iterations, size of small batch data, etc.

[0071] According to some embodiments, internal memory may be used to implement cache data interaction.

[0072] According to some embodiments, the core processor side may be used to implement floating point operations and thread scheduling during data interaction between the image sensor, the external memory, the core processor side, and the programmable logic side.

[0073] In some embodiments, data may be transmitted between the programmable logic end and the core processor end via an AXI-HP interface, for example.

[0074] Alternatively, if Figure 1 As shown, the core processor end includes an image sensor control module, an image processing module, an external memory control module, an internal memory control module, a Contact module, an Add module, and a data and thread scheduling module; wherein,

[0075] A first end of the image sensor control module is connected to the image sensor, a second end of the image sensor control module is connected to a first end of the image processing module, and the image sensor control module is used to control data interaction between the image processing module and the image sensor;

[0076] The second end of the image processing module is connected to the first end of the data and thread scheduling module, and the image processing module is used to perform image processing on the image coding signal, obtain and input the processed image coding signal into the data and thread scheduling module;

[0077] A first end of the external memory control module is connected to the external memory, a second end of the external memory control module is connected to a second end of the data and thread scheduling module, and the external memory control module is used to control data interaction between the external memory and the data and thread scheduling module;

[0078] A first end of the internal memory control module is connected to the internal memory, a second end of the internal memory control module is connected to a third end of the data and thread scheduling module, and the internal memory control module is used to control data interaction between the internal memory and the data and thread scheduling module;

[0079] The Contact module is connected to the fourth terminal of the data and thread scheduling module, and the Add module is connected to the fifth terminal of the thread scheduling module. Both the Contact module and the Add module are used to perform data interaction with the data and thread scheduling module to realize the residual operation of the neural network;

[0080] The sixth end of the data and thread scheduling module is connected to the programmable logic end, and the data and thread scheduling module is used to realize data transmission management and thread scheduling during data interaction between the image processing module, the external memory, the internal memory, the Contact module, the Add module and the programmable logic end.

[0081] According to some embodiments, the image sensor control module may adopt a MIPI interface control module, for example.

[0082] In some embodiments, image processing includes but is not limited to preprocessing, subject target segmentation, etc.

[0083] Alternatively, if Figure 1 As shown, the programmable logic end also includes a global control module, and the reusable reconfigurable network acceleration framework includes a transposition module, an input cache, a parameter cache, a result cache, an output cache, a convolution and full connection module, and a neural network calculation module; wherein,

[0084] A first end of the global control module is connected to a fourth end of the core processor end, a second end of the global control module is connected to a first end of the input cache and a first end of the parameter cache, a third end of the global control module is connected to a second end of the output cache, and the global control module is used to receive instruction operations and input data input by the core processor end, transmit the input data to the input cache and the parameter cache, and read output results from the output cache;

[0085] A first end of the transposition module is connected to a second end of the input buffer, a second end of the transposition module is connected to a second end of the parameter buffer, and the transposition module is used to perform a matrix transposition operation on the buffer data input by the input buffer, and obtain and input the matrix transposed data into the parameter buffer;

[0086] The first end of the convolution and fully connected module is connected to the third end of the input buffer, the second end of the convolution and fully connected module is connected to the third end of the parameter buffer, the third end of the convolution and fully connected module is connected to the first end of the result buffer, the convolution and fully connected module, the convolution and fully connected module is used to uniformly map convolution operations and matrix multiplication operations of different dimensions based on the cache data input by the input buffer and the cache data input by the parameter buffer to multiplex the convolution operations and the matrix multiplication operations, and obtain and input the calculation results into the result buffer;

[0087] The first end of the neural network calculation module is connected to the second end of the result cache, and the second end of the neural network calculation module is connected to the first end of the output cache. The neural network calculation module is used to perform neural network calculation based on the cache data input by the result cache, and obtain and input the output result into the output cache.

[0088] According to some embodiments, the first end of the input cache and the first end of the parameter cache are also connected to the second end of the result cache, and are used to repeatedly perform multiplexing to implement convolution operations and matrix multiplication operations based on the cache data input by the result cache until a calculation result that meets the output condition is obtained, so that the neural network calculation module performs neural network calculations based on the calculation results that meet the output condition.

[0089] In some embodiments, the input data input to the core processor end includes but is not limited to data input to the core processor end by the image processing module, the external memory, and the internal memory through the data and thread scheduling module.

[0090] In some embodiments, the Contact module and the Add module can receive data input by the global control module through the data and thread scheduling module, and use the logic processing advantages of the core processor end to efficiently complete the residual operation of the neural network.

[0091] According to some embodiments, Figure 1 As shown, the neural network calculation module includes at least one of the following submodules:

[0092] Pooling module, used to implement 2D pooling operations in neural networks;

[0093] Upsampling module, used to implement image upsampling operations in neural networks;

[0094] Downsampling module, used to implement image downsampling operations in neural networks;

[0095] GELU module, used to implement nonlinear GELU activation operations in neural networks;

[0096] Softmax module, used to implement nonlinear Softmax activation operation in neural networks;

[0097] The layer normalization module is used to implement the layer normalization operation in the neural network.

[0098] In some embodiments, each module on the programmable logic side can achieve full pipeline parallel acceleration.

[0099] Optionally, the neural network deployed in the reusable reconfigurable network acceleration framework includes but is not limited to at least one of the following:

[0100] Spectral reconstruction network;

[0101] Object recognition network.

[0102] According to some embodiments, the spectral reconstruction network and the target recognition network may be, for example, neural networks independently designed and trained, and the network parameters with the highest accuracy and the best reconstruction effect are saved to an external memory by a PC through a data reading and writing device.

[0103] In some embodiments, the neural network can be deployed based on a reusable reconfigurable network acceleration framework using network function modules according to the network structure of the neural network independently designed and trained.

[0104] According to some embodiments, when the neural network includes a spectral reconstruction network, the image processing module can perform spectral reconstruction image preprocessing on the image coding signal to obtain an image coding signal after spectral reconstruction image preprocessing; then, the multiplexed reconfigurable network acceleration framework can input the image coding signal after spectral reconstruction image preprocessing into the spectral reconstruction network, read the network parameters from the external memory, control the spectral reconstruction network to perform forward reasoning to obtain a high-channel hyperspectral image result, and input the high-channel hyperspectral image result as an output result into the external memory.

[0105] In some embodiments, the high-channel hyperspectral image result may be input into an external memory in the form of a binary file.

[0106] According to some embodiments, in the case where the neural network includes a spectral reconstruction network and a target recognition network, first, the image processing module can perform target recognition image preprocessing on the image coding signal to obtain the image coding signal after the target recognition image preprocessing. Then, the multiplexed reconfigurable network acceleration framework can input the image coding signal after the target recognition image preprocessing into the target recognition network, read the network parameters from the external memory, and control the target recognition network to perform forward reasoning to obtain the recognition result. Then, the image processing module can perform main target segmentation on the recognition result, and determine the pixel size of the main target obtained after segmentation. Finally, the multiplexed reconfigurable network acceleration framework can input the main target and pixel size into the spectral reconstruction network, read the network parameters from the external memory, control the spectral reconstruction network to perform forward reasoning to obtain the high-channel key target hyperspectral information result, and input the high-channel key target hyperspectral information result as the output result into the external memory.

[0107] In some embodiments, the high-channel key target hyperspectral information result may be input into an external memory in the form of a binary file.

[0108] Take a scenario as an example. Figure 2 The following is a workflow diagram of a hyperspectral sensing and processing system provided by an embodiment of the present disclosure. Figure 2 As shown, the following steps may be specifically included:

[0109] Step 1: Independently design and train the spectral reconstruction network and target recognition network, and save the network parameters with the highest accuracy and the best reconstruction effect to the external memory through the PC data reading and writing device.

[0110] Step 2: Build a reusable reconfigurable network acceleration framework in the programmable logic end, and use the network function module to deploy the target recognition network and spectrum reconstruction network according to the network structure designed in step 1.

[0111] Step 3: The image sensor inputs the image.

[0112] Option 1 in step 4: Use the image processing module to preprocess the spectral reconstruction image, input the spectral reconstruction network after the processing is completed, read the network parameters from the external memory, and perform forward reasoning.

[0113] Option 1 in step 5: Output the multi-channel hyperspectral image result output by the spectral reconstruction network to an external storage in the form of a binary file.

[0114] Option 2 in step 4: Use the image processing module to preprocess the target recognition image. After the processing is completed, input it into the target recognition network, read the network parameters from the external memory, perform forward reasoning, and obtain the recognition result.

[0115] Option 2 in step 5: Use the image processing module to segment the main target of the recognition result after the target recognition network, calculate the pixel size of the main target, input the spectral reconstruction network, read the network parameters from the external memory, and perform forward reasoning.

[0116] Step 6: Output the multi-channel key target hyperspectral information results output by the spectral reconstruction network to the external storage in the form of a binary file.

[0117] It is easy to understand that the image encoding signal after being encoded by the image sensor can be directly input into the spectral reconstruction network, and the hyperspectral image can be obtained by combining the encoding information. It can also be input into the target recognition network for target recognition, and then the main target is segmented, and the complex environmental signals are discarded. Finally, the main target image is input into the spectral reconstruction network, and the key target spectral information is obtained by combining the encoding information. Therefore, the flexibility of the system can be improved.

[0118] Optionally, Figure 3 The structure diagram of a spectrum reconstruction network provided by the embodiment of the present disclosure is shown in FIG. Figure 3 As shown, first, the input target image is processed by the first convolution layer to generate the first intermediate feature map. Subsequently, the first intermediate feature map passes through the first downsampling layer and the multi-head self-attention module to generate the second intermediate feature map. Then, the second intermediate feature map passes through the second downsampling layer and the multi-head self-attention module to generate the third intermediate feature map. Then, the third intermediate feature map passes through the third downsampling layer, the multi-head self-attention module and the first upsampling layer to generate the fourth intermediate feature map, and the third intermediate feature map and the fourth intermediate feature map are refined and spliced ​​to obtain the fifth intermediate feature map. Then, the fifth intermediate feature map passes through the second upsampling layer and the multi-head self-attention module to generate the sixth intermediate feature map, and the second intermediate feature map and the sixth intermediate feature map are refined and spliced ​​to obtain the seventh intermediate feature map. Then, the seventh intermediate feature map passes through the third upsampling layer and the multi-head self-attention module to generate the eighth intermediate feature map, and the first intermediate feature map and the eighth intermediate feature map are refined and spliced ​​to obtain the ninth intermediate feature map. Finally, the ninth intermediate feature map is processed by the second convolution layer to generate the final reconstructed hyperspectral image.

[0119] According to some embodiments, the input target image is an image coding signal after spectral reconstruction image preprocessing, or a main target obtained after main target segmentation.

[0120] In some embodiments, the final reconstructed hyperspectral image is a high-channel hyperspectral image result, or a high-channel key target hyperspectral information result.

[0121] According to some embodiments, the multi-head self-attention module includes layer normalization, matrix multiplication operations, and a feed-forward neural network.

[0122] In some embodiments, the matrix multiplication operation may be, for example, a query (Query, Q ), Key K ), Value, V ) matrix multiplication operation. In order to guide the network to focus on the high-fidelity spectral representation area, different from the traditional visual transformer model, the transformer network for hyperspectral reconstruction can be Q , K , V The matrix multiplication operation is rewritten as follows:

[0123]

[0124] in, is a matrix K The transpose of is a learnable parameter. and The weight of the dot product is used to adjust the value of self-attention Attention .

[0125] According to some embodiments, in the process of using a multiplexed reconfigurable network acceleration framework to control the spectral reconstruction network for forward reasoning, a fully connected operation can be used to implement the multi-head self-attention operation in the spectral reconstruction network, and sparse matrix multiplication can be used for optimization when using the fully connected operation to implement the multi-head self-attention operation in the spectral reconstruction network.

[0126] Optionally, Figure 4 The schematic diagram of the principle of a multiplexed reconfigurable network acceleration framework provided by the embodiment of the present disclosure is shown in FIG. Figure 4 As shown in the figure, due to the limitations of cache and computing resources on the programmable logic side, the reusable reconfigurable network acceleration framework cannot implement large-scale convolution and matrix multiplication operations at one time, and the attention mechanism in the convolution and Transformer networks follows different calculation rules. The convolution operation uses the convolution kernel to calculate the output value on the receptive field, while the attention mechanism operates on the input matrix and focuses on different areas. Therefore, large-scale calculations can be divided into small modules with limited calculations by uniformly mapping convolution operations of different dimensions to matrix multiplication operations.

[0127] Among them, the division of convolution operation is as follows Figure 4As shown in (a), T_w, T_h and T_m represent the width, height and number of channels of the output feature map convolution block, respectively, T_in_w, T_in_h and T_in_m represent the width, height and number of channels of the input feature map convolution block, respectively, and T_in_w and T_in_h can be calculated by T_w, T_h, convolution kernel size k and step size. In other words, the convolution block corresponding to the convolution operation can be represented by four parameters (T_in_m, T_m, T_w, T_h).

[0128] Among them, the division of matrix multiplication is as follows Figure 4 As shown in (b) in the attention mechanism, T_h = T_in_h, so the matrix block corresponding to the matrix multiplication operation can be represented by three parameters (T_in_w, T_w, T_h).

[0129] According to some embodiments, the computation arrays in the convolution and fully connected modules are composed of binary trees, which can be represented by (T i , T o ) to configure parameters, T i represents the computation array input channel, T o Represents the computed array output channel.

[0130] In some embodiments, (T i , T o ) correspond to the first two parameters in the convolution block and the matrix block, that is, to calculate the array input channel T i Corresponding to the channel input parameter T_in_m and the width input parameter T_in_w respectively, the array output channel T is calculated o They correspond to the channel output parameter T_m and the width output parameter T_w respectively, so that the convolution operation and the matrix multiplication operation can be reused.

[0131] Optionally, Figure 5 The schematic diagram of the principle of a double buffering strategy provided by the embodiment of the present disclosure is as follows. Figure 5 As shown in the figure, during the operation of the hyperspectral perception and processing system, a double buffering strategy is used to optimize all network functional modules. Under the same working main frequency and time margin, the double buffering strategy can increase the computing efficiency by 1.5 times, greatly shorten the computing delay, and realize the real-time and rapid reconstruction of hyperspectral images on the embedded platform.

[0132] Optionally, in the multiplexed reconfigurable network acceleration framework, all input and output parameters are low-bit quantized, which can greatly optimize the computational delay and make real-time and rapid reconstruction of hyperspectral images on embedded platforms possible.

[0133] In summary, the hyperspectral perception processing system provided in this embodiment, by adopting software and hardware collaborative design, makes full use of the logic processing advantages of the core processor end and the pipeline parallel computing advantages of the programmable logic end according to the functional characteristics of the module, and can efficiently realize computational spectral imaging perception, with high recognition accuracy, fast reconstruction speed, and low operating power consumption. Secondly, it has the characteristics of high integration, can realize embedded or chip-level spectral imaging perception, and can be widely used in satellite remote sensing, smart agriculture, smart transportation and other fields.

[0134] In order to implement the above embodiments, the present disclosure further proposes a hyperspectral sensing processing chip, including: the hyperspectral sensing processing system provided by the above embodiments.

[0135] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0136] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0137] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0138] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0139] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0140] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0142] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0143] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0144] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0145] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A hyperspectral perception and processing system, characterized in that: include: A programmable logic end, wherein a multiplexed reconfigurable network acceleration framework is built in the programmable logic end; The multiplexed reconfigurable network acceleration framework is used to deploy a neural network, and implement full-pipeline parallel acceleration operations in the process of deploying the neural network, and uniformly map convolution operations and matrix multiplication operations of different dimensions to implement convolution operations and matrix multiplication operations in a multiplexed manner; Wherein, the hyperspectral perception processing system further includes: An image sensor, wherein a mask array is bonded in front of a photosensitive array of the image sensor, and is used to encode an input image to obtain an image encoding signal; External memory, used to store network parameters and output results; Internal memory, used to implement buffer data interaction; The core processor end is used to implement floating point operations and thread scheduling in the process of data interaction between the image sensor, the external memory, the core processor end and the programmable logic end; wherein the programmable logic end and the core processor end perform data transmission via an AXI-HP interface; The programmable logic end also includes a global control module, and the multiplexed reconfigurable network acceleration framework includes a transposition module, an input cache, a parameter cache, a result cache, an output cache, a convolution and full connection module, and a neural network calculation module; wherein, The global control module is used to receive the instruction operation and input data input by the core processor end, transmit the input data to the input buffer and the parameter buffer, and read the output result from the output buffer; The transposition module is used to perform a matrix transposition operation on the cache data input by the input cache, obtain the matrix-transposed data and input it into the parameter cache; The convolution and fully connected module is used to uniformly map convolution operations and matrix multiplication operations of different dimensions based on the cache data input by the input cache and the cache data input by the parameter cache to multiplex the convolution operations and the matrix multiplication operations, and obtain and input the calculation results into the result cache; The neural network calculation module is used to perform neural network calculation based on the cache data input by the result cache, obtain and input the output result into the output cache; The calculation array in the convolution and fully connected module is composed of a binary tree, the convolution block parameters corresponding to the convolution operation include a channel input parameter and a channel output parameter, and the matrix block parameters corresponding to the matrix multiplication operation include a width input parameter and a width output parameter; The calculation array input channels corresponding to the calculation array correspond to the channel input parameters and the width input parameters respectively, and the calculation array output channels corresponding to the calculation array correspond to the channel output parameters and the width output parameters respectively, so as to multiplex the convolution operation and the matrix multiplication operation.

2. The system according to claim 1, characterized in that The neural network calculation module includes at least one of the following submodules: Pooling module, used to implement 2D pooling operations in neural networks; Upsampling module, used to implement image interpolation upsampling operations in neural networks; Downsampling module, used to implement random downsampling of images in neural networks; GELU module, used to implement nonlinear GELU activation operations in neural networks; Softmax module, used to implement nonlinear Softmax activation operation in neural networks; The layer normalization module is used to implement the layer normalization operation in the neural network.

3. The system according to claim 1, characterized in that The core processor side includes: An image sensor control module, used to control data interaction between the image processing module and the image sensor; The image processing module is used to perform image processing on the image coding signal, obtain the processed image coding signal and input it into the data and thread scheduling module; An external memory control module, used for controlling data interaction between the external memory and the data and thread scheduling module; An internal memory control module, used for controlling data interaction between the internal memory and the data and thread scheduling module; The Contact module and the Add module are used to perform data interaction with the data and thread scheduling module to realize the residual operation of the neural network; The data and thread scheduling module is used to implement data transmission management and thread scheduling during the process of data interaction between the image processing module, the external memory, the internal memory, the Contact module, the Add module and the programmable logic end.

4. The system according to claim 3, characterized in that The neural network comprises a spectral reconstruction network, wherein, The image processing module is further used to perform spectral reconstruction image preprocessing on the image coding signal to obtain the image coding signal after spectral reconstruction image preprocessing; The multiplexed reconfigurable network acceleration framework is also used to input the image encoding signal after the spectral reconstruction image preprocessing into the spectral reconstruction network, read the network parameters, control the spectral reconstruction network to perform forward reasoning to obtain a high-channel hyperspectral image result, and input the high-channel hyperspectral image result as an output result into the external memory.

5. The system according to claim 3, characterized in that The neural network includes a spectrum reconstruction network and a target recognition network, wherein: The image processing module is further used to perform target recognition image preprocessing on the image coding signal to obtain an image coding signal after target recognition image preprocessing; The multiplexed reconfigurable network acceleration framework is also used to input the image encoding signal after the target recognition image preprocessing into the target recognition network, read the network parameters, and control the target recognition network to perform forward reasoning to obtain a recognition result; The image processing module is further used to segment the recognition result into a main object and determine the pixel size of the main object obtained after segmentation; The multiplexed reconfigurable network acceleration framework is also used to input the main target and the pixel size into the spectral reconstruction network, read the network parameters, control the spectral reconstruction network to perform forward reasoning to obtain high-channel key target hyperspectral information results, and input the high-channel key target hyperspectral information results as output results into the external memory.

6. The system according to claim 4 or 5, characterized in that: The multiplexed reconfigurable network acceleration framework is used to control the spectral reconstruction network to perform forward reasoning, and when a fully connected operation is used to implement a multi-head self-attention operation in the spectral reconstruction network, sparse matrix multiplication is used for optimization.

7. The system according to claim 1, characterized in that The hyperspectral sensing and processing system adopts a double buffer optimization strategy during operation.