Construction method for spectrum processing chip and spectrum processing chip

Abstract

The present application relates to the technical field of image processing. Provided are a construction method for a spectrum processing chip and a spectrum processing chip. The chip comprises a filter layer and an image sensor, the filter layer being formed by a plurality of modulation units, the plurality of modulation units being attached to the surface of the image sensor, and the plurality of modulation units and the image sensor jointly acting to perform optical analog computation of the inner product of feature vectors on incident natural spectral images to obtain spectral image features. The present application uses the plurality of modulation units and the image sensor to jointly perform optical analog computation of the inner product of feature vectors on incident natural spectral images to obtain spectral image features, thus directly detecting, acquiring and sensing visual light information in both spatial and frequency domains in real time, allowing for efficient and rapid subsequent analysis of feature information, facilitating convenient deployment on various mobile devices and edge devices, and introducing hyperspectral sensing capability for devices.

Description

Method for constructing spectrum processing chip and spectrum processing chip

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2024102309475, filed on February 29, 2024, entitled “Construction Method of Spectral Processing Chip and Spectral Processing Chip,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the field of image processing technology, and in particular to a method for constructing a spectral processing chip and a spectral processing chip. Background Art

[0004] Artificial neural networks are a key engine driving the intelligent era. However, neural network architectures based on the von Neumann architecture struggle to achieve optimal performance, power consumption, and computational speed. Optical neural networks, with their high parallelism and low power consumption, are expected to overcome the shortcomings of electrical neural networks once large-scale optical neural networks are realized, becoming a leading technology in the next generation of intelligent technology.

[0005] Existing optical neural networks mainly implement matrix multiplication operations based on optical devices such as spatial optical paths or on-chip waveguides. The solution based on spatial optical paths uses the propagation of light in free space and its coherent interaction with components such as SLMs and optical masks to implement matrix-vector multiplication operations. This solution can directly use the spatial distribution of light carrying visual information as a computing resource, and use the wave nature of light to calculate the speed of light propagation. It can reflect the high-speed and parallel advantages of optical computing. Compared with electrical computing, it has a natural advantage in the intelligent processing of visual information. The optical neural network solution based on micro-nano optical waveguides is based on the electromagnetic wave nature of light and uses the interference of light transmitted in different waveguides to realize optical computing functions. It can reflect the high-speed and low-power advantages of optical computing.

[0006] Optical artificial neural networks based on spatial optical paths have limited their device integration potential and reduced system stability due to their optical path requirements. Optical artificial neural networks based on micro-nano optical waveguides require coherent light as an excitation source, but the number of computational units increases with the square of the data dimension, and the strict precision requirements of interferometer elements also limit their ability to handle large-scale data volumes and complex tasks.

[0007] Most of the existing optical artificial neural network solutions mentioned above are based on matrix multiplication and implement a fully connected network architecture. They can only process one-dimensional vectors and have difficulty processing two-dimensional image information. As a result, the threshold for expanding existing optical artificial neural networks to real-world computer vision tasks is high, and it is difficult to give full play to the advantages of optical neural networks in processing visual information.

[0008] Furthermore, an image sensor is required to convert natural images into digital images, which are then serialized into one-dimensional vectors before being encoded onto coherent light for optical computing. This approach cannot directly process natural images, and the optoelectronic-to-electrical-optical conversion process does not fully exploit the characteristics of optical computing. Furthermore, during the initial optoelectronic conversion process, converting natural images into digital images, only the color information of the object is retained, while a large amount of light field information, such as the spectrum, is lost.

[0009] Summary of the Invention

[0010] In response to the problems existing in the prior art, the present application provides a method for constructing a spectral processing chip and a spectral processing chip.

[0011] The present application provides a method for constructing a spectrum processing chip, wherein the spectrum processing chip includes a filter layer and an image sensor, and the method includes:

[0012] A filter layer is constructed by multiple modulation units, and the multiple modulation units are attached to the surface of an image sensor. The multiple modulation units and the image sensor work together to perform optical simulation calculation of the inner product of the characteristic vectors of the incident natural spectrum image to obtain the spectral image characteristics.

[0013] In one embodiment, the method further includes: constructing the modulation unit, including:

[0014] Constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between structural parameters of the micro-nanostructure of each modulation unit and requirements of a target application scenario;

[0015] Optimizing and solving the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0016] The modulation unit is constructed according to the determined structural parameters of the micro-nano structure of the modulation unit.

[0017] In one embodiment, the natural spectrum image sample set includes natural spectrum image samples of multiple categories, and the structural parameters include period parameters and shape parameters. Accordingly, constructing an objective function for designing a modulation unit includes:

[0018] Constructing a first loss function, wherein the first loss function represents the variance relationship of the measurement results of samples of different categories and the variance relationship of the measurement results of samples of the same category;

[0019] Constructing a second loss function, wherein the second loss function characterizes the correlation of the transmission spectrum matrix of the modulation unit;

[0020] Constructing a third loss function, wherein the third loss function represents the period similarity and shape similarity of the modulation unit;

[0021] The objective function is constructed based on the first loss function, the second loss function and the third loss function.

[0022] In one embodiment, the first loss function comprises:

[0023] Among them, L fas (p1, q1, ..., p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

[0024] In one embodiment, the second loss function comprises:

[0025] Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

[0026] In one embodiment, the third loss function includes:

[0027] Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

[0028] In one embodiment, the objective function includes:

[0029] L total =αL fas +βLcorr +γL fab

[0030] Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

[0031] In one embodiment, constructing a filter layer from a plurality of modulation units includes:

[0032] Arrange the modulation units in an m×m array to obtain the convolution kernel;

[0033] Arranging the convolution kernels in an n×n array to obtain a convolution operation unit;

[0034] The convolution operation units are arranged in an H×W array to obtain a filter layer.

[0035] In one embodiment, each modulation unit includes at least one group of micro-nano structures.

[0036] The spectral processing chip includes a filter layer and an image sensor, wherein:

[0037] The filter layer is composed of multiple modulation units, which are attached to the surface of the image sensor. The multiple modulation units and the image sensor work together to perform optical simulation calculation of the inner product of the characteristic vector of the incident natural spectrum image to obtain the spectral image characteristics.

[0038] In one embodiment, the filter layer includes a plurality of convolution operation units arranged in an array, each of the convolution operation units includes a plurality of convolution kernels arranged in an array, and each of the convolution kernels includes one or more modulation units arranged in an array.

[0039] In one embodiment, each modulation unit includes at least one group of micro-nano structures.

[0040] In one embodiment, the parameters of the micro-nano structures in the same modulation unit are the same or different.

[0041] In one embodiment, the parameters of the micro-nano structures in different modulation units are the same or different.

[0042] In one embodiment, when the product of n and m is a constant, the number and size of the convolution kernels can be dynamically adjusted, and n and m satisfy an inverse proportional relationship.

[0043] In one embodiment, when the modulation unit is multiplexed, the number of convolution kernels in each convolution operation unit is (nm) 2 indivual.

[0044] In one embodiment, a metasurface is used to construct a modulation unit, wherein a plurality of modulation units are used to construct a filter layer with n=m=3, H=160, and W=120.

[0045] The present application also provides a spectrum processing chip, including a filter layer and an image sensor, wherein:

[0046] The filter layer is composed of multiple modulation units, which are attached to the surface of the image sensor. The multiple modulation units and the image sensor work together to perform optical simulation calculation of the inner product of the characteristic vector of the incident natural spectrum image to obtain the spectral image characteristics.

[0047] The present application provides a method for constructing a spectral processing chip and a spectral processing chip, which obtains spectral image features by jointly performing optical simulation calculations of the inner product of feature vectors of an incident natural spectral image through multiple modulation units and image sensors. The chip can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making subsequent analysis of the feature information efficient and rapid, and enabling it to be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities for the devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] FIG1 is a schematic flow chart of a method for constructing a spectrum processing chip according to an embodiment of the present application;

[0050] FIG2 is a diagram of the micro-nanostructure shape of a modulation unit provided in an embodiment of the present application;

[0051] FIG3 is a schematic structural diagram of a filter layer provided in an embodiment of the present application;

[0052] FIG4 is a diagram showing a dynamic change process of a convolution kernel according to an embodiment of the present application;

[0053] FIG5 is a schematic diagram of the size and quantity of convolution kernels provided in an embodiment of the present application;

[0054] FIG6 is a schematic diagram showing the matching of the transmission spectrum of the micro-nanostructure provided in an embodiment of the present application with the characteristic absorption peak of hemoglobin;

[0055] FIG7 is a schematic diagram of a chip pressing process according to an embodiment of the present application;

[0056] FIG8 is a schematic diagram of a neural network model architecture provided in an embodiment of the present application;

[0057] FIG9 is a diagram showing pixel-level accuracy and image-level accuracy provided by an embodiment of the present application;

[0058] FIG10 is a schematic diagram of the metasurface structure obtained by the present application to achieve spectral recognition of thyroid pathological section microscopic images;

[0059] FIG11 is a schematic diagram of the predicted and measured metasurface transmission spectra for realizing spectral recognition of thyroid pathological microscopic images in the present application;

[0060] FIG12 is a schematic diagram of the present application for realizing spectral recognition of spectral characteristics of thyroid pathological slice microscopic images in different pathological states;

[0061] FIG13 is a schematic diagram of the clustering results of spectral features of thyroid pathological slice microscopic images in different pathological states implemented by the present application;

[0062] FIG14 is a diagram showing the effect of the present application on the spectral recognition of thyroid pathological slice microscopic images in slice microscopic images of different pathological conditions;

[0063] FIG15 is a diagram showing the final recognition results of spectral recognition of thyroid pathological slice microscopic images for slice microscopic images of different pathological conditions implemented in the present application;

[0064] FIG16 is a schematic diagram of the structure of the spectrum processing chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0066] The following describes a method for constructing a spectrum processing chip, a spectrum processing chip, a spectrum analysis device, and an electronic device of the present application in conjunction with Figures 1 to 16.

[0067] FIG1 shows a method for constructing a spectrum processing chip provided in an embodiment of the present application. The spectrum processing chip includes a filter layer and an image sensor. The method includes:

[0068] S11, constructing a filter layer by combining multiple modulation units;

[0069] S12. Attach a plurality of modulation units to a surface of an image sensor. The plurality of modulation units and the image sensor work together to perform an optical simulation calculation of an inner product of feature vectors on an incident natural spectrum image to obtain spectral image features.

[0070] In this regard, it should be noted that in this application, the spectral processing chip can be designed and constructed based on a filter layer and an image sensor. For example, the image sensor can be a CMOS image sensor (CIS), CCD, etc. In the application scenario of image analysis based on a neural network model algorithm, the filter layer and the image sensor together constitute an optical convolutional layer (OCL) of the neural network model, which has the ability to perform integrated sensing and computing on natural spectral images.

[0071] In this application, the filter layer is composed of multiple modulation units arranged in an array, and the multiple modulation units are attached to the surface of the image sensor. Specifically, the multiple modulation units are aligned with and attached to the surface of the pixel units in the image sensor. Because the modulation units are used for light filtering, they can also be referred to as filter structure units in this application.

[0072] This natural spectrum image serves as the input to the optical computation convolution layer. The modulation unit of the filter layer and the CMOS image sensor array jointly perform an optical analog calculation of the inner product of the feature vectors of the input natural spectrum image. The calculation result is converted into an electrical signal, which is the spectral image feature output by the optical computation convolution layer. This spectral image feature serves as the input to the later layers of the neural network. Integrating artificial intelligence algorithms such as multilayer perceptrons and convolutional neural networks in these later layers enables real-time decoding processing on platforms such as processors.

[0073] The optical analog calculation of the inner product of the eigenvectors of an input natural spectrum image can be performed using either coherent or incoherent light illumination. The intensity distribution of the incident light on the spatial light modulation plane is equivalent to the input vector value. Different pixels of the spatial light modulation are encoded according to the weighting coefficients. The light beam is sequentially passed through the spatial light modulation and lens to a focal point. A detector is placed at the lens focal point and the total light intensity on the spatial light modulator plane is collected. The result is equivalent to the inner product between the input vector and the weighting coefficient vector.

[0074] In the present application, further, each modulation unit includes at least one micro-nano structure, which can be implemented as a modulation hole, nanopillar, nanowire, etc. Specifically, the modulation unit can be a structure or material with filtering properties, such as a metasurface, a modulation hole, a nanopillar, a nanowire, a photonic crystal, a multilayer film, a dye, a quantum dot, a MEMS (micro-electromechanical system), an FP etalon, a cavity layer, a waveguide layer, or a diffraction element.

[0075] Furthermore, the micro-nano structure of the modulation unit can be represented by a pair of parameters (period parameter and shape parameter). In order to comprehensively consider the application objects of various application scenarios, the micro-nano structure can be optimized and adjusted. See Figure 2 for a schematic diagram of the micro-nano structure under different designed structural parameters.

[0076] In this application, the micro-nanostructure of the modulation unit can be represented by a pair of parameters. Taking into account the application objects in different scenarios, optimizing and adjusting the parameters of the micro-nanostructure can determine the specific form of the modulation unit, and then construct the specific form of the convolution kernel of the filter layer and the calculation unit. To this end, this application first needs to construct the specific form of the modulation unit.

[0077] In order to design the modulation unit, this application proposes an application-oriented micro-nanostructure discrete topology optimization algorithm, which represents the micro-nanostructure of the modulation unit through a pair of parameters, and comprehensively considers the application objectives, feature extraction capabilities, and processing error tolerance. The parameters of the micro-nanostructure of the modulation unit are obtained using the gradient descent method. To this end, an objective function for designing the modulation unit is constructed. The objective function represents the mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario. By optimally solving the objective function, the parameters of the micro-nanostructure of each modulation unit are determined.

[0078] Since the modulation unit is aimed at the application goals in a specific application scenario, it is necessary to obtain a natural spectrum image sample set in the target application scenario. This sample set is used as a test sample. With the sample set as data support, the constructed objective function is optimized and solved to determine the structural parameters of the micro-nano structure of the modulation unit.

[0079] A modulation unit is constructed based on the structural parameters of the determined micro-nanostructure of the modulation unit, and a spectral processing chip is generated based on the constructed modulation unit and image sensor. In this application, the purpose of constructing the spectral processing chip is to enable the modulation unit and image sensor array to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectral image to obtain spectral image characteristics.

[0080] In the further chip of the above-mentioned spectral processing chip, the specific structure of the chip is mainly explained. The filter layer includes multiple convolution operation units arranged in an array, each convolution operation unit includes multiple convolution kernels arranged in an array, and each convolution kernel includes multiple modulation units (i.e., filter structure units) arranged in an array. It should be noted that the effect of the array arrangement is to ensure that the multiple convolution operation units can be reasonably distributed. The preferred array arrangement is: each convolution kernel includes m×m modulation units, each convolution operation unit includes n×n convolution kernels, and the filter layer includes H×W convolution operation units.

[0081] In this regard, it should be noted that, in this application, referring to FIG3 , the basic component unit of the filter layer is the modulation unit ( FIG3 takes the metasurface as an example), and each modulation unit is aligned with at least one pixel unit of the CIS and attached to the surface of the pixel unit. Assuming that the transmission spectrum of the modulation unit is h i (λ), the incident natural spectrum is x i (λ), then the spectrum of the transmitted light is x i (λ)h i (λ), the electrical signal generated by the CIS below the modulation unit can be calculated by the following formula:

[0082] It can be seen that the output electrical signal is actually x i , h i The inner product of the two, the modulation unit and CIS actually jointly complete the optical simulation calculation of the vector inner product.

[0083] A convolution kernel consists of M=m×m modulation units. The CIS under each modulation unit will produce an inner product calculation result I i， Then the calculation result of the convolution kernel v k The sum of these M electrical signals can be obtained:

[0084] The optical filter layer usually has multiple convolution kernels. Arranging N=n×n convolution kernels together constitutes an optical convolutional unit (OCU). Therefore, an OCU with N convolution kernels generates N outputs after calculation: v1, v2, ..., v N Arranging a large number of OCUs in an H×W array at different spatial positions constitutes a complete OCL (optical computation convolution layer, i.e., filter layer). The OCUs at different spatial positions implement optical parallel convolution operations. Let the calculation result of the kth convolution kernel of the OCU at position (h, w) be v (h,w)k (1≤h≤H, 1≤w≤W, 1≤k≤N), then the spectral image feature output after OCL calculation is F={v (h,w)k}∈RH×W×N .

[0085] The micro-nanostructure can be optimized and adjusted, specifically by constructing an objective function for designing the modulation unit, where the objective function represents the mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario; for example, the structural parameters can be the period parameter p and the shape parameter q of the micro-nanostructure;

[0086] The objective function is optimized and solved based on the natural spectrum image sample set in the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit.

[0087] The main purpose is to explain the process of constructing the objective function of the designed modulation unit, as follows:

[0088] In the present application, the natural spectrum image sample set includes natural spectrum image samples of various categories, and the structural parameters include period parameters and shape parameters.

[0089] Construct a first loss function that characterizes the variance relationship of the measurement results of samples of different categories, as well as the variance relationship of the measurement results of samples of the same category. It is used to ensure that the intra-class variance is as small as possible and the inter-class variance is as large as possible.

[0090] A second loss function is constructed, which characterizes the correlation of the transmission spectrum matrix of the modulation unit. The correlation includes row correlation or column correlation, which determines the perception ability of the group of filter structure units in the spectral dimension.

[0091] A third loss function is constructed, which characterizes the period similarity and shape similarity of the modulation unit, which determines the tolerance to the processing error of the micro-nano structure.

[0092] An objective function is constructed based on the first loss function, the second loss function, and the third loss function.

[0093] Furthermore, the first loss function includes:

[0094] Among them, L fas (p1, q1, ..., p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

[0095] The second loss function includes:

[0096] Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

[0097] The third loss function includes:

[0098] Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

[0099] The objective function includes:

[0100] L total =αL fas +βL corr +γL fab

[0101] Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

[0102] Continuing with Figure 3, the parameters of the micro-nano structures within the same modulation unit are the same or different, and the parameters of the micro-nano structures in different modulation units are the same and / or different. By modifying the structure and parameters of the filter layer, the present application can conveniently implement a variety of complex machine vision tasks based on hyperspectral images without changing the physical structure of the chip, achieving real-time hyperspectral image classification, hyperspectral target detection, hyperspectral instance segmentation, and the like.

[0103] Furthermore, an OCU has nm×nm pixels, which is composed of n×n convolution kernels, and the size of each convolution kernel is m×m. When the product of n and m is a constant, the number of convolution kernels and the size of the convolution kernel can be changed dynamically, as shown in Figure 4. However, n and m must satisfy an inverse proportional relationship, that is, the more convolution kernels there are, the smaller the size of the convolution kernel. However, through the scheme of modulation unit multiplexing, the number of convolution kernels in an OCU can be greatly increased. Continuing with Figure 4, there can be a maximum of (nm) 2 The convolution kernel size and number in Figure 5 can be changed dynamically. a) Convolution kernel size is 6×6; b) Convolution kernel size is 2×2; c) Convolution kernel size is 3×3. This application facilitates adaptation to different application scenarios by dynamically changing the convolution kernel data and size.

[0104] For example, in face anti-counterfeiting applications, a 3×3 convolution kernel with a total of 9 kernels is selected to consider the robustness under different lighting conditions; when detecting the sugar content of specific fruits (such as grapes), a 2×2 convolution kernel with a total of 4 kernels is selected.

[0105] This application can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time. For spatial information, that is, the spatial light energy distribution of the image, based on the convolutional neural network architecture, using its shared weight characteristics and the advantages of the metasurface for large-scale array integration, the spectral spatial resolution can be achieved to 10 6 This level of resolution can meet the spatial resolution requirements of convolutional neural networks in most machine vision tasks. For frequency domain information, through feature matching and precise design of the metasurface, low-power, light-speed hyperspectral feature extraction is achieved, compressing hundreds of channels in hyperspectral images to just a few or a dozen channels. While utilizing optical metasurfaces to significantly reduce the difficulty of hyperspectral data detection and acquisition, it also compresses the frequency domain data volume, further improving the processing capabilities for complex application scenarios.

[0106] In the application scenario of spectral anti-counterfeiting face recognition, the present application uses n=m=3, H=160, and W=122, and adopts metasurface technology to realize the modulation unit. The 9 different optimized metasurfaces are shown in Figure 3, and the corresponding transmission spectra and the matching with the characteristic absorption peak of hemoglobin are shown in Figure 6. During the processing and preparation process, based on SOI (Silicon-On-Insulator, silicon on an insulating substrate), OCL is prepared through the PDMS imprint transfer scheme (Figure 7). The network structure of the filter layer (i.e., optical computing convolution layer) used is shown in Figure 8. The pixel-level face anti-counterfeiting accuracy rate reaches 96%, and the image-level face anti-counterfeiting accuracy rate reaches 100%, as shown in Figure 9.

[0107] In this embodiment, the metasurface convolution layer can perform direct frequency domain operations on natural spectrum images, and combine with the CIS and processor to realize parallel computing and processing of large-scale complex hyperspectral machine vision tasks, that is, sensing and computing in one; the freeform shaped meta-atoms metasurface based on feature matching and optimized design can obtain hyperspectral visual information in real time and reduce feature dimensionality, thereby improving the real-time processing capability of the optical neural network for complex application scenario tasks.

[0108] At the same time, large-scale arrays of metasurfaces are conducive to expanding the number of convolution kernels in the optical convolution layer and improving the optical neural network's ability to process spatial optical information, thereby realizing complex task processing; the preparation and integration of metasurfaces can adopt CMOS-compatible processes, which is conducive to the large-scale production of optical neural network chips.

[0109] The proposed application-oriented nanostructure discrete topology optimization algorithm is used to realize spectral recognition of thyroid pathological slice microscopic images, where n=m=3, H=160, and W=120 are used. Metasurface technology is used to realize the filtering structure unit. The 9 different metasurface structures obtained by optimization design are shown in Figure 10, and the corresponding metasurface transmission spectra are shown in Figure 11. The spectral characteristics of slice microscopic images with different pathological states are shown in Figure 12, and the spectral feature clustering results are shown in Figure 13. The recognition effect of spectral convolutional neural network for slice microscopic images with different pathological states is shown in Figure 14, and the final recognition result is shown in Figure 15.

[0110] The method for constructing a spectral processing chip provided in the present application, by constructing a spectral processing chip, enables multiple modulation units and an image sensor array in the chip to jointly perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis, which can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the characteristic information efficient and rapid, so that it can be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities to the devices.

[0111] FIG16 shows a schematic structural diagram of a spectrum processing chip provided in an embodiment of the present application. Referring to FIG16 , the spectrum processing chip includes a filter layer 161 and an image sensor array 162 , wherein:

[0112] The filter layer 161 is composed of multiple modulation units 1611, which are attached to the surface of the image sensor 162. The multiple modulation units 1611 and the image sensor 162 work together to perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectrum image to obtain spectral image characteristics.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a spectrum processing chip, wherein: The spectrum processing chip includes a filter layer and an image sensor, and the method includes: A filter layer is constructed by multiple modulation units, and the multiple modulation units are attached to the surface of an image sensor. The multiple modulation units and the image sensor work together to perform optical simulation calculation of the inner product of the characteristic vectors of the incident natural spectrum image to obtain the spectral image characteristics.

2. The method for constructing a spectrum processing chip according to claim 1, wherein: The method further includes: constructing the modulation unit, including: Constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between structural parameters of the micro-nanostructure of each modulation unit and requirements of a target application scenario; Optimizing and solving the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; The modulation unit is constructed according to the determined structural parameters of the micro-nano structure of the modulation unit.

3. The method for constructing a spectrum processing chip according to claim 2, wherein: The natural spectrum image sample set includes natural spectrum image samples of various categories, and the structural parameters include period parameters and shape parameters. Accordingly, constructing an objective function for designing a modulation unit includes: Constructing a first loss function, wherein the first loss function represents the variance relationship of the measurement results of samples of different categories and the variance relationship of the measurement results of samples of the same category; Constructing a second loss function, wherein the second loss function characterizes the correlation of the transmission spectrum matrix of the modulation unit; Construct a third loss function, the third loss function characterizing the period similarity of the modulation unit Degree and shape similarity; The objective function is constructed based on the first loss function, the second loss function and the third loss function.

4. The method for constructing a spectrum processing chip according to claim 3, wherein: The first loss function includes: Among them, L fas (p1, q1, ..., p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

5. The method for optimizing a spectrum processing chip according to claim 4, wherein: The second loss function includes: Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

6. The method for constructing a spectrum processing chip according to claim 5, wherein: The third loss function includes: Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

7. The method for optimizing a spectrum processing chip according to claim 6, wherein: The objective function includes: L total =αL fas +βL corr +γL fab Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

8. The method for constructing a spectrum processing chip according to claim 1, wherein: The method of constructing a filter layer from a plurality of modulation units includes: Arrange the modulation units in an m×m array to obtain the convolution kernel; Arranging the convolution kernels in an n×n array to obtain a convolution operation unit; The convolution operation units are arranged in an H×W array to obtain a filter layer.

9. The method for constructing a spectrum processing chip according to claim 1 or 8, wherein: Each modulation unit includes at least one group of micro-nano structures.

10. The method for constructing a spectrum processing chip according to claim 9, wherein: The parameters of the micro-nano structures in the same modulation unit are the same or different.

11. The method for constructing a spectrum processing chip according to claim 10, wherein: The parameters of the micro-nano structures in different modulation units are the same or different.

12. The method for constructing a spectrum processing chip according to claim 8, wherein: When the product of n and m is a constant, the number and size of the convolution kernels can be dynamically controlled, and n and m satisfy an inverse proportional relationship.

13. The method for constructing a spectrum processing chip according to claim 12, wherein: When the modulation unit is multiplexed, the number of convolution kernels in each convolution operation unit is (nm) 2 indivual.

14. The method for constructing a spectrum processing chip according to claim 8, wherein: A metasurface is used to construct a modulation unit, wherein a plurality of modulation units are used to construct a filter layer with n=m=3, H=160, and W=120.

15. A spectrum processing chip, comprising a filter layer and an image sensor, wherein: The filter layer is composed of multiple modulation units, which are attached to the surface of the image sensor. The multiple modulation units and the image sensor work together to perform optical simulation calculation of the inner product of the characteristic vector of the incident natural spectrum image to obtain the spectral image characteristics.

Images