Spectral processing chip optimization method and apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of image processing, and provides a spectral processing chip optimization method and apparatus, an electronic device, and a storage medium. The method comprises: constructing and designing a target function of modulation units, the target function representing a mapping relationship between a structural parameter of a micro-nano structure of each modulation unit and a requirement of a target application scenario; performing optimized solving on the target function on the basis of a natural spectral image sample set in the target application scenario, and determining structural parameters of micro-nano structures of the modulation units; and constructing the modulation units on the basis of the determined structural parameters of the micro-nano structures of the modulation units, and generating a spectral processing chip on the basis of the constructed modulation units and an image sensor. According to the present application, the spectral processing chip is constructed, the plurality of modulation units and the image sensor in the chip jointly perform optical analog computing on a feature vector inner product of an incident natural spectral image, and can directly detect, acquire and sense visual light information of a spatial domain and a frequency domain in real time, making subsequent feature information analysis efficient and rapid.

Description

Optimization method, device, electronic device and storage medium for spectrum processing chip

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2024102309600, filed on February 29, 2024, entitled “Optimization method, device, electronic device and storage medium for spectral processing chip,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the field of image processing technology, and in particular to an optimization method, device, electronic device and storage medium for a spectral processing chip. Background Art

[0004] Artificial neural networks are a key engine driving the intelligent era. However, neural network architectures based on the von Neumann architecture struggle to achieve optimal performance, power consumption, and computational speed. Optical neural networks, with their high parallelism and low power consumption, are expected to overcome the shortcomings of electrical neural networks once large-scale optical neural networks are realized, becoming a leading technology in the next generation of intelligent technology.

[0005] Existing optical neural networks mainly implement matrix multiplication operations based on optical devices such as spatial optical paths or on-chip waveguides. The solution based on spatial optical paths uses the propagation of light in free space and its coherent interaction with components such as SLMs and optical masks to implement matrix-vector multiplication operations. This solution can directly use the spatial distribution of light carrying visual information as a computing resource, and use the wave nature of light to calculate the speed of light propagation. It can reflect the high-speed and parallel advantages of optical computing. Compared with electrical computing, it has a natural advantage in the intelligent processing of visual information. The optical neural network solution based on micro-nano optical waveguides is based on the electromagnetic wave nature of light and uses the interference of light transmitted in different waveguides to realize optical computing functions. It can reflect the high-speed and low-power advantages of optical computing.

[0006] Optical artificial neural networks based on spatial optical paths have limited their device integration potential and reduced system stability due to their optical path requirements. Optical artificial neural networks based on micro-nano optical waveguides require coherent light as an excitation source, but the number of computational units increases with the square of the data dimension, and the strict precision requirements of interferometer elements also limit their ability to handle large-scale data volumes and complex tasks.

[0007] Most of the existing optical artificial neural network solutions mentioned above are based on matrix multiplication and implement a fully connected network architecture. They can only process one-dimensional vectors and have difficulty processing two-dimensional image information. As a result, the threshold for expanding existing optical artificial neural networks to real-world computer vision tasks is high, and it is difficult to give full play to the advantages of optical neural networks in processing visual information.

[0008] Furthermore, an image sensor is required to convert natural images into digital images, which are then serialized into one-dimensional vectors before being encoded onto coherent light for optical computing. This approach cannot directly process natural images, and the optoelectronic-to-electrical-optical conversion process does not fully exploit the characteristics of optical computing. Furthermore, during the initial optoelectronic conversion process, converting natural images into digital images, only the color information of the object is retained, while a large amount of light field information, such as the spectrum, is lost.

[0009] Summary of the Invention

[0010] In response to the problems existing in the prior art, the present application provides an optimization method, device, electronic device and storage medium for a spectral processing chip.

[0011] The present application provides a method for optimizing a spectrum processing chip, comprising:

[0012] Constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between structural parameters of the micro-nanostructure of each modulation unit and requirements of a target application scenario;

[0013] Optimizing and solving the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0014] A modulation unit is constructed according to the determined structural parameters of the micro-nano structure of the modulation unit, and a spectrum processing chip is generated according to the constructed modulation unit and the image sensor.

[0015] In one embodiment, the modulation unit and the image sensor are used to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectrum image to obtain the spectrum image features.

[0016] In one embodiment, the natural spectrum image sample set includes natural spectrum image samples of multiple categories, and the structural parameters include period parameters and shape parameters. Accordingly, constructing an objective function for designing a modulation unit includes:

[0017] Constructing a first loss function, wherein the first loss function represents the variance relationship of the measurement results of samples of different categories and the variance relationship of the measurement results of samples of the same category;

[0018] Constructing a second loss function, wherein the second loss function characterizes the correlation of the transmission spectrum matrix of the modulation unit;

[0019] Constructing a third loss function, wherein the third loss function represents the period similarity and shape similarity of the modulation unit;

[0020] The objective function is constructed based on the first loss function, the second loss function and the third loss function.

[0021] In one embodiment, the first loss function comprises:

[0022] Among them, L fas (p1, q1, ..., p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

[0023] In one embodiment, the second loss function comprises:

[0024] Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

[0025] In one embodiment, the third loss function includes:

[0026] Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

[0027] In one embodiment, the objective function includes:

[0028] L total =αL fas +βL corr +γL fab

[0029] Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

[0030] In one embodiment, the method of generating a spectrum processing chip based on the constructed modulation unit and image sensor includes:

[0031] Arranging the modulation units in an m×m array to obtain a convolution kernel;

[0032] Arranging the convolution kernels in an n×n array to obtain a convolution operation unit;

[0033] Arranging the convolution operation units in an H×W array to obtain a filter layer;

[0034] The filter layer and the image sensor are aligned to form a spectrum processing chip.

[0035] In one embodiment, a metasurface is used to construct a modulation unit, wherein a plurality of modulation units are used to construct a filter layer with n=m=3, H=160, and W=120.

[0036] The present application also provides an optimization device for a spectrum processing chip, comprising:

[0037] A construction module is used to construct an objective function for designing a modulation unit, where the objective function represents the mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario;

[0038] An optimization module is used to optimize and solve the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0039] The generation module is used to construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generate a spectrum processing chip according to the constructed modulation unit and the image sensor.

[0040] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned optimization method for the spectral processing chip when executing the program.

[0041] The present application provides an optimization method, device, electronic device and storage medium for a spectral processing chip. By constructing a spectral processing chip, multiple modulation units and image sensors in the chip jointly perform optical simulation calculations of the inner product of the feature vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis. The method can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the feature information efficient and rapid, and enabling it to be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities to the devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] FIG1 is a schematic flow chart of a method for constructing a spectrum processing chip according to an embodiment of the present application;

[0044] FIG2 is a diagram of the micro-nanostructure shape of a modulation unit provided in an embodiment of the present application;

[0045] FIG3 is a schematic structural diagram of a filter layer provided in an embodiment of the present application;

[0046] FIG4 is a schematic flow chart of a method for optimizing a spectrum processing chip according to an embodiment of the present application;

[0047] FIG5 is a diagram showing the dynamic change process of the convolution kernel provided in an embodiment of the present application;

[0048] FIG6 is a schematic diagram of the size and quantity of convolution kernels provided in an embodiment of the present application;

[0049] FIG7 is a schematic diagram showing the matching of the transmission spectrum of the micro-nanostructure provided in an embodiment of the present application with the characteristic absorption peak of hemoglobin;

[0050] FIG8 is a schematic diagram of a chip pressing process according to an embodiment of the present application;

[0051] FIG9 is a schematic diagram of a neural network model architecture provided in an embodiment of the present application;

[0052] FIG10 is a diagram showing pixel-level accuracy and image-level accuracy provided by an embodiment of the present application;

[0053] FIG11 is a schematic diagram of the metasurface structure obtained by the present application to achieve spectral recognition of thyroid pathological section microscopic images;

[0054] FIG12 is a schematic diagram of the predicted and measured metasurface transmission spectra for realizing spectral recognition of thyroid pathological microscopic images in the present application;

[0055] FIG13 is a schematic diagram of the present application for realizing spectral recognition of spectral characteristics of thyroid pathological slice microscopic images in different pathological states;

[0056] FIG14 is a schematic diagram of the clustering results of spectral features of thyroid pathological slice microscopic images in different pathological states implemented by the present application;

[0057] FIG15 is a diagram showing the effect of the present application on the spectral recognition of thyroid pathological slice microscopic images in slice microscopic images of different pathological conditions;

[0058] FIG16 is a diagram showing the final recognition results of spectral recognition of thyroid pathological slice microscopic images for slice microscopic images of different pathological conditions implemented in the present application;

[0059] FIG17 is a schematic diagram of the structure of a spectrum processing chip provided in an embodiment of the present application;

[0060] FIG18 is a schematic diagram of the structure of an optimization device for a spectrum processing chip provided in an embodiment of the present application;

[0061] FIG19 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application are described clearly and completely below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0063] FIG1 shows a method for constructing a spectrum processing chip provided by the present application. The spectrum processing chip includes a filter layer and an image sensor. The method includes:

[0064] S11, constructing a filter layer by combining multiple modulation units;

[0065] S12. Attach a plurality of modulation units to a surface of an image sensor. The plurality of modulation units and the image sensor work together to perform an optical simulation calculation of an inner product of feature vectors on an incident natural spectrum image to obtain spectral image features.

[0066] In this regard, it should be noted that in this application, the spectral processing chip can be designed and constructed based on a filter layer and an image sensor. For example, the image sensor can be a CMOS image sensor (CIS), CCD, etc. In the application scenario of image analysis based on a neural network model algorithm, the filter layer and the image sensor together constitute an optical convolutional layer (OCL) of the neural network model, which has the ability to perform integrated sensing and computing on natural spectral images.

[0067] In the present application, the filter layer is composed of a plurality of modulation units arranged in an array. These modulation units are formed in the optical path of the image sensor. Preferably, the modulation units are aligned with and attached to the surface of the pixel units in the image sensor. Because the modulation units are used for light filtering, they can serve as filter structural units.

[0068] This natural spectrum image serves as the input to the optical computation convolution layer. The modulation unit of the filter layer and the CMOS image sensor array jointly perform an optical analog calculation of the inner product of the feature vectors of the input natural spectrum image. The calculation result is converted into an electrical signal, which is the spectral image feature output by the optical computation convolution layer. This spectral image feature serves as the input to the later layers of the neural network. Integrating artificial intelligence algorithms such as multilayer perceptrons and convolutional neural networks in these later layers enables real-time decoding processing on platforms such as processors.

[0069] The optical analog calculation of the inner product of the eigenvectors of an input natural spectrum image can be performed using either coherent or incoherent light illumination. The intensity distribution of the incident light on the spatial light modulation plane is equivalent to the input vector value. Different pixels of the spatial light modulation are encoded according to the weighting coefficients. The light beam is sequentially passed through the spatial light modulation and lens to a focal point. A detector is placed at the lens focal point and the total light intensity on the spatial light modulator plane is collected. The result is equivalent to the inner product between the input vector and the weighting coefficient vector.

[0070] In the present application, further, each modulation unit includes at least one micro-nano structure, which can be implemented as a modulation hole, nanopillar, nanowire, etc. Specifically, the modulation unit can be a structure or material with filtering properties, such as a metasurface, a modulation hole, a nanopillar, a nanowire, a photonic crystal, a multilayer film, a dye, a quantum dot, a MEMS (micro-electromechanical system), an FP etalon, a cavity layer, a waveguide layer, or a diffraction element.

[0071] Furthermore, the micro-nano structure of the modulation unit can be represented by a pair of parameters (period parameter and shape parameter). In order to comprehensively consider the application objects of various application scenarios, the micro-nano structure can be optimized and adjusted. See Figure 2 for a schematic diagram of the micro-nano structure under different designed structural parameters.

[0072] In this application, the micro-nanostructure of the modulation unit can be represented by a pair of parameters. Taking into account the application objects in different scenarios, optimizing and adjusting the parameters of the micro-nanostructure can determine the specific form of the modulation unit, and then construct the specific form of the convolution kernel of the filter layer and the calculation unit. To this end, this application first needs to construct the specific form of the modulation unit.

[0073] In order to design the modulation unit, this application proposes an application-oriented micro-nanostructure discrete topology optimization algorithm, which represents the micro-nanostructure of the modulation unit through a pair of parameters, and comprehensively considers the application objectives, feature extraction capabilities, and processing error tolerance. The parameters of the micro-nanostructure of the modulation unit are obtained using the gradient descent method. To this end, an objective function for designing the modulation unit is constructed. The objective function represents the mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario. By optimally solving the objective function, the parameters of the micro-nanostructure of each modulation unit are determined.

[0074] Since the modulation unit is aimed at the application goals in a specific application scenario, it is necessary to obtain a natural spectrum image sample set in the target application scenario. This sample set is used as a test sample. With the sample set as data support, the constructed objective function is optimized and solved to determine the structural parameters of the micro-nano structure of the modulation unit.

[0075] A modulation unit is constructed based on the structural parameters of the determined micro-nanostructure of the modulation unit, and a spectral processing chip is generated based on the constructed modulation unit and image sensor. In this application, the purpose of constructing the spectral processing chip is to enable the modulation unit and image sensor array to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectral image to obtain spectral image characteristics.

[0076] In the further chip of the above-mentioned spectral processing chip, the specific structure of the chip is mainly explained. The filter layer includes multiple convolution operation units arranged in an array, each convolution operation unit includes multiple convolution kernels arranged in an array, and each convolution kernel includes multiple modulation units (i.e., filter structure units) arranged in an array. It should be noted that the effect of the array arrangement is to ensure that the multiple convolution operation units can be reasonably distributed. The preferred array arrangement is: each convolution kernel includes m×m modulation units, each convolution operation unit includes n×n convolution kernels, and the filter layer includes H×W convolution operation units.

[0077] In this regard, it should be noted that, in this application, referring to FIG3 , the basic component unit of the filter layer is the modulation unit ( FIG3 takes the metasurface as an example), and each modulation unit is aligned with at least one pixel unit of the CIS and attached to the surface of the pixel unit. Assuming that the transmission spectrum of the modulation unit is h i (λ), the incident natural spectrum is x i (λ), then the spectrum of the transmitted light is x i (λ)h i (λ), the electrical signal generated by the CIS below the modulation unit can be calculated by the following formula:

[0078] It can be seen that the output electrical signal is actually x i , h i The inner product of the two, the modulation unit and CIS actually jointly complete the optical simulation calculation of the vector inner product.

[0079] A convolution kernel consists of M=m×m modulation units. The CIS under each modulation unit will produce an inner product calculation result I i , then the calculation result of the convolution kernel v k The sum of these M electrical signals can be obtained:

[0080] The optical filter layer usually has multiple convolution kernels. Arranging N=n×n convolution kernels together constitutes an optical convolutional unit (OCU). Therefore, an OCU with N convolution kernels generates N outputs after calculation: v1, v2, ..., v N Arranging a large number of OCUs in an H×W array at different spatial positions constitutes a complete OCL (optical computation convolution layer, i.e., filter layer). The OCUs at different spatial positions implement optical parallel convolution operations. Let the calculation result of the kth convolution kernel of the OCU at position (h, w) be v (h，w)k (1≤h≤H, 1≤w≤W, 1≤k≤N), then the spectral image feature output after OCL calculation is F={v (h，w)k}∈R H×W×N .

[0081] FIG4 shows a schematic flow chart of a method for optimizing a spectrum processing chip provided by the present application. Referring to FIG4 , the method includes:

[0082] 41. Construct an objective function for designing a modulation unit. The objective function represents the mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario. For example, the structural parameters may be the period parameter p and shape parameter q of the micro-nanostructure.

[0083] 42. Optimize and solve the objective function based on the natural spectrum image sample set in the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0084] 43. Construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generate a spectrum processing chip according to the constructed modulation unit and the image sensor.

[0085] The micro-nanostructure can be optimized and adjusted, specifically by constructing an objective function for designing the modulation unit, wherein the objective function represents a mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario; for example, the structural parameters can be a period parameter p and a shape parameter q of the micro-nanostructure;

[0086] The objective function is optimized and solved based on the natural spectrum image sample set in the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit.

[0087] The main purpose is to explain the process of constructing the objective function of the designed modulation unit, as follows:

[0088] In the present application, the natural spectrum image sample set includes natural spectrum image samples of various categories, and the structural parameters include period parameters and shape parameters.

[0089] Construct a first loss function that characterizes the variance relationship of the measurement results of samples of different categories, as well as the variance relationship of the measurement results of samples of the same category. It is used to ensure that the intra-class variance is as small as possible and the inter-class variance is as large as possible.

[0090] A second loss function is constructed, which characterizes the correlation of the transmission spectrum matrix of the modulation unit. The correlation includes row correlation or column correlation, which determines the perception ability of the group of filter structure units in the spectral dimension.

[0091] A third loss function is constructed, which characterizes the period similarity and shape similarity of the modulation unit, which determines the tolerance to the processing error of the micro-nano structure.

[0092] An objective function is constructed based on the first loss function, the second loss function, and the third loss function.

[0093] Furthermore, the first loss function includes:

[0094] Among them, L fas (p1, q1, ..., p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, Xi is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

[0095] The second loss function includes:

[0096] Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

[0097] The third loss function includes:

[0098] Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

[0099] The objective function includes:

[0100] L total =αL fas +βL corr +γL fab

[0101] Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

[0102] Furthermore, a spectrum processing chip is generated based on the constructed modulation unit and image sensor, including:

[0103] Arrange the modulation units in an m×m array to obtain the convolution kernel;

[0104] Arrange the convolution kernels in an n×n array to obtain a convolution operation unit;

[0105] Arrange the convolution operation units in an H×W array to obtain a filter layer;

[0106] Align the filter layer and image sensor to create a spectral processing chip.

[0107] Continuing with Figure 3, the parameters of the micro-nano structures within the same modulation unit are the same or different, and the parameters of the micro-nano structures in different modulation units are the same and / or different. By modifying the structure and parameters of the filter layer, the present application can conveniently implement a variety of complex machine vision tasks based on hyperspectral images without changing the physical structure of the chip, achieving real-time hyperspectral image classification, hyperspectral target detection, hyperspectral instance segmentation, and the like.

[0108] Furthermore, an OCU has nm×nm pixels, which is composed of n×n convolution kernels, and the size of each convolution kernel is m×m. When the product of n and m is a constant, the number of convolution kernels and the size of the convolution kernel can be changed dynamically, as shown in Figure 5. However, n and m must satisfy an inverse proportional relationship, that is, the more convolution kernels there are, the smaller the size of the convolution kernel. However, through the scheme of multiplexing modulation units, the number of convolution kernels in an OCU can be greatly increased, as shown in Figure 6, and the maximum number can be (nm) 2 The convolution kernel size and number in Figure 6 can be changed dynamically. a) Convolution kernel size is 6×6; b) Convolution kernel size is 2×2; c) Convolution kernel size is 3×3. This application facilitates adaptation to different application scenarios by dynamically changing the convolution kernel data and size.

[0109] For example, in face anti-counterfeiting applications, a 3×3 convolution kernel with a total of 9 kernels is selected to consider the robustness under different lighting conditions; when detecting the sugar content of specific fruits (such as grapes), a 2×2 convolution kernel with a total of 4 kernels is selected.

[0110] This application can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time. For the spatial information, that is, the spatial light energy distribution of the image, based on the convolutional neural network architecture, using its shared weight characteristics and the advantages of the metasurface for large-scale array integration, the spectral spatial resolution can be achieved to 10 6 This level of resolution can meet the spatial resolution requirements of convolutional neural networks in most machine vision tasks. For frequency domain information, through feature matching and precise design of the metasurface, low-power, light-speed hyperspectral feature extraction is achieved, compressing hundreds of channels in hyperspectral images to just a few or a dozen channels. While utilizing optical metasurfaces to significantly reduce the difficulty of hyperspectral data detection and acquisition, it also compresses the frequency domain data volume, further improving the processing capabilities for complex application scenarios.

[0111] In the application scenario of spectral anti-counterfeiting face recognition, the present application uses n=m=3, H=160, and W=122, and adopts metasurface technology to realize the modulation unit. The 9 different optimized metasurfaces are shown in Figure 3, and the corresponding transmission spectra and the matching with the characteristic absorption peak of hemoglobin are shown in Figure 7. During the processing and preparation process, based on SOI (Silicon-On-Insulator, silicon on an insulating substrate), OCL is prepared through the PDMS imprint transfer scheme (Figure 8). The network structure of the filter layer (i.e., optical computing convolution layer) used is shown in Figure 9. The pixel-level face anti-counterfeiting accuracy rate reaches 96%, and the image-level face anti-counterfeiting accuracy rate reaches 100%, as shown in Figure 10.

[0112] In this embodiment, the metasurface convolution layer can perform direct frequency domain operations on natural spectrum images, and combine with the CIS and processor to realize parallel computing and processing of large-scale complex hyperspectral machine vision tasks, that is, sensing and computing in one; the freeform shaped meta-atoms metasurface based on feature matching and optimized design can obtain hyperspectral visual information in real time and reduce feature dimensionality, thereby improving the real-time processing capability of the optical neural network for complex application scenario tasks.

[0113] At the same time, large-scale arrays of metasurfaces are conducive to expanding the number of convolution kernels in the optical convolution layer and improving the optical neural network's ability to process spatial optical information, thereby realizing complex task processing; the preparation and integration of metasurfaces can adopt CMOS-compatible processes, which is conducive to the large-scale production of optical neural network chips.

[0114] The proposed application-oriented nanostructure discrete topology optimization algorithm is used to realize spectral recognition of thyroid pathological slice microscopic images, where n=m=3, H=160, and W=120 are used. Metasurface technology is used to realize the filtering structure unit. The 9 different metasurface structures obtained by optimization design are shown in Figure 11, and the corresponding metasurface transmission spectra are shown in Figure 12. The spectral characteristics of slice microscopic images with different pathological states are shown in Figure 13, and the spectral feature clustering results are shown in Figure 14. The recognition effect of spectral convolutional neural network for slice microscopic images with different pathological states is shown in Figure 15, and the final recognition result is shown in Figure 16.

[0115] The optimization method of the spectral processing chip provided in the present application constructs a spectral processing chip, so that multiple modulation units and image sensor arrays in the chip jointly perform optical simulation calculations of the inner product of the feature vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis. It can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the feature information efficient and rapid, so that it can be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities to the devices.

[0116] FIG17 shows a schematic structural diagram of a spectrum processing chip provided by the present application. Referring to FIG17 , the spectrum processing chip includes a filter layer 171 and an image sensor array 172 , wherein:

[0117] The filter layer 171 is composed of multiple modulation units 1711, which are attached to the surface of the image sensor 172. The multiple modulation units 1711 and the image sensor 172 work together to perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectrum image to obtain spectral image characteristics.

[0118] The following describes the optimization device for the spectrum processing chip provided in the present application. The optimization device for the spectrum processing chip described below and the optimization method for the spectrum processing chip described above can be referenced to each other.

[0119] FIG18 shows an optimization device for a spectrum processing chip provided by the present application, which includes a construction module 181, an optimization module 182, and a generation module 183, wherein:

[0120] A construction module 181 is used to construct an objective function for designing a modulation unit, where the objective function represents a mapping relationship between structural parameters of the micro-nano structure of each modulation unit and requirements of a target application scenario;

[0121] An optimization module 182 is configured to optimize and solve an objective function based on a natural spectrum image sample set in a target application scenario to determine structural parameters of the micro-nano structure of the modulation unit;

[0122] A generating module 183 is configured to construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and to generate a spectrum processing chip according to the constructed modulation unit and the image sensor;

[0123] The modulation unit and the image sensor are used to jointly perform optical simulation calculation of the inner product of the feature vector of the incident natural spectrum image to obtain the spectrum image characteristics.

[0124] The optimization device for the spectral processing chip provided in the present application constructs a spectral processing chip so that multiple modulation units and image sensors in the chip jointly perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis. It can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the characteristic information efficient and rapid, so that it can be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities for the devices.

[0125] FIG19 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG19 , the electronic device may include: a processor 191, a communications interface 192, a memory 193, and a communication bus 194, wherein the processor 191, the communications interface 192, and the memory 193 communicate with each other via the communication bus 194. The processor 191 may call the logic instructions in the memory 193 to execute a method for optimizing a spectral processing chip. The method includes: constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario; optimizing and solving the objective function based on a natural spectrum image sample set under the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; constructing the modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generating a spectral processing chip based on the constructed modulation unit and the image sensor.

[0126] In addition, the logic instructions in the above-mentioned memory 193 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0127] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the optimization method of the spectral processing chip provided by the above methods, which includes: constructing an objective function for designing a modulation unit, the objective function characterizing the mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario; optimizing and solving the objective function based on a natural spectrum image sample set under the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; constructing a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generating a spectral processing chip based on the constructed modulation unit and the image sensor.

[0128] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for optimizing a spectral processing chip provided by the above-mentioned methods, the method comprising: constructing an objective function for designing a modulation unit, the objective function characterizing a mapping relationship between structural parameters of the micro-nano structure of each modulation unit and the requirements of a target application scenario; optimizing and solving the objective function based on a natural spectrum image sample set under the target application scenario, and determining the structural parameters of the micro-nano structure of the modulation unit; constructing a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generating a spectral processing chip based on the constructed modulation unit and the image sensor.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0130] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing a spectral processing chip, comprising: Constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between structural parameters of the micro-nanostructure of each modulation unit and requirements of a target application scenario; Optimizing and solving the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; A modulation unit is constructed according to the determined structural parameters of the micro-nano structure of the modulation unit, and a spectrum processing chip is generated according to the constructed modulation unit and the image sensor.

2. The method for optimizing a spectrum processing chip according to claim 1, wherein: The modulation unit and the image sensor are used to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectrum image to obtain the spectrum image features.

3. The method for optimizing a spectrum processing chip according to claim 1, wherein: The natural spectrum image sample set includes natural spectrum image samples of various categories, and the structural parameters include period parameters and shape parameters. Accordingly, constructing an objective function for designing a modulation unit includes: Constructing a first loss function, wherein the first loss function represents the variance relationship of the measurement results of samples of different categories and the variance relationship of the measurement results of samples of the same category; Constructing a second loss function, wherein the second loss function characterizes the correlation of the transmission spectrum matrix of the modulation unit; Constructing a third loss function, wherein the third loss function represents the period similarity and shape similarity of the modulation unit; The first loss function, the second loss function and the third loss function are used to construct the Objective function.

4. The method for constructing a spectrum processing chip according to claim 2, wherein: The first loss function includes: Among them, L fas (p1, q1, ..., p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

5. The method for optimizing a spectrum processing chip according to claim 4, wherein: The second loss function includes: Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

6. The method for constructing a spectrum processing chip according to claim 5, wherein: The third loss function includes: Among them, L fab is the similarity of the modulation unit, is the i-th modulation The similarity between the unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

7. The method for optimizing a spectrum processing chip according to claim 6, wherein: The objective function includes: L total =αL fas +βL corr +γL fab Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

8. The method for optimizing a spectrum processing chip according to claim 1, wherein: The spectrum processing chip is generated based on the constructed modulation unit and image sensor, including: Arranging the modulation units in an m×m array to obtain a convolution kernel; Arranging the convolution kernels in an n×n array to obtain a convolution operation unit; Arranging the convolution operation units in an H×W array to obtain a filter layer; The filter layer and the image sensor are aligned to form a spectrum processing chip.

9. The method for optimizing a spectrum processing chip according to claim 8, wherein: A metasurface is used to construct a modulation unit, wherein a plurality of modulation units are used to construct a filter layer with n=m=3, H=160, and W=120.

10. An optimization device for a spectrum processing chip, comprising: A construction module is used to construct an objective function for designing a modulation unit, where the objective function represents the mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario; An optimization module is used to optimize and solve the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; The generation module is used to construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generate a spectrum processing chip according to the constructed modulation unit and the image sensor.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the optimization method of the spectrum processing chip according to any one of claims 1 to 9 is implemented.

Images