Optimization method and device of spectrum processing chip, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and provides an optimization method and device for a spectrum processing chip, electronic equipment and a storage medium, and the method comprises the steps: constructing a target function for designing modulation units, and enabling the target function to represent a mapping relation between the structure parameters of the micro-nano structure of each modulation unit and the demands of a target application scene; performing optimization solution on the target function based on the natural spectral image sample set in the target application scene, and determining structure parameters of the micro-nano structure of the modulation unit; and constructing the modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generating the spectrum processing chip according to the constructed modulation unit and the image sensor. According to the invention, the spectrum processing chip is constructed, so that the plurality of modulation units and the image sensor in the chip jointly carry out optical simulation calculation of the inner product of the feature vectors on the incident natural spectrum image, visual light information of a spatial domain and a frequency domain can be directly detected, acquired and perceived in real time, and subsequent analysis of the feature information is efficient and rapid.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an optimization method, device, electronic device and storage medium for a spectral processing chip. Background Art

[0002] Artificial neural networks are a key engine driving the intelligent era. However, neural network architectures based on the von Neumann architecture struggle to achieve optimal performance, power consumption, and computational speed. Optical neural networks, with their high parallelism and low power consumption, are expected to overcome the shortcomings of electrical neural networks once large-scale optical neural networks are realized, becoming a leading technology in the next generation of intelligent technology.

[0003] Existing optical neural networks mainly implement matrix multiplication operations based on optical devices such as spatial optical paths or on-chip waveguides. The solution based on spatial optical paths uses the propagation of light in free space and its coherent interaction with components such as SLMs and optical masks to implement matrix-vector multiplication operations. This solution can directly use the spatial distribution of light carrying visual information as a computing resource, and use the wave nature of light to calculate the speed of light propagation. It can reflect the high-speed and parallel advantages of optical computing. Compared with electrical computing, it has a natural advantage in the intelligent processing of visual information. The optical neural network solution based on micro-nano optical waveguides is based on the electromagnetic wave nature of light and uses the interference of light transmitted in different waveguides to realize optical computing functions. It can reflect the high-speed and low-power advantages of optical computing.

[0004] Optical artificial neural networks based on spatial optical paths have limited their device integration potential and reduced system stability due to their optical path requirements. Optical artificial neural networks based on micro-nano optical waveguides require coherent light as an excitation source, but the number of computational units increases with the square of the data dimension, and the strict precision requirements of interferometer elements also limit their ability to handle large-scale data volumes and complex tasks.

[0005] Most of the existing optical artificial neural network solutions mentioned above are based on matrix multiplication and implement a fully connected network architecture. They can only process one-dimensional vectors and have difficulty processing two-dimensional image information. As a result, the threshold for expanding existing optical artificial neural networks to real-world computer vision tasks is high, and it is difficult to give full play to the advantages of optical neural networks in processing visual information.

[0006] Furthermore, an image sensor is required to convert natural images into digital images, which are then serialized into one-dimensional vectors before being encoded onto coherent light for optical computing. This approach cannot directly process natural images, and the optoelectronic-to-electrical-optical conversion process does not fully exploit the characteristics of optical computing. Furthermore, during the initial optoelectronic conversion process, converting natural images into digital images, only the color information of the object is retained, while a large amount of light field information, such as the spectrum, is lost. Summary of the Invention

[0007] In response to the problems existing in the prior art, the present invention provides a method, device, electronic device and storage medium for optimizing a spectrum processing chip.

[0008] The present invention provides a method for optimizing a spectrum processing chip, comprising:

[0009] Constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between structural parameters of the micro-nanostructure of each modulation unit and requirements of a target application scenario;

[0010] Optimizing and solving the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0011] A modulation unit is constructed according to the determined structural parameters of the micro-nano structure of the modulation unit, and a spectrum processing chip is generated according to the constructed modulation unit and the image sensor.

[0012] In one embodiment, the modulation unit and the image sensor are used to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectrum image to obtain the spectrum image features.

[0013] In one embodiment, the natural spectrum image sample set includes natural spectrum image samples of multiple categories, and the structural parameters include period parameters and shape parameters. Accordingly, constructing an objective function for designing a modulation unit includes:

[0014] Constructing a first loss function, wherein the first loss function represents the variance relationship of the measurement results of samples of different categories and the variance relationship of the measurement results of samples of the same category;

[0015] Constructing a second loss function, wherein the second loss function characterizes the correlation of the transmission spectrum matrix of the modulation unit;

[0016] Constructing a third loss function, wherein the third loss function represents the period similarity and shape similarity of the modulation unit;

[0017] The objective function is constructed based on the first loss function, the second loss function and the third loss function.

[0018] In one embodiment, the first loss function comprises:

[0019]

[0020]

[0021] Among them, L fas (p1,q1,…,p N ,q N ) is the first loss value, si is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

[0022] In one embodiment, the second loss function comprises:

[0023]

[0024] Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

[0025] In one embodiment, the third loss function includes:

[0026]

[0027]

[0028] Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

[0029] In one embodiment, the objective function includes:

[0030] L total =αL fas +βL corr +γL fab

[0031]

[0032] Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

[0033] In one embodiment, the method of generating a spectrum processing chip based on the constructed modulation unit and image sensor includes:

[0034] Arranging the modulation units in an m×m array to obtain a convolution kernel;

[0035] Arranging the convolution kernels in an n×n array to obtain a convolution operation unit;

[0036] Arranging the convolution operation units in an H×W array to obtain a filter layer;

[0037] The filter layer and the image sensor are aligned to form a spectrum processing chip.

[0038] In one embodiment, a metasurface is used to construct a modulation unit, wherein a plurality of modulation units are used to construct a filter layer with n=m=3, H=160, and W=120.

[0039] The present invention also provides an optimization device for a spectrum processing chip, comprising:

[0040] A construction module is used to construct an objective function for designing a modulation unit, where the objective function represents the mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario;

[0041] An optimization module is used to optimize and solve the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0042] The generation module is used to construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generate a spectrum processing chip according to the constructed modulation unit and the image sensor.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned optimization method for the spectral processing chip when executing the program.

[0044] The present invention provides an optimization method, device, electronic device and storage medium for a spectral processing chip. By constructing a spectral processing chip, multiple modulation units and image sensors in the chip jointly perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis. The method can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the characteristic information efficient and rapid, and enabling it to be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities to the devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 Schematic diagram of the process of constructing the spectrum processing chip provided by the present invention;

[0047] Figure 2 This is a diagram of the micro-nanostructure shape of the modulation unit provided by the present invention;

[0048] Figure 3 Schematic diagram of the structure of the filter layer provided by the present invention;

[0049] Figure 4 Schematic diagram of the process of optimizing the spectrum processing chip provided by the present invention;

[0050] Figure 5 This is a diagram of the dynamic change process of the convolution kernel provided by the present invention;

[0051] Figure 6 Schematic diagram of the size and quantity of convolution kernels provided by the present invention;

[0052] Figure 7 This is a schematic diagram showing the matching between the transmission spectrum of the micro-nanostructure provided by the present invention and the characteristic absorption peak of hemoglobin;

[0053] Figure 8 This is a schematic diagram of the chip pressing process provided by the present invention;

[0054] Figure 9 This is a schematic diagram of the neural network model architecture provided by the present invention;

[0055] Figure 10 This is a graph showing the pixel-level accuracy and image-level accuracy provided by the present invention;

[0056] Figure 11 This is a schematic diagram of the metasurface structure obtained by the present invention through optimized design for realizing spectral recognition of thyroid pathological section microscopic images;

[0057] Figure 12 It is a schematic diagram of the predicted and measured metasurface transmission spectra for realizing spectral recognition of thyroid pathological section microscopic images according to the present invention;

[0058] Figure 13 Schematic diagram of the present invention for realizing spectral recognition of spectral characteristics of thyroid pathological slice microscopic images in different pathological states;

[0059] Figure 14 2. It is a schematic diagram of the clustering results of spectral recognition of thyroid pathological slice microscopic images in different pathological states by the present invention;

[0060] Figure 15 This is a diagram showing the effect of the present invention on realizing spectral recognition of thyroid pathological slice microscopic images in slice microscopic images of different pathological conditions;

[0061] Figure 16 This is a diagram showing the final recognition results of the present invention for spectral recognition of thyroid pathological slice microscopic images in slice microscopic images of different pathological states;

[0062] Figure 17 It is a structural schematic diagram of the spectrum processing chip provided by the present invention;

[0063] Figure 18 It is a structural schematic diagram of the optimization device of the spectrum processing chip provided by the present invention;

[0064] Figure 19 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0066] Figure 1 The present invention provides a method for constructing a spectrum processing chip, which includes a filter layer and an image sensor. The method includes:

[0067] S11, constructing a filter layer by combining multiple modulation units;

[0068] S12. Attach a plurality of modulation units to a surface of an image sensor. The plurality of modulation units and the image sensor work together to perform an optical simulation calculation of an inner product of feature vectors on an incident natural spectrum image to obtain spectral image features.

[0069] In this regard, it should be noted that in the present invention, the spectral processing chip can be designed and constructed based on a filter layer and an image sensor. For example, the image sensor can be a CMOS image sensor (CIS), CCD, etc. In the application scenario of image analysis based on a neural network model algorithm, the filter layer and image sensor together constitute an optical convolutional layer (OCL) of the neural network model, which has the ability to perform integrated sensing and computing on natural spectral images.

[0070] In the present invention, the filter layer is composed of a plurality of modulation units arranged in an array. These modulation units are formed in the optical path of the image sensor. Preferably, the modulation units are aligned with and attached to the surface of the pixel units in the image sensor. Because the modulation units are used for light filtering, they can serve as filter structural units.

[0071] This natural spectrum image serves as the input to the optical computation convolution layer. The modulation unit of the filter layer and the CMOS image sensor array jointly perform an optical analog calculation of the inner product of the feature vectors of the input natural spectrum image. The calculation result is converted into an electrical signal, which is the spectral image feature output by the optical computation convolution layer. This spectral image feature serves as the input to the later layers of the neural network. Integrating artificial intelligence algorithms such as multilayer perceptrons and convolutional neural networks in these later layers enables real-time decoding processing on platforms such as processors.

[0072] The optical analog calculation of the inner product of the eigenvectors of an input natural spectrum image can be performed using either coherent or incoherent light illumination. The intensity distribution of the incident light on the spatial light modulation plane is equivalent to the input vector value. Different pixels of the spatial light modulation are encoded according to the weighting coefficients. The light beam is sequentially passed through the spatial light modulation and lens to a focal point. A detector is placed at the lens focal point and the total light intensity on the spatial light modulator plane is collected. The result is equivalent to the inner product between the input vector and the weighting coefficient vector.

[0073] In the present invention, further, each modulation unit includes at least one micro-nanostructure, which can be implemented as a modulation hole, nanopillar, nanowire, etc. Specifically, the modulation unit can be a structure or material with filtering properties, such as a metasurface, a modulation hole, a nanopillar, a nanowire, a photonic crystal, a multilayer film, a dye, a quantum dot, a MEMS (micro-electromechanical system), an FP etalon, a cavity layer, a waveguide layer, or a diffraction element.

[0074] Furthermore, the micro-nano structure of the modulation unit can be represented by a pair of parameters (period parameter and shape parameter). In order to comprehensively consider the application objects of various application scenarios, the micro-nano structure can be optimized and adjusted. Figure 2 Schematic diagram of the designed micro-nano structure with different structural parameters.

[0075] In the present invention, the micro-nanostructure of the modulation unit can be represented by a pair of parameters. Taking into account the application objects in different scenarios, optimizing and adjusting the parameters of the micro-nanostructure can determine the specific form of the modulation unit, and thus construct the specific form of the convolution kernel of the filter layer and the calculation unit. To this end, the present invention first determines the specific form of the modulation unit.

[0076] To design modulation units, this paper proposes an application-oriented discrete topology optimization algorithm for micro-nanostructures. This algorithm represents the modulation unit's micro-nanostructure via a pair of parameters, comprehensively considering the application objective, feature extraction capability, and tolerance for processing errors. The algorithm then uses a gradient descent method to determine the parameters of the modulation unit's micro-nanostructure. To achieve this, an objective function for the modulation unit design is constructed. This objective function characterizes the mapping between the structural parameters of each modulation unit's micro-nanostructure and the requirements of the target application scenario. By optimally solving the objective function, the parameters of each modulation unit's micro-nanostructure are determined.

[0077] Since the modulation unit is aimed at the application goals in a specific application scenario, it is necessary to obtain a natural spectrum image sample set in the target application scenario. This sample set is used as a test sample. With the sample set as data support, the constructed objective function is optimized and solved to determine the structural parameters of the micro-nano structure of the modulation unit.

[0078] A modulation unit is constructed based on the structural parameters of the determined micro-nanostructure of the modulation unit, and a spectrum processing chip is generated based on the constructed modulation unit and image sensor. In the present invention, the purpose of constructing the spectrum processing chip is to enable the modulation unit and image sensor array to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectrum image to obtain spectral image characteristics.

[0079] In the further chip of the above-mentioned spectral processing chip, the specific structure of the chip is mainly explained. The filter layer includes multiple convolution operation units arranged in an array, each convolution operation unit includes multiple convolution kernels arranged in an array, and each convolution kernel includes multiple modulation units (i.e., filter structure units) arranged in an array. It should be noted that the effect of the array arrangement is to ensure that the multiple convolution operation units can be reasonably distributed. The preferred array arrangement is: each convolution kernel includes m×m modulation units, each convolution operation unit includes n×n convolution kernels, and the filter layer includes H×W convolution operation units.

[0080] In this regard, it should be noted that in the present invention, see Figure 3 The basic unit of the filter layer is the modulation unit ( Figure 3 Taking the metasurface as an example, each modulation unit is aligned with at least one pixel unit of the CIS and attached to the surface of the pixel unit. Assume that the transmission spectrum of the modulation unit is h i (λ), the incident natural spectrum is x i (λ), then the spectrum of the transmitted light is x i (λ)h i (λ), the electrical signal generated by the CIS below the modulation unit can be calculated by the following formula:

[0081] I i =∫x i (λ)h i (λ)dλ=x i ·h i

[0082] It can be seen that the output electrical signal is actually x i ,h i The inner product of the two, the modulation unit and CIS actually jointly complete the optical simulation calculation of the vector inner product.

[0083] A convolution kernel consists of M=m×m modulation units. The CIS under each modulation unit will produce an inner product calculation result I i , then the calculation result of the convolution kernel v k The sum of these M electrical signals can be obtained:

[0084]

[0085] The optical filter layer usually has multiple convolution kernels. Arranging N=n×n convolution kernels together constitutes an optical convolutional unit (OCU). Therefore, an OCU with N convolution kernels generates N outputs after calculation: v1, v 2,…, v N Arranging a large number of OCUs in an H×W array at different spatial positions constitutes a complete OCL (optical computation convolution layer, i.e., filter layer). The OCUs at different spatial positions implement optical parallel convolution operations. Let the calculation result of the kth convolution kernel of the OCU at position (h, w) be v (h,w)k (1≤h≤H,1≤w≤W,1≤k≤N), then the spectral image feature output after OCL calculation is F={v (h,w)k}∈R H×W×N .

[0086] Figure 4A schematic diagram showing a process flow of an optimization method for a spectrum processing chip provided by the present invention is shown. Figure 4 , the method comprising:

[0087] 41. Construct an objective function for designing a modulation unit. The objective function represents the mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario. For example, the structural parameters may be the period parameter p and shape parameter q of the micro-nanostructure.

[0088] 42. Optimize and solve the objective function based on the natural spectrum image sample set in the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit;

[0089] 43. Construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generate a spectrum processing chip according to the constructed modulation unit and the image sensor.

[0090] The micro-nanostructure can be optimized and adjusted, specifically by constructing an objective function for designing the modulation unit, where the objective function represents the mapping relationship between the structural parameters of the micro-nanostructure of each modulation unit and the requirements of the target application scenario; for example, the structural parameters can be the period parameter p and the shape parameter q of the micro-nanostructure;

[0091] The objective function is optimized and solved based on the natural spectrum image sample set in the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit.

[0092] The main purpose is to explain the process of constructing the objective function of the designed modulation unit, as follows:

[0093] In the present invention, the natural spectrum image sample set includes natural spectrum image samples of various categories, and the structural parameters include period parameters and shape parameters.

[0094] Construct a first loss function that characterizes the variance relationship of the measurement results of samples of different categories, as well as the variance relationship of the measurement results of samples of the same category. It is used to ensure that the intra-class variance is as small as possible and the inter-class variance is as large as possible.

[0095] A second loss function is constructed, which characterizes the correlation of the transmission spectrum matrix of the modulation unit. The correlation includes row correlation or column correlation, which determines the perception ability of the group of filter structure units in the spectral dimension.

[0096] A third loss function is constructed, which characterizes the period similarity and shape similarity of the modulation unit, which determines the tolerance to the processing error of the micro-nano structure.

[0097] An objective function is constructed based on the first loss function, the second loss function, and the third loss function.

[0098] Furthermore, the first loss function includes:

[0099]

[0100]

[0101] Among them, L fas (p1,q1,…,p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

[0102] The second loss function includes:

[0103]

[0104] Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

[0105] The third loss function includes:

[0106]

[0107]

[0108] Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

[0109] The objective function includes:

[0110] L total =αLfas +βL corr +γL fab

[0111]

[0112] Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

[0113] Furthermore, a spectrum processing chip is generated based on the constructed modulation unit and image sensor, including:

[0114] Arrange the modulation units in an m×m array to obtain the convolution kernel;

[0115] Arrange the convolution kernels in an n×n array to obtain a convolution operation unit;

[0116] Arrange the convolution operation units in an H×W array to obtain a filter layer;

[0117] Align the filter layer and image sensor to create a spectral processing chip.

[0118] Continue to see Figure 3 The parameters of the micro-nanostructures within a modulation unit may be the same or different, and the parameters of the micro-nanostructures within different modulation units may be the same and / or different. By modifying the structure and parameters of the filter layer, the present invention can conveniently implement a variety of complex machine vision tasks based on hyperspectral images without changing the physical structure of the chip, enabling real-time hyperspectral image classification, hyperspectral target detection, and hyperspectral instance segmentation.

[0119] Furthermore, an OCU has nm×nm pixels, which is composed of n×n convolution kernels, and the size of each convolution kernel is m×m. When the product of n and m is a constant, the number and size of the convolution kernels can be changed dynamically, such as Figure 5 As shown. However, n and m must satisfy an inverse relationship, that is, the more convolution kernels there are, the smaller the size of the convolution kernel. However, by using the modulation unit multiplexing scheme, the number of convolution kernels in an OCU can be greatly increased, see Figure 6 , can have at most (nm) 2 The convolution kernel. Figure 6 The convolution kernel size and number can be dynamically changed. a) The convolution kernel size is 6×6; b) The convolution kernel size is 2×2; c) The convolution kernel size is 3×3. This invention adapts to different application scenarios by dynamically changing the convolution kernel data and size.

[0120] For example, in face anti-counterfeiting applications, a 3×3 convolution kernel with a total of 9 kernels is selected to consider the robustness under different lighting conditions; when detecting the sugar content of specific fruits (such as grapes), a 2×2 convolution kernel with a total of 4 kernels is selected.

[0121] The present invention can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time. For the spatial information, that is, the spatial light energy distribution of the image, based on the convolutional neural network architecture, using its shared weight characteristics and the advantages of the metasurface for large-scale array integration, the spectral spatial resolution can be achieved to 10 6 This level of resolution can meet the spatial resolution requirements of convolutional neural networks in most machine vision tasks. For frequency domain information, through feature matching and precise design of the metasurface, low-power, light-speed hyperspectral feature extraction is achieved, compressing hundreds of channels in hyperspectral images to just a few or a dozen channels. While utilizing optical metasurfaces to significantly reduce the difficulty of hyperspectral data detection and acquisition, it also compresses the frequency domain data volume, further improving the processing capabilities for complex application scenarios.

[0122] In the application scenario of spectral anti-counterfeiting face recognition, the present invention adopts n=m=3, H=160, W=122, and adopts metasurface technology to realize the modulation unit. The optimized 9 different metasurfaces are as follows: Figure 3 The corresponding transmission spectrum and its matching with the characteristic absorption peak of hemoglobin are shown in Figure 7 In the process of processing and preparation, based on SOI (Silicon-On-Insulator, silicon on insulating substrate), the PDMS imprint transfer scheme ( Figure 8 ) is used to prepare OCL, and the filter layer (i.e., optical computing convolution layer) network structure is as follows Figure 9 , the accuracy of pixel-level face anti-counterfeiting is 96%, and the accuracy of image-level face anti-counterfeiting is 100%. Figure 10 shown.

[0123] In this embodiment, the metasurface convolution layer can perform direct frequency domain operations on natural spectrum images, and combine with the CIS and processor to realize parallel computing and processing of large-scale complex hyperspectral machine vision tasks, that is, sensing and computing in one; the freeform shaped meta-atoms metasurface based on feature matching and optimized design can obtain hyperspectral visual information in real time and reduce feature dimensionality, thereby improving the real-time processing capability of the optical neural network for complex application scenario tasks.

[0124] At the same time, large-scale arrays of metasurfaces are conducive to expanding the number of convolution kernels in the optical convolution layer and improving the optical neural network's ability to process spatial optical information, thereby realizing complex task processing; the preparation and integration of metasurfaces can adopt CMOS-compatible processes, which is conducive to the large-scale production of optical neural network chips.

[0125] The proposed application-oriented nanostructure discrete topology optimization algorithm is used to realize spectral recognition of thyroid pathological section microscopic images, where n=m=3, H=160, W=120, and metasurface technology is used to realize the filtering structure unit. The optimized design of 9 different metasurface structures is as follows Figure 11 The corresponding metasurface transmission spectrum is shown as Figure 12 As shown in Figure 2, the spectral characteristics of microscopic images of slices in different pathological states are as follows: Figure 13 As shown in the figure, the spectral feature clustering results are as follows: Figure 14 As shown in the figure, the spectral convolutional neural network can recognize microscopic images of slices with different pathological conditions. Figure 15 The final recognition result is shown as Figure 16 shown.

[0126] The optimization method of the spectral processing chip provided by the present invention constructs a spectral processing chip, so that multiple modulation units and image sensor arrays in the chip jointly perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis. It can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the characteristic information efficient and rapid, so that it can be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities to the devices.

[0127] Figure 17 A schematic diagram of the structure of a spectrum processing chip provided by the present invention is shown. Figure 17 The spectrum processing chip includes a filter layer 171 and an image sensor array 172, wherein:

[0128] The filter layer 171 is composed of multiple modulation units 1711, which are attached to the surface of the image sensor 172. The multiple modulation units 1711 and the image sensor 172 work together to perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectrum image to obtain spectral image characteristics.

[0129] The following describes the optimization device for the spectrum processing chip provided by the present invention. The optimization device for the spectrum processing chip described below and the optimization method for the spectrum processing chip described above can be referred to in correspondence with each other.

[0130] Figure 18 The present invention provides an optimization device for a spectrum processing chip, which includes a construction module 181, an optimization module 182, and a generation module 183, wherein:

[0131] A construction module 181 is used to construct an objective function for designing a modulation unit, where the objective function represents a mapping relationship between structural parameters of the micro-nano structure of each modulation unit and requirements of a target application scenario;

[0132] An optimization module 182 is configured to optimize and solve an objective function based on a natural spectrum image sample set in a target application scenario to determine structural parameters of the micro-nano structure of the modulation unit;

[0133] A generating module 183 is configured to construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and to generate a spectrum processing chip according to the constructed modulation unit and the image sensor;

[0134] The modulation unit and the image sensor are used to jointly perform optical simulation calculation of the inner product of the feature vector of the incident natural spectrum image to obtain the spectrum image characteristics.

[0135] The optimization device for the spectral processing chip provided by the present invention constructs a spectral processing chip, so that multiple modulation units and image sensors in the chip jointly perform optical simulation calculations of the inner product of the characteristic vectors of the incident natural spectral image, obtain spectral image features and perform subsequent analysis. It can directly detect, acquire and perceive visual light information in the spatial and frequency domains in real time, making the subsequent analysis of the characteristic information efficient and rapid, so that it can be easily deployed on various mobile devices and edge devices, introducing hyperspectral perception capabilities to the devices.

[0136] Figure 19 An example of a physical structure diagram of an electronic device is shown below. Figure 19 As shown, the electronic device may include: a processor 191, a communications interface 192, a memory 193, and a communications bus 194, wherein the processor 191, the communications interface 192, and the memory 193 communicate with each other via the communications bus 194. The processor 191 may call logic instructions in the memory 193 to execute a method for optimizing a spectral processing chip, the method comprising: constructing an objective function for designing a modulation unit, the objective function representing a mapping relationship between structural parameters of the micro-nano structure of each modulation unit and the requirements of a target application scenario; optimizing and solving the objective function based on a natural spectrum image sample set under the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; constructing a modulation unit based on the determined structural parameters of the micro-nano structure of the modulation unit, and generating a spectral processing chip based on the constructed modulation unit and an image sensor.

[0137] In addition, the logic instructions in the above-mentioned memory 193 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the optimization method of the spectral processing chip provided by the above methods, which includes: constructing an objective function for designing a modulation unit, the objective function characterizing the mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario; optimizing and solving the objective function based on a natural spectrum image sample set under the target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; constructing a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generating a spectral processing chip based on the constructed modulation unit and the image sensor.

[0139] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for optimizing a spectral processing chip provided by the above-mentioned methods, the method comprising: constructing an objective function for designing a modulation unit, the objective function characterizing a mapping relationship between structural parameters of the micro-nano structure of each modulation unit and the requirements of a target application scenario; optimizing and solving the objective function based on a natural spectrum image sample set under the target application scenario, and determining the structural parameters of the micro-nano structure of the modulation unit; constructing a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generating a spectral processing chip based on the constructed modulation unit and the image sensor.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0141] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for optimizing a spectrum processing chip, characterized in that: include: Constructing an objective function for designing a modulation unit, wherein the objective function represents a mapping relationship between structural parameters of the micro-nanostructure of each modulation unit and requirements of a target application scenario; Optimizing and solving the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; A modulation unit is constructed according to the determined structural parameters of the micro-nano structure of the modulation unit, and a spectrum processing chip is generated according to the constructed modulation unit and the image sensor.

2. The method for optimizing a spectrum processing chip according to claim 1, wherein: The modulation unit and the image sensor are used to jointly perform an optical simulation calculation of the inner product of the feature vectors of the incident natural spectrum image to obtain the spectrum image features.

3. The method for optimizing a spectrum processing chip according to claim 1, wherein: The natural spectrum image sample set includes natural spectrum image samples of various categories, and the structural parameters include period parameters and shape parameters. Accordingly, constructing an objective function for designing a modulation unit includes: Constructing a first loss function, wherein the first loss function represents the variance relationship of the measurement results of samples of different categories and the variance relationship of the measurement results of samples of the same category; Constructing a second loss function, wherein the second loss function characterizes the correlation of the transmission spectrum matrix of the modulation unit; Constructing a third loss function, wherein the third loss function represents the period similarity and shape similarity of the modulation unit; The objective function is constructed based on the first loss function, the second loss function and the third loss function.

4. The method for constructing a spectrum processing chip according to claim 2, wherein: The first loss function includes: Among them, L fas (p1,q1,…,p N ,q N ) is the first loss value, s i is the variance of the measurement results of all samples of category i, X i is the vector of measurement results of all samples of category i, is the transmission spectrum of the kth modulation unit, m i is the mean of the measurement results of all samples of category i, m0 is the mean of the measurement results of all samples of all categories, the period parameter p and shape parameter q of the micro-nanostructure.

5. The method for optimizing a spectrum processing chip according to claim 4, wherein: The second loss function includes: Among them, L corr is the correlation of the transmission spectrum matrix of the modulation unit, is the transmission spectrum of the j-th modulation unit, is the transpose of the transmission spectrum of the i-th modulation unit.

6. The method for constructing a spectrum processing chip according to claim 5, characterized in that: The third loss function includes: Among them, L fab is the similarity of the modulation unit, is the similarity between the i-th modulation unit and the j-th modulation unit, s(q i ) and s(q j ) respectively represent the i-th modulation unit The jth modulation unit The shape is serialized as a 1D vector, abs means taking the absolute value, p i and p j are the period parameters of the i-th modulation unit and the j-th modulation unit respectively.

7. The method for optimizing a spectrum processing chip according to claim 6, wherein: The objective function includes: L total =αL fas +βL corr +γL fab Among them, L total is the target value, α, β, γ are coefficients, and k is the number of modulation units to be determined.

8. The method for optimizing a spectrum processing chip according to claim 1, wherein: The spectrum processing chip is generated based on the constructed modulation unit and image sensor, including: Arranging the modulation units in an m×m array to obtain a convolution kernel; Arranging the convolution kernels in an n×n array to obtain a convolution operation unit; Arranging the convolution operation units in an H×W array to obtain a filter layer; The filter layer and the image sensor are aligned to form a spectrum processing chip.

9. The method for optimizing a spectrum processing chip according to claim 8, wherein: A metasurface is used to construct a modulation unit, wherein a plurality of modulation units are used to construct a filter layer with n=m=3, H=160, and W=120.

10. An optimization device for a spectrum processing chip, characterized in that: include: A construction module is used to construct an objective function for designing a modulation unit, where the objective function represents the mapping relationship between the structural parameters of the micro-nano structure of each modulation unit and the requirements of the target application scenario; An optimization module is used to optimize and solve the objective function based on a natural spectrum image sample set in a target application scenario to determine the structural parameters of the micro-nano structure of the modulation unit; The generation module is used to construct a modulation unit according to the determined structural parameters of the micro-nano structure of the modulation unit, and generate a spectrum processing chip according to the constructed modulation unit and the image sensor.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the optimization method of the spectrum processing chip according to any one of claims 1 to 9 is implemented.

Images