Spectral convolutional neural network processing system, method and device

Abstract

The invention provides a spectral convolutional neural network processing system, method and device, and the system comprises a filtering structure which covers the surface of an image sensor, and each filtering unit in the filtering structure corresponds to a pixel point of a pixel array in the image sensor; different light filtering units composed of different pigments in the light filtering structure have different wide-spectrum light filtering responses; the filtering structure is used for modulating incident light to obtain a spectral vector of modulated transmission light; the image sensor is used for carrying out parallel detection and photoelectric conversion processing on the spectrum vector of the modulated transmission light and then outputting an electric signal; and sending the electric signal to a processor module for preset summation processing to obtain a feature map, and inputting the feature map to a neural network model for processing to obtain an output result of the neural network model. The system provided by the invention does not depend on a coherent light source, can directly carry out processing by utilizing continuous spectrum dimensions, and effectively improves the processing efficiency of the system.

Description

Technical Field

[0001] The present invention relates to the field of visual perception technology, and in particular to a spectral convolutional neural network processing system, method and device, and also to an electronic device, a non-transitory computer-readable storage medium and a computer program product. Background Art

[0002] At present, RGB (Red, Green, Blue) color cameras are the most widely used visual perception devices, which are composed of grayscale cameras and red, green, and blue filters. RGB color cameras are designed to imitate the color vision of the human eye, and do not have the function of light processing, nor are they specially designed to achieve light processing. One of the key technologies of optical neural networks (spectral convolutional neural networks) is the optical processing scheme of matrix-vector multiplication. Existing optical neural networks generally require coherent light to be implemented, and can be divided into two categories: spatial optical path schemes and on-chip integration schemes. The spatial optical path scheme is generally implemented using diffractive-Deep-Neural-Networks (D2NN) or vertical-cavity surface laser (VCSEL) arrays. The diffractive neural network uses multiple diffraction templates to superimpose to achieve matrix-vector multiplication calculations, and the VCSEL array needs to be combined with spatial light modulators to achieve parallel processing of light. On-chip integration solutions generally use resonant structures such as micro-ring cavities or diffraction structures such as Mach-Zehnder interferometers to modulate light to achieve the function of scalar multiplication, and then achieve parallel processing through wavelength division multiplexing and space division multiplexing to achieve vector inner products and matrix-vector multiplication. However, existing optical neural network solutions all rely on coherent light sources. Both spatial optical path solutions and on-chip integration solutions need to first encode information into coherent light, and then use optical devices to modulate the coherent light to achieve the processing function. Visual perception devices such as single RGB color cameras do not have the function of light processing, and existing optical neural network technologies do not have the function of visual perception. RGB cameras generally use red, green, and blue dyes or pigment filters to simulate the color vision of the human eye, and do not have the functions of optical modulation and optical matrix-vector multiplication calculations.

[0003] Existing optical neural network technology relies on coherent light sources and cannot directly process wide-spectrum natural light. Therefore, in order to process visual light field information, it is necessary to first use an RGB color camera to sample the visual light field information into a digital signal, then encode the digital signal onto coherent light and perform light processing, and finally use photodiodes and other photoelectric conversion devices to convert the optical signal into an electrical signal to obtain the final processing result. In this processing architecture, it is necessary to perform photoelectric / electro-optical conversion multiple times, making the overall structure of the system complex and the performance low. In addition, the dependence of optical neural networks on coherent light sources makes it impossible to directly use continuous spectral dimensions for processing. These defects limit the scale and practical application of optical neural networks, especially in the processing of visual image information. Summary of the invention

[0004] The present invention provides a spectral convolutional neural network processing system, method and device to solve the defect that the optical neural network processing scheme in the prior art has high limitations, resulting in poor system processing efficiency in practical applications.

[0005] The present invention provides a spectral convolutional neural network processing system, comprising: a filter structure and an image sensor; the filter structure is a filter that modulates incident light in the frequency domain dimension as a whole within a wide spectrum band; the filter structure covers the surface of the image sensor, and each filter unit in the filter structure corresponds to a pixel point of a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses; The filter structure is used to modulate the incoherent wide-spectrum natural light incident on each pixel point of the pixel array in the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the spectral vector of the modulated transmitted light is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the incident light is incoherent wide-spectrum natural light; The image sensor is used to perform parallel detection and photoelectric conversion processing on the spectral vector of the modulated transmitted light based on each pixel point in the pixel array, and then output an electrical signal; send the electrical signal to the processor module for preset summation processing to obtain a corresponding feature map; and input the feature map into a preset neural network model for processing to obtain an output result of the neural network model; Wherein, the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filtering structure; the neural network model is an artificial neural network model.

[0006] The spectral convolutional neural network processing system according to the present invention also includes: a microlens; a microlens is correspondingly arranged above each filtering unit in the filtering structure, and the microlens is used to focus the incident light of each filtering unit to improve the efficiency of the pixels in the image sensor sensing the incident light of each filtering unit.

[0007] The present invention also provides a spectral convolutional neural network processing method, comprising: The incident light is modulated by using a filter structure preset on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the filter structure comprises a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filter responses; After obtaining the spectral vector of the modulated transmitted light based on the image sensor for parallel detection and photoelectric conversion processing, a corresponding electrical signal is obtained, and the electrical signal is sent to the processor module for preset summation processing to obtain a corresponding feature map, and the feature map is input into a preset neural network model for processing to obtain the output result of the neural network model; the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is an artificial neural network model.

[0008] According to the spectral convolutional neural network processing method of the present invention, the incident light is modulated by using a filter structure preset on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light, including: The transmitted light focused by the microlens is modulated by using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the microlens is used to focus the incident light; the incident light is incoherent wide-spectrum natural light.

[0009] According to the spectral convolutional neural network processing method of the present invention, the spectral vector of the modulated transmitted light obtained by the image sensor is subjected to parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal, which specifically includes: Based on the image sensor, perception calculation is performed on the incident spectral image. Each pixel point of the image sensor is used to perceive the spectral vector of the superpixel at the corresponding spatial position in the incident spectral image, so as to calculate the inner product of the spectral vector and the transmission response vector of the spectral modulation unit on the pixel surface of the image sensor to obtain the corresponding electrical signal.

[0010] According to the spectral convolutional neural network processing method of the present invention, the electrical signal is sent to the processor module for a preset summation process to obtain a corresponding feature map, including: The electrical signal is sent to a preset processor module, and the processor module performs a summation operation on the electrical signal to obtain a feature map of the convolution kernels in the optical computing convolution unit.

[0011] The present invention also provides a spectral convolutional neural network processing device, comprising: A modulation module, used for modulating incident light using a preset filter structure on the surface of an image sensor to obtain a spectral vector of the modulated transmitted light; wherein the filter structure comprises a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filter responses; the microlens is used for focusing the incident light; the incident light is incoherent wide-spectrum natural light; A processing module is used to obtain a spectral vector of the modulated transmitted light based on the image sensor, perform parallel detection and photoelectric conversion processing, obtain a corresponding electrical signal, send the electrical signal to the processor module for preset summation processing, obtain a corresponding feature map, and input the feature map into a preset neural network model for processing to obtain an output result of the neural network model; wherein the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; and the neural network model is an artificial neural network model.

[0012] According to the spectral convolutional neural network processing device of the present invention, the incident light is modulated by using a filter structure preset on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light, including: The transmitted light focused by the microlens is modulated by using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the microlens is used to focus the incident light; the incident light is incoherent wide-spectrum natural light.

[0013] According to the spectral convolutional neural network processing device of the present invention, the spectral vector of the modulated transmitted light obtained by the image sensor is subjected to parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal, which specifically includes: Based on the image sensor, perception calculation is performed on the incident spectral image. Each pixel point of the image sensor is used to perceive the spectral vector of the superpixel at the corresponding spatial position in the incident spectral image, so as to calculate the inner product of the spectral vector and the transmission response vector of the spectral modulation unit on the pixel surface of the image sensor to obtain the corresponding electrical signal.

[0014] According to the spectral convolutional neural network processing device of the present invention, the step of sending the electrical signal to the processor module for a preset summation process to obtain a corresponding feature map includes: The electrical signal is sent to a preset processor module, and the processor module performs a summation operation on the electrical signal to obtain a feature map of the convolution kernels in the optical computing convolution unit.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the spectral convolutional neural network processing method as described in any one of the above items is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the spectral convolutional neural network processing method as described in any of the above items is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the spectral convolutional neural network processing method as described in any one of the above items.

[0018] The spectral convolutional neural network processing system provided by the present invention comprises a filter structure and an image sensor; the filter structure covers the surface of the image sensor, and each filter unit in the filter structure corresponds to a pixel point of a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filter responses; the filter structure is used to modulate the incident light to obtain the spectral vector of the modulated transmitted light; the image sensor is used to perform parallel detection and photoelectric conversion processing on the spectral vector of the modulated transmitted light and then output an electrical signal; the electrical signal is sent to a processor module for processing to obtain a feature map, and the feature map is input into a neural network model for processing to obtain an output result of the neural network model, which no longer relies on a coherent light source and can be directly processed using a continuous spectral dimension, thereby effectively improving the efficiency of system processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a structural schematic diagram of the spectral convolutional neural network processing system provided by the present invention.

[0021] Figure 2 It is a flow chart of the spectral convolutional neural network processing method provided by the present invention.

[0022] Figure 3 It is a schematic diagram of the calculation principle of the optical computing convolution layer provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the spectral convolutional neural network processing device provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In order to solve the deficiencies in the existing technical solutions, the purpose of the present invention is to propose a neural network solution that integrates sensing and computing. A variety of pigments or dyes can be used to prepare filters, and can be mass-produced through integrated circuit processes. RGB color cameras are composed of ordinary monochrome image sensors and color filter arrays (Bayer Filter Array, CFA) made of pigments or dyes. The present invention proposes to design a wide-spectrum filter structure based on pigments and combine it with an image sensor to obtain a spectral convolutional neural network processing system to realize a light computing convolution layer that can be mass-produced. The difference from the ordinary RGB color camera is that the wide-spectrum filter structure proposed in the present invention does not filter out red, green, and blue light, but modulates the light in the frequency domain dimension as a whole within a wide spectrum band to realize the function of light computing. In addition, the number of different wide-spectrum filter structures on the light computing convolution layer is not limited to 3. Combined with some electrical neural network structures (such as neural network models for realizing different functions), a hybrid optoelectronic convolutional neural network processing system, that is, a spectral convolutional neural network processing system, can be realized. The spectral convolutional neural network processing system can get rid of the dependence of optical computing on coherent light and directly process natural light field information, that is, directly process the incident wide-spectrum natural light. And while processing, it completes the perception of the incident wide-spectrum natural light, only requires one photoelectric conversion and can simultaneously realize the two functions of visual perception and optical vector inner product calculation. The spectral convolutional neural network processing system can also solve the problem that existing optical computing solutions rely on coherent light and multiple photoelectric / electro-optical conversions, and solve the problem that existing image sensor devices cannot perform optical computing while perceiving.

[0027] Combine the following Figure 1-Figure 5 The spectral convolutional neural network processing system, method and device of the present invention are described, and its embodiments are described in detail.

[0028] Based on the spectral convolutional neural network processing system of the present invention, the embodiments are described in detail below. Figure 1As shown, it is a structural schematic diagram of the spectral convolutional neural network processing system provided by the present invention, and the specific implementation process includes the following parts: a filter structure 101, an image sensor 102 and a microlens 103. The filter structure 101 is a filter that modulates the incident light in the frequency domain dimension as a whole within a wide spectrum band; the filter structure 101 covers the surface of the image sensor 102, and each filter unit in the filter structure 101 corresponds to a pixel point of the pixel array in the image sensor 102; different filter units composed of different pigments in the filter structure 101 have different wide-spectrum filtering responses. Different filter units are prepared based on different pigments or dyes. The filter structure 101 is used to modulate the incoherent wide-spectrum natural light incident on each pixel point of the pixel array in the image sensor 102 to obtain the spectral vector of the modulated transmitted light (i.e., the incident spectral image); wherein the spectral vector of the modulated transmitted light is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure 101; and the incident light is incoherent wide-spectrum natural light. The image sensor 102 is used to perform parallel detection and photoelectric conversion processing on the spectral vector of the modulated transmitted light based on each pixel point in the pixel array, and then output an electrical signal 105; send the electrical signal 105 to the processor module for preset summation processing to obtain a corresponding feature map; and input the feature map into a preset neural network model for processing to obtain the output result of the neural network model. The electrical signal 105 is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure 101.

[0029] The neural network model is an artificial neural network model, such as a fully connected neural network, a convolutional neural network, etc., to achieve functions such as image classification, image segmentation, and target detection.

[0030] It should be noted that the spectral convolutional neural network processing system of the present invention can be applied to different scenarios, such as: detection of spectral information: spectral pathological diagnosis (pathological diagnosis of crops, at this time the output result is which type of disease), spectral live target detection (identification of true and false fingerprints, detection of true and false faces, at this time the output result is whether the target is true or false), thereby realizing the combination of optical convolutional neural network and convolutional neural network models in electronics (such as convolutional neural network, fully connected neural network), and improving the efficiency of system processing.

[0031] Furthermore, the spectral convolutional neural network processing system may also be provided with a microlens 103 correspondingly above each filter unit in the filter structure 101, and the microlens 103 is used to focus the incident light 104 of each filter unit to improve the efficiency of the pixel points in the image sensor 102 in sensing the incident light of each filter unit.

[0032] In an embodiment of the present invention, an optical computing convolution layer based on a filter structure 101 prepared from multiple pigments and an image sensor 102 is proposed, and an optoelectronic hybrid spectral convolution neural network processing system is further implemented in combination with an electrical computing convolution layer.

[0033] The overall structure of the optical computing convolutional layer is shown in the attached figure. Figure 1 As shown, it is mainly composed of a filter structure prepared from multiple pigments and an image sensor. Each filter unit in the filter structure (such as Figure 3 middle , , … … … Different filter units) are associated with a pixel of an image sensor (such as Figure 3 middle The pixels in the image sensor area are aligned and covered on the surface of the image sensor. Different filter units composed of different pigments have different wide-spectrum filter responses. A microlens can also be added above each filter unit to focus the incident light and improve the efficiency of sensing light.

[0034] The input of the optical computing convolutional layer is the incident incoherent wide-spectrum natural light, which has different spectra at different spatial positions, that is, at different pixel points. For the wide-spectrum natural light incident on each pixel, it is first optionally focused by a microlens to improve the detection efficiency. Then it is modulated by the filter structure covering the pixel. The spectrum of the modulated transmitted light (that is, the spectrum vector of the modulated transmitted light) is the spectrum vector of the incident light. and the transmission response vector of the filter structure The modulated transmitted light is then detected by the pixel point and converted into an electrical signal. , the output electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, that is, . Mathematically, it is equal to the inner product of the spectral vector of the incident light and the transmission response vector of the multi-pigment filter structure, that is, Since the image sensor is composed of a large number of pixel arrays, these arrayed pixels are combined with the filter units in the arrayed multi-pixel filter structure to realize large-scale parallel calculation of the inner product of the optical vector. The filter units are arranged in an array in the filter structure. It should be noted that Figure 1 101 is a filter structure prepared by multiple pigments, 102 is an image sensor, 103 is an array of microlenses corresponding to each filter unit in the filter structure, 104 is incident light (i.e., incoherent wide-spectrum natural light), and 105 is an electrical signal. , i =1, 2, 3, 4…N.

[0035] Compared with existing image sensors, the spectral convolutional neural network processing system provided by the present invention not only has the function of image sensing, but also has the perception and computing capabilities of spectral dimensions. The spectral convolutional neural network is an integrated sensing and computing processing device for wide-spectrum natural light images. It can extract the spatial dimension features and spectral dimension features of the image while visually perceiving the spectral image, greatly reducing the amount of calculation of subsequent electrical neural networks, and realizing sensing and computing integration and spectral perception on edge devices with limited computing power. Compared with existing optical neural network solutions, the solution proposed by the present invention does not rely on coherent light sources, can directly perform calculations using continuous spectral dimensions, and only needs to perform one photoelectric conversion. The overall structure of the system is simple and can be easily integrated into various portable devices.

[0036] The spectral convolutional neural network structure (i.e., spectral convolutional neural network processing system) described in the present invention utilizes an image sensor combined with a filter structure to realize the integrated sensing and spectral perception of incident wide-spectrum natural light. By utilizing multiple pigments to realize the filter structure, a variety of different filter responses can be achieved by mixing multiple pigments, and integrated circuit process flow can be used for production. It should be noted that the ability of the spectral convolutional neural network structure to extract spectral dimension features is affected by the richness of the transmission response of the filter structure used. By using richer pigments or dyes to form a more complex transmission response, a more powerful spectral convolutional neural network can be achieved.

[0037] The spectral convolutional neural network processing method provided by the present invention is described below. The spectral convolutional neural network processing method described below and the spectral convolutional neural network processing system described above can be referred to each other. Figure 2 As shown, it is a flow chart of the spectral convolutional neural network processing method provided by the present invention, and the specific implementation process includes the following steps: Step 201, modulate the incident light using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the filter structure includes a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in the pixel array of the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses.

[0038] Specifically, the transmitted light focused by the microlens can be modulated using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the microlens is used to focus the incident light; the incident light is incoherent wide-spectrum natural light.

[0039] Step 202: After obtaining the spectral vector of the modulated transmitted light based on the image sensor and performing parallel detection and photoelectric conversion processing, a corresponding electrical signal is obtained, and the electrical signal is sent to the processor module for preset summation processing to obtain a corresponding feature map, and the feature map is input into a preset neural network model for processing to obtain an output result of the neural network model; the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is an artificial neural network model.

[0040] It should be noted that the spectral convolutional neural network processing system of the present invention can be applied to different scenarios, such as: detection of spectral information: spectral pathological diagnosis (pathological diagnosis of crops, at this time the output result is which type of disease), spectral live target detection (identification of true and false fingerprints, detection of true and false faces, at this time the output result is whether the target is true or false), thereby realizing the combination of optical convolutional neural network and convolutional neural network models in electronics (such as convolutional neural network, fully connected neural network), and improving the efficiency of system processing.

[0041] Specifically, the image sensor performs perception calculation on the incident spectral image, and each pixel in the image sensor is used to perceive the spectral vector of the superpixel at the corresponding spatial position in the incident spectral image, which is equivalent to calculating the inner product of the spectral vector and the transmission response vector of the spectral modulation unit on the pixel surface of the image sensor, thereby obtaining the corresponding electrical signal. The spectral modulation units on the pixel surface of the image sensor at a set of preset positions correspond to a set of transmission vectors, which together constitute the weight of a convolution kernel. By summing the current values ​​output by the pixels of a group of image sensors, a convolution operation result of the convolution kernel at the spatial position (i.e., the corresponding electrical signal obtained) is obtained.

[0042] The electrical signal is sent to a preset processor module, and the processor module performs a summation operation on the electrical signal to obtain a feature map of the convolution kernels in the optical computing convolution unit. Wherein, the optical computing convolution layer is an array structure composed of multiple optical computing convolution units; each optical computing convolution unit is composed of a super pixel array, each super pixel is composed of multiple pixel points, different super pixels correspond to different filter areas in the filter structure, and each filter area contains multiple filter units. In addition, in the process of inputting the feature map into a preset neural network model for processing to obtain the output result of the neural network model, it includes but is not limited to: inputting the feature map into a neural network model for image classification for processing to obtain the target classification result output by the neural network model for image classification; or, inputting the feature map into a neural network model for target detection for processing to obtain the target detection result output by the neural network model for target detection.

[0043] It should be noted that, in the embodiments of the present invention, Figure 3 As shown, each filter unit in the filter structure is completely aligned with a pixel point in the image sensor. The pixels of an image sensor form a superpixel. Superpixels form an optical convolutional unit (OCU). A superpixel is a k*k pixel of an image sensor; a convolutional unit is n*n superpixels, that is, nk*nk pixels. The complete optical convolution layer consists of In the processing of a single optical convolution unit, there are The size is The convolution kernel (for example, Figure 3 As shown in , one of the convolution kernels can be Kernel), a convolution kernel corresponds to multiple super pixels and multiple pixels, and these convolution kernels cover superpixel. ( ) convolution kernels have weight vectors, and the weight vectors are ,in The convolution kernel contains The transmission response vector of the filter structure and the quantum efficiency of image sensors Assume that the light incident on each superpixel is uniform and its spectral vector is , then the calculation result of the convolution kernel is: in, It is the electrical signal output by each pixel in the image sensor. Each optical computation convolution unit has convolution kernels, and the optical computation convolution layer consists of The same light calculation convolution unit is formed; and They are the upper and lower breakpoints of the convolution interval respectively; Represents a feature map output by a convolution kernel. If there are p convolution kernels, p feature maps will be obtained; function = ;function = .

[0044] If in The first convolution unit in the light calculation at position ( The computational output of the convolution kernel is , then the feature map of the final output of the light convolution layer is: Therefore, the optical computation convolution layer has The size and step length are The convolution kernel of the input wide-spectrum natural light image information has a super-pixel number of , then the output feature map size is .

[0045] It can be seen that the optical computing convolution layer can complete the convolution operation of the image while perceiving the wide-spectrum natural light image. The output of its perception calculation is the electrical signal output by the image sensor, which is input to the processor module (Central Processing Unit, CPU) and calculated by the subsequent electrical neural network (i.e., a neural network model with different functions), i.e. , and obtain the final output, and obtain the output result of the neural network model.

[0046] It should be noted that an optical computing convolution unit is composed of The optical computing convolution unit is divided, which is specifically determined by the arrangement of the designed filter units. The optical computing convolution layer and the processor module are independent of each other. The spectral convolution neural network processing system described in the present invention is implemented by an optical computing convolution layer and an electrical computing neural network layer. The optical computing convolution layer corresponds to the filter structure and the image sensor; the electrical computing neural network layer is the neural network model described in the present invention, and the electrical computing neural network layer is calculated on the processor module.

[0047] The spectral convolutional neural network processing method provided by the present invention modulates the incident light by using a preset filter structure on the surface of an image sensor to obtain a spectral vector of the modulated transmitted light; the filter structure comprises a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses; and based on the spectral vector of the modulated transmitted light obtained by the image sensor, parallel detection and photoelectric conversion processing are performed to obtain a corresponding electrical signal, and the electrical signal is sent to a processor module for preset summation processing to obtain a corresponding The electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is an artificial neural network model, which can extract the spatial dimension features and spectral dimension features of the image while visually perceiving the spectral image, greatly reducing the calculation amount of the subsequent electrical neural network, and realizing the integrated sensing and computing and spectral perception on edge devices with limited computing power.

[0048] The spectral convolutional neural network processing device provided by the present invention is described below. The spectral convolutional neural network processing device described below and the spectral convolutional neural network processing method and system described above can be referred to each other. Figure 4 As shown, it is a schematic diagram of the structure of the spectral convolutional neural network processing device provided by the present invention. The spectral convolutional neural network processing device of the present invention specifically includes the following parts: The modulation module 401 is used to modulate the incident light using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the filter structure includes a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in the pixel array of the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses; the microlens is used to focus the incident light; and the incident light is incoherent wide-spectrum natural light.

[0049] The processing module 402 is used to obtain the spectral vector of the modulated transmitted light based on the image sensor, perform parallel detection and photoelectric conversion processing, obtain the corresponding electrical signal, send the electrical signal to the processor module for preset summation processing, obtain the corresponding feature map, and input the feature map into the preset neural network model for processing to obtain the output result of the neural network model; wherein, the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is a neural network model.

[0050] According to the spectral convolutional neural network processing device of the present invention, the incident light is modulated by using a filter structure preset on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light, including: The transmitted light focused by the microlens is modulated by using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the microlens is used to focus the incident light; the incident light is incoherent wide-spectrum natural light.

[0051] According to the spectral convolutional neural network processing device of the present invention, the spectral vector of the modulated transmitted light obtained by the image sensor is subjected to parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal, which specifically includes: Based on the image sensor, perception calculation is performed on the incident spectral image. Each pixel point of the image sensor is used to perceive the spectral vector of the superpixel at the corresponding spatial position in the incident spectral image, so as to calculate the inner product of the spectral vector and the transmission response vector of the spectral modulation unit on the pixel surface of the image sensor to obtain the corresponding electrical signal.

[0052] According to the spectral convolutional neural network processing device of the present invention, the step of sending the electrical signal to the processor module for a preset summation process to obtain a corresponding feature map includes: The electrical signal is sent to a preset processor module, and the processor module performs a summation operation on the electrical signal to obtain a feature map of the convolution kernels in the optical computing convolution unit.

[0053] The spectral convolutional neural network processing device provided by the present invention modulates the incident light by using a preset filter structure on the surface of an image sensor to obtain a spectral vector of the modulated transmitted light; the filter structure comprises a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in the pixel array of the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses; and based on the spectral vector of the modulated transmitted light obtained by the image sensor, parallel detection and photoelectric conversion processing are performed to obtain a corresponding electrical signal, and the electrical signal is sent to a processor module for preset summation processing to obtain a corresponding The electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is an artificial neural network model, which can extract the spatial dimension features and spectral dimension features of the image while visually perceiving the spectral image, greatly reducing the calculation amount of the subsequent electrical neural network, and realizing the integrated sensing and computing and spectral perception on edge devices with limited computing power.

[0054] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device (i.e., a power grid real-time monitoring system or a power grid real-time monitoring device) may include: a processor (processor) 501, a communication interface (Communications Interface) 504, a memory (memory) 502 and a communication bus 503, wherein the processor 501, the communication interface 504, and the memory 502 communicate with each other via the communication bus 503. The processor 501 can call the logic instructions in the memory 502 to execute the spectral convolutional neural network processing method, which includes: using a preset filter structure on the surface of the image sensor to modulate the incident light to obtain a spectral vector of the modulated transmitted light; wherein the filter structure includes a plurality of filter units arranged in sequence, and each filter unit corresponds to a pixel point in the pixel array of the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses; based on the image sensor, the spectral vector of the modulated transmitted light is obtained through parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal, and the electrical signal is sent to the processor module for preset summation processing to obtain a corresponding feature map, and the feature map is input into a preset neural network model for processing to obtain an output result of the neural network model; the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is a neural network model.

[0055] In addition, the logic instructions in the above-mentioned memory 502 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0056] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the spectral convolutional neural network processing method provided by the above methods, and the method includes: using a preset filter structure on the surface of the image sensor to modulate the incident light to obtain a spectral vector of the modulated transmitted light; wherein the filter structure includes a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in the pixel array of the image sensor; different filter units composed of different pigments in the filter structure have Different wide-spectrum filtering responses; based on the image sensor, the spectral vector of the modulated transmitted light is obtained for parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal, and the electrical signal is sent to the processor module for preset summation processing to obtain a corresponding feature map, and the feature map is input into a preset neural network model for processing to obtain the output result of the neural network model; the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filtering structure; the neural network model is a neural network model.

[0057] On the other hand, the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, wherein the program executes the spectral convolutional neural network processing method provided by the above methods when it is run, the method comprising: using a preset filter structure on the surface of an image sensor to modulate the incident light to obtain a spectral vector of the modulated transmitted light; wherein the filter structure includes a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in the pixel array of the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filtering responses; based on the image sensor, the spectral vector of the modulated transmitted light is obtained through parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal, the electrical signal is sent to a processor module for preset summation processing to obtain a corresponding feature map, and the feature map is input into a preset neural network model for processing to obtain an output result of the neural network model; the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the inner product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is an artificial neural network model.

[0058] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0059] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A spectral convolutional neural network processing system, characterized in that: include: A light filtering structure and an image sensor; the light filtering structure is a filter that modulates incident light in the frequency domain as a whole within a wide spectrum band; The filter structure covers the surface of the image sensor, and each filter unit in the filter structure corresponds to a pixel point of a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filter responses; The filter structure is used to modulate the incoherent wide-spectrum natural light incident on each pixel point of the pixel array in the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the spectral vector of the modulated transmitted light is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the incident light is incoherent wide-spectrum natural light; The image sensor is used to perform parallel detection and photoelectric conversion processing on the spectral vector of the modulated transmitted light based on each pixel point in the pixel array, and then output an electrical signal; send the electrical signal to the processor module for preset summation processing to obtain a corresponding feature map; and input the feature map into a preset neural network model for processing to obtain an output result of the neural network model; Wherein, the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filtering structure; the neural network model is an artificial neural network model.

2. The spectral convolutional neural network processing system according to claim 1, characterized in that: Also includes: Microlenses; A microlens is correspondingly arranged above each filter unit in the filter structure, and the microlens is used to focus the incident light of each filter unit.

3. A spectral convolutional neural network processing method, characterized in that: include: The incident light is modulated by using a filter structure preset on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the filter structure comprises a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filter responses; After obtaining the spectral vector of the modulated transmitted light based on the image sensor for parallel detection and photoelectric conversion processing, a corresponding electrical signal is obtained, and the electrical signal is sent to the processor module for preset summation processing to obtain a corresponding feature map, and the feature map is input into a preset neural network model for processing to obtain the output result of the neural network model; the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; the neural network model is an artificial neural network model.

4. The spectral convolutional neural network processing method according to claim 3, characterized in that: The method of modulating the incident light by using a filter structure preset on the surface of the image sensor to obtain a spectrum vector of the modulated transmitted light specifically includes: The transmitted light focused by the microlens is modulated by using a preset filter structure on the surface of the image sensor to obtain a spectral vector of the modulated transmitted light; wherein the microlens is used to focus the incident light; the incident light is incoherent wide-spectrum natural light.

5. The spectral convolutional neural network processing method according to claim 3, characterized in that: The obtaining of the spectral vector of the modulated transmitted light based on the image sensor and performing parallel detection and photoelectric conversion processing to obtain a corresponding electrical signal specifically includes: Based on the image sensor, perception calculation is performed on the incident spectral image. Each pixel point of the image sensor is used to perceive the spectral vector of the superpixel at the corresponding spatial position in the incident spectral image, so as to calculate the inner product of the spectral vector and the transmission response vector of the spectral modulation unit on the pixel surface of the image sensor to obtain the corresponding electrical signal.

6. The spectral convolutional neural network processing method according to claim 5, characterized in that: The step of sending the electrical signal to a processor module for a preset summation process to obtain a corresponding characteristic graph includes: The electrical signal is sent to a preset processor module, and the processor module performs a summation operation on the electrical signal to obtain a feature map of the convolution kernels in the optical computing convolution unit.

7. A spectral convolutional neural network processing device, characterized in that: include: A modulation module, used for modulating incident light using a preset filter structure on the surface of an image sensor to obtain a spectral vector of the modulated transmitted light; wherein the filter structure comprises a plurality of filter units arranged in sequence, each filter unit corresponds to a pixel point in a pixel array in the image sensor; different filter units composed of different pigments in the filter structure have different wide-spectrum filter responses; the microlens is used for focusing the incident light; the incident light is incoherent wide-spectrum natural light; A processing module is used to obtain a spectral vector of the modulated transmitted light based on the image sensor, perform parallel detection and photoelectric conversion processing, obtain a corresponding electrical signal, send the electrical signal to the processor module for preset summation processing, obtain a corresponding feature map, and input the feature map into a preset neural network model for processing to obtain an output result of the neural network model; wherein the electrical signal is obtained by summing the energy of the modulated transmitted light in the spectral dimension, and the value of the energy of the modulated transmitted light in the spectral dimension is the Hadamard product of the spectral vector of the incident light and the transmission response vector of the filter structure; and the neural network model is an artificial neural network model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the spectral convolutional neural network processing method as described in any one of claims 3 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the spectral convolutional neural network processing method as described in any one of claims 3 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the spectral convolutional neural network processing method as described in any one of claims 3 to 7 is implemented.

Images