Optical artificial neural network intelligent chip and preparation method

Abstract

The present invention provides an optical artificial neural network intelligent chip and a preparation method thereof. An optical filter layer is used as an input layer and a linear layer of the artificial neural network, a filtering effect of the optical filter layer on incident light is used as a connection weight from the input layer to the linear layer, a square detection response of an image sensor is used as a first nonlinear activation function in a nonlinear layer of the artificial neural network, and a processor is used as a fully connected, second nonlinear activation function in a nonlinear layer and an output layer of the artificial neural network. Thus, the filter layer and the image sensor realize related functions of the input layer, the linear layer and the nonlinear activation function in the artificial neural network in a hardware manner, so that complex signal and algorithm processing corresponding to the input layer and the linear layer does not need to be performed in subsequent intelligent processing, thereby greatly reducing power consumption and delay in artificial neural network processing.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optical artificial neural network intelligent chip and a preparation method thereof. Background Art

[0002] Existing intelligent recognition technology usually requires first imaging a person or object, and then inputting the image into a neural network recognition model for processing, thereby realizing the recognition of the person or object.

[0003] It can be seen that the current intelligent recognition tasks generally rely on neural network recognition models, that is, the current intelligent recognition tasks need to be imaged first and then transmitted to the computer for subsequent neural network recognition model algorithm processing. The transmission and processing of large amounts of data causes greater power consumption and delays. Summary of the invention

[0004] In view of the problems existing in the prior art, the embodiments of the present invention provide an optical artificial neural network intelligent chip and a preparation method thereof.

[0005] Specifically, the embodiment of the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides an optical artificial neural network intelligent chip, comprising: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to an input layer, a linear layer and a connection weight from the input layer to the linear layer of the artificial neural network, and the square detection response of the image sensor corresponds to a first nonlinear activation function in a nonlinear layer of the artificial neural network; the processor corresponds to a fully connected and output layer of the artificial neural network, or the processor corresponds to a fully connected, second nonlinear activation function in a nonlinear layer and an output layer of the artificial neural network;

[0007] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor;

[0008] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0009] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an output signal of the artificial neural network.

[0010] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0011] Furthermore, the optical artificial neural network intelligent chip is used for intelligent processing tasks of target objects; the intelligent processing tasks include at least one or more of intelligent perception, intelligent recognition and intelligent decision-making tasks;

[0012] The reflected light, transmitted light and / or radiated light of the target object enters the trained optical artificial neural network intelligent chip to obtain the intelligent processing result of the target object; the intelligent processing result at least includes one or more of the intelligent perception result, the intelligent recognition result and / or the intelligent decision result;

[0013] The trained optical artificial neural network smart chip refers to an optical artificial neural network smart chip including a trained optical modulation structure, an image sensor and a processor;

[0014] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network intelligent chip containing different light modulation structures, image sensors and processors with different fully connected parameters using input training samples and output training samples corresponding to the intelligent processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network intelligent chip containing different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using input training samples and output training samples corresponding to the intelligent processing task.

[0015] Furthermore, when training an optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0016] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0017] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0018] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0019] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0020] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0021] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0022] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0023] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0024] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0025] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0026] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0027] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0028] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0029] Furthermore, the image sensor is any one or more of the following:

[0030] CMOS image sensor CIS, charge coupled device CCD, single photon avalanche diode SPAD array and focal plane photodetector array.

[0031] In a second aspect, an embodiment of the present invention further provides an intelligent device, including: the optical artificial neural network intelligent chip as described in the first aspect.

[0032] Furthermore, the smart device includes one or more of a smart phone, a smart computer, a smart identification device, a smart sensing device and a smart decision-making device.

[0033] In a third aspect, an embodiment of the present invention further provides a method for preparing the optical artificial neural network intelligent chip as described in the first aspect, comprising:

[0034] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0035] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0036] connecting the image sensor and the processor;

[0037] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light carrying information includes light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

[0038] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0039] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an output signal of the artificial neural network.

[0040] Furthermore, an optical filter layer including a light modulation structure is prepared on the surface of the image sensor, comprising:

[0041] Growing one or more layers of a preset material on the surface of the image sensor;

[0042] Etching the one or more layers of the preset material into a light modulation structure pattern to obtain a light filter layer including the light modulation structure;

[0043] or performing embossing transfer on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure;

[0044] or obtaining an optical filter layer including an optical modulation structure by externally dynamically modulating the one or more layers of the preset material;

[0045] or performing zone printing on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure;

[0046] or performing partitioned growth on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure;

[0047] Alternatively, quantum dots are transferred to the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure.

[0048] Furthermore, when the optical artificial neural network smart chip is used for an intelligent processing task of a target object, the optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters is trained using input training samples and output training samples corresponding to the intelligent processing task to obtain a light modulation structure, image sensor and processor that meet the training convergence conditions; or, the optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain a light modulation structure, image sensor and processor that meet the training convergence conditions.

[0049] The embodiment of the present invention further provides an optical artificial neural network environmental monitoring chip, which is used for environmental monitoring intelligent processing tasks, and includes: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, and the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0050] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of environmental pollutants;

[0051] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0052] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain intelligent processing results of environmental monitoring;

[0053] Among them, the environmental monitoring intelligent processing tasks include the identification and / or qualitative analysis of environmental pollutants; the environmental monitoring intelligent processing results include the identification results of environmental pollutants and / or the qualitative analysis results of environmental pollution.

[0054] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0055] Furthermore, the optical artificial neural network environmental monitoring chip includes a trained light modulation structure, an image sensor and a processor;

[0056] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network environmental monitoring chip containing different light modulation structures, image sensors and processors with different fully connected parameters using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network environmental monitoring chip containing different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing task;

[0057] Wherein, the input training samples include incident light reflected, transmitted and / or radiated by different environmental pollutants; the output training samples include corresponding environmental pollutant identification results; and / or, the input training samples include incident light reflected, transmitted and / or radiated by different environmental pollutants; the output training samples include corresponding environmental pollution qualitative analysis results.

[0058] Furthermore, when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0059] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0060] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0061] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0062] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0063] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0064] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0065] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0066] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0067] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0068] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0069] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0070] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0071] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0072] Furthermore, an embodiment of the present invention also provides an environmental monitoring device, including the optical artificial neural network environmental monitoring chip as described above.

[0073] The embodiment of the present invention also provides a method for preparing the optical artificial neural network environmental monitoring chip as described above, comprising:

[0074] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0075] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0076] connecting the image sensor and the processor;

[0077] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light carrying information includes light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

[0078] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0079] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an intelligent processing result of environmental monitoring.

[0080] Furthermore, the method for preparing the optical artificial neural network environmental monitoring chip further includes: a training process for the optical artificial neural network environmental monitoring chip, specifically including:

[0081] Using input training samples and output training samples corresponding to the environmental monitoring intelligent processing task, an optical artificial neural network environmental monitoring chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0082] Alternatively, using input training samples and output training samples corresponding to the environmental monitoring intelligent processing task, an optical artificial neural network environmental monitoring chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0083] Furthermore, when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0084] The embodiment of the present invention further provides an optical artificial neural network fingerprint recognition chip for fingerprint recognition processing tasks, comprising: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0085] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of the user's fingerprint;

[0086] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0087] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a fingerprint recognition processing result.

[0088] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0089] Furthermore, the optical artificial neural network fingerprint recognition chip includes a trained light modulation structure, an image sensor and a processor;

[0090] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters using the input training samples and output training samples corresponding to the fingerprint recognition processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using the input training samples and output training samples corresponding to the fingerprint recognition processing task;

[0091] The input training samples include incident light reflected, transmitted and / or radiated by different human fingerprints; and the output training samples include corresponding fingerprint recognition results.

[0092] Furthermore, when training an optical artificial neural network fingerprint recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network fingerprint recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0093] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0094] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0095] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0096] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0097] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0098] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0099] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0100] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0101] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0102] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0103] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0104] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0105] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0106] An embodiment of the present invention further provides a fingerprint recognition device, comprising the optical artificial neural network fingerprint recognition chip as described above.

[0107] An embodiment of the present invention further provides a method for preparing an optical artificial neural network fingerprint recognition chip as described in any one of the above items, comprising:

[0108] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0109] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0110] connecting the image sensor and the processor;

[0111] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0112] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0113] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a fingerprint recognition processing result.

[0114] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0115] Furthermore, the method for preparing the optical artificial neural network fingerprint recognition chip further includes: a training process for the optical artificial neural network fingerprint recognition chip, specifically including:

[0116] Using input training samples and output training samples corresponding to the fingerprint recognition processing task, an optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0117] Alternatively, using input training samples and output training samples corresponding to the fingerprint recognition processing task, an optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors.

[0118] The embodiment of the present invention also provides an optical artificial neural network face recognition chip for face recognition processing tasks, comprising: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0119] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of the user's face;

[0120] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0121] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a face recognition processing result.

[0122] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0123] Furthermore, the optical artificial neural network face recognition chip includes a trained light modulation structure, an image sensor and a processor;

[0124] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network face recognition chip that includes different light modulation structures, image sensors and processors with different fully connected parameters using input training samples and output training samples corresponding to the face recognition processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network face recognition chip that includes different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using input training samples and output training samples corresponding to the face recognition processing task;

[0125] The input training samples include incident light reflected, transmitted and / or radiated by different faces; and the output training samples include corresponding face recognition results.

[0126] Furthermore, when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0127] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0128] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0129] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0130] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0131] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0132] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0133] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0134] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0135] Furthermore, the micro-nano unit has a polarization-independent characteristic, and in particular, the micro-nano unit has a four-fold rotational symmetry.

[0136] Further, the optical filter layer is composed of one or more layers;

[0137] The filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is made of one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0138] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0139] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0140] An embodiment of the present invention further provides a face recognition device, comprising the optical artificial neural network face recognition chip as described above.

[0141] The embodiment of the present invention further provides a method for preparing the optical artificial neural network face recognition chip as described above, comprising:

[0142] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0143] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0144] connecting the image sensor and the processor;

[0145] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0146] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0147] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a face recognition processing result.

[0148] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0149] Furthermore, the method for preparing the optical artificial neural network face recognition chip further includes: a training process for the optical artificial neural network face recognition chip, specifically including:

[0150] Using input training samples and output training samples corresponding to the face recognition processing task, an optical artificial neural network face recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0151] Alternatively, using input training samples and output training samples corresponding to the face recognition processing task, an optical artificial neural network face recognition chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0152] Furthermore, when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0153] The embodiment of the present invention further provides an optical artificial neural network machine vision enhancement chip, which is used for machine vision intelligent processing tasks, and includes: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, and the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0154] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of the target object in the machine vision scene;

[0155] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0156] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result;

[0157] Among them, the machine vision intelligent processing task includes the recognition and / or qualitative analysis of the target object in the machine vision scene; the machine vision intelligent processing result includes the recognition result and / or qualitative analysis result of the target object in the machine vision scene.

[0158] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0159] Further, the optical artificial neural network machine vision enhancement chip includes a trained light modulation structure, an image sensor and a processor;

[0160] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network machine vision enhancement chip containing different light modulation structures, image sensors and processors with different fully connected parameters using input training samples and output training samples corresponding to the machine vision intelligent processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network machine vision enhancement chip containing different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using input training samples and output training samples corresponding to the machine vision intelligent processing task;

[0161] Wherein, the input training samples include incident light reflected, transmitted and / or radiated by a target object in a specific machine vision scene, and the output training samples include target object recognition results in the specific machine vision scene; and / or, the input training samples include incident light reflected, transmitted and / or radiated by a target object in a specific machine vision scene, and the output training samples include qualitative analysis results of the target object in the specific machine vision scene.

[0162] Furthermore, when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0163] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0164] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0165] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0166] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0167] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0168] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0169] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0170] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0171] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0172] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0173] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0174] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0175] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0176] An embodiment of the present invention further provides an enhanced machine vision system, comprising a control mechanism and the optical artificial neural network machine vision enhancement chip as described above.

[0177] The embodiment of the present invention further provides a method for preparing an optical artificial neural network machine vision enhancement chip as described in any one of the above items, comprising:

[0178] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0179] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0180] connecting the image sensor and the processor;

[0181] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0182] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0183] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result.

[0184] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0185] Furthermore, the method for preparing the optical artificial neural network machine vision enhancement chip further includes: a training process for the optical artificial neural network machine vision enhancement chip, specifically including:

[0186] Using input training samples and output training samples corresponding to the machine vision intelligent processing task, an optical artificial neural network machine vision enhancement chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0187] Alternatively, using input training samples and output training samples corresponding to the machine vision intelligent processing task, an optical artificial neural network machine vision enhancement chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0188] Furthermore, when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0189] The optical artificial neural network smart chip and preparation method provided in the embodiment of the present invention realize a new smart chip capable of realizing the function of an artificial neural network. In the smart chip, the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network, the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to the other part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, which is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In the embodiment of the present invention, the optical modulation structure on the optical filter layer modulates the incident light equivalent to the connection weight from the input layer to the linear layer. At the same time, in an embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in this intelligent chip, the optical filter layer corresponds to the input layer, linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network. , the second nonlinear activation function in the nonlinear layer and the output layer, that is, the optical filter layer and the image sensor in the smart chip realize the related functions of the input layer, the linear layer and some nonlinear activation functions in the artificial neural network, that is, the embodiment of the present invention removes the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the smart chip is used for artificial neural network intelligent processing in the future, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions. The processor in the smart chip only needs to perform full connection processing with the electrical signal or full connection and the second nonlinear activation processing, which can greatly reduce the power consumption and delay during artificial neural network processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0190] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0191] Figure 1 It is a schematic diagram of the structure of the optical artificial neural network intelligent chip provided by the first embodiment of the present invention;

[0192] Figure 2 This is a schematic diagram of the recognition principle of an optical artificial neural network intelligent chip provided by an embodiment of the present invention;

[0193] Figure 3 This is a schematic diagram of disassembling an optical artificial neural network intelligent chip provided by an embodiment of the present invention;

[0194] Figure 4 is a schematic diagram of a target object recognition process provided by an embodiment of the present invention;

[0195] Figure 5 is a top view of an optical filter layer provided by one embodiment of the present invention;

[0196] Figure 6 is a top view of another optical filter layer provided by one embodiment of the present invention;

[0197] Figure 7 is a top view of another optical filter layer provided by an embodiment of the present invention;

[0198] Figure 8 is a top view of another optical filter layer provided by an embodiment of the present invention;

[0199] Fig. 9 is a top view of yet another optical filter layer provided by an embodiment of the present invention;

[0200] Fig.10 is a top view of yet another optical filter layer provided by an embodiment of the present invention;

[0201] Fig.11 This is a schematic diagram of the broadband filtering effect of a micro-nano structure provided by an embodiment of the present invention;

[0202] Fig.12 This is a schematic diagram of the narrow-band filtering effect of a micro-nano structure provided by an embodiment of the present invention;

[0203] Fig.13 is a schematic structural diagram of a front-illuminated image sensor provided by an embodiment of the present invention;

[0204] Fig.14 is a schematic structural diagram of a back-illuminated image sensor provided by an embodiment of the present invention;

[0205] Fig.15 It is a schematic flow chart of a method for preparing an optical artificial neural network intelligent chip provided by a third embodiment of the present invention;

[0206] Fig.16 is a schematic diagram of a pollutant sample identification process provided by an embodiment of the present invention;

[0207] Fig.17 is a schematic diagram of a fingerprint recognition process provided by an embodiment of the present invention;

[0208] Fig.18 is a schematic diagram of a face recognition process provided by an embodiment of the present invention;

[0209] Fig.19 It is a schematic diagram of a machine vision enhanced recognition process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0210] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0211] Existing intelligent recognition technology usually requires first imaging a person or object, and then inputting the image into a neural network recognition model for processing, thereby realizing the recognition of the person or object. It can be seen that the current intelligent recognition tasks generally rely on neural network recognition models, that is, the current intelligent recognition tasks need to be imaged first and then transmitted to a computer for subsequent neural network recognition model algorithm processing. The transmission and processing of large amounts of data causes greater power consumption and delays. Based on this, an embodiment of the present invention provides an optical artificial neural network smart chip, in which an optical filter layer corresponds to an input layer, a linear layer and a connection weight from the input layer to the linear layer of the artificial neural network, and a square detection response of an image sensor corresponds to a first nonlinear activation function in a nonlinear layer of the artificial neural network; a processor corresponds to a full connection and an output layer of the artificial neural network, or the processor corresponds to a full connection, a second nonlinear activation function in a nonlinear layer and an output layer of the artificial neural network. An embodiment of the present invention uses an optical filter layer and an image sensor to project spatial spectral information of a target object into an electrical signal, and then implements full connection processing or full connection processing and a second nonlinear activation processing of the electrical signal in the processor. It can be seen that the embodiment of the present invention can omit the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions in the prior art. The embodiment of the present invention strips the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the smart chip is used for the subsequent intelligent processing of the artificial neural network, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and part or all of the nonlinear activation functions. The processor in the smart chip only needs to perform full connection processing with the electrical signal or full connection and a second nonlinear activation processing, which can greatly reduce the power consumption and delay during the artificial neural network processing. The content provided by the present invention will be explained and illustrated in detail through specific embodiments.

[0212] like Figure 1 As shown, the optical artificial neural network smart chip provided by the first embodiment of the present invention includes: an optical filter layer 1, an image sensor 2 and a processor 3; the optical filter layer 1 corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor 2 corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor 3 corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0213] The optical filter layer 1 is arranged on the surface of the image sensor or the surface of the photosensitive area of ​​the image sensor, and the optical filter layer 1 includes an optical modulation structure. The optical filter layer 1 is used to perform spectrum modulation of the intensity modulation that varies with the wavelength on the incident light entering different positions of the optical modulation structure through the optical modulation structure, that is, different intensity modulation is performed on the incident light of different wavelengths, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light carrying information includes image information and / or various optical spatial information of the target object to be processed by the optical artificial neural network intelligent chip, for example, the incident light carrying information includes at least one of light intensity distribution information, spectrum information, angle information of the incident light and phase information of the incident light;

[0214] In this embodiment, the square detection response of the image sensor 2 means that the image sensor detects the intensity information of the incident light field, and the intensity information of the incident light field is the square of the modulus of the light field signal, that is, the image sensor 2 converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer 1 into electrical signals corresponding to different positions through the square detection response after the first nonlinear activation processing, and sends the electrical signals corresponding to different positions to the processor 3; the electrical signal is the image signal modulated by the optical filter layer;

[0215] The processor 3 is used to perform full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain the output signal of the artificial neural network.

[0216] In this embodiment, the optical filter layer 1 is disposed on the surface of the image sensor, and the optical filter layer 1 includes an optical modulation structure. The optical filter layer 1 is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain the information carried by the modulated incident light corresponding to the different positions on the surface of the image sensor. It can be seen that in the embodiment, the modulation effect of the optical modulation structure on the optical filter layer on the incident light can be regarded as the connection weight from the input layer to the linear layer;

[0217] In this embodiment, when the image sensor 2 performs photoelectric conversion on the information carried by the modulated incident light, since the image sensor 2 can detect the intensity information of the light, the electrical signal obtained by processing the light field distribution signal is proportional to the square of the modulus of the light field distribution signal. Therefore, the image sensor 2 has a square detection response. Therefore, the image sensor 2 can be regarded as a part of the nonlinear layer of the artificial neural network, that is, the square detection response of the image sensor 2 can be regarded as the first nonlinear activation function of the artificial neural network.

[0218] In this embodiment, the image sensor 2 converts the information carried by the incident light corresponding to different position points after modulation by the optical filter layer 1 into electrical signals corresponding to different position points after the first nonlinear activation processing through the square detection response, that is, the image signal after modulation by the optical filter layer. At the same time, the processor 3 connected to the image sensor 2 is used to perform full connection processing on the electrical signals corresponding to different position points or perform full connection and second nonlinear activation processing to obtain the output signal of the artificial neural network.

[0219] In this embodiment, the optical filter layer 1 includes an optical modulation structure, through which the spectrum of the incident light (such as reflected light, transmitted light, radiated light and other related action light of the target to be identified) entering different positions of the optical modulation structure is modulated with different intensities, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor 2.

[0220] In this embodiment, it can be understood that the modulation intensity is related to the specific structural form of the light modulation structure. For example, different modulation intensities can be achieved by designing different light modulation structures (such as changing the shape and / or size parameters of the light modulation structure).

[0221] In this embodiment, it can be understood that the optical modulation structures at different positions on the optical filter layer 1 have different spectrum modulation effects on the incident light, and the modulation intensity of the optical modulation structure on different wavelength components of the incident light corresponds to the connection strength of the linear layer of the artificial neural network, that is, the connection weight from the input layer to the linear layer. It should be noted that the optical filter layer 1 is composed of a plurality of optical filter units, and the optical modulation structures at different positions in each optical filter unit are different, so they have different spectrum modulation effects on the incident light; the optical modulation structures at different positions between the optical filter units can be the same or different, so they have the same or different spectrum modulation effects on the incident light.

[0222] In this embodiment, the image sensor 2 converts the information carried by the incident light corresponding to different position points after modulation by the optical filter layer 1 into electrical signals corresponding to different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to different position points to the processor 3. The image sensor 2 corresponds to a part of the nonlinear layer of the neural network.

[0223] In this embodiment, the processor 3 performs full connection processing on the electrical signals at different positions or performs full connection processing and a second nonlinear activation processing on the electrical signals at different positions to obtain the output signal of the artificial neural network.

[0224] It can be understood that, in this embodiment, the image sensor 2 corresponds to a part of the nonlinear layer of the neural network, and the processor 3 corresponds to another part of the nonlinear layer of the neural network and the output layer. It can also be understood as corresponding to the remaining layers (all other layers) of the neural network except the input layer, the linear layer and the first nonlinear activation function in the nonlinear layer.

[0225] In this embodiment, it should be noted that the square detection response of the image sensor 2 corresponds to the first nonlinear activation function in the nonlinear layer of the neural network. In this case, the processor may only perform full connection processing without performing the second nonlinear activation processing, or the processor may perform both full connection processing and the second nonlinear activation processing. The specific details may be determined according to the actual application scenario of the chip, and this embodiment does not limit this.

[0226] In addition, it should be noted that the processor 3 can be set in the smart chip, that is, the processor 3 can be set in the smart chip together with the filter layer 1 and the image sensor 2, or it can be set separately outside the smart chip and connected to the image sensor 2 in the smart chip through a data line or a connecting device. This embodiment does not limit this.

[0227] In addition, it should be noted that the processor 3 can be implemented by a computer, or by an ARM or FPGA circuit board with certain computing power, or by a microprocessor, which is not limited in this embodiment. In addition, as mentioned above, the processor 3 can be integrated in the smart chip, or can be independently arranged outside the smart chip. When the processor 3 is independently arranged outside the smart chip, the electrical signal in the image sensor 2 can be read out to the processor 3 through the signal readout circuit, and then the processor 3 performs full connection processing and nonlinear activation processing on the readout electrical signal.

[0228] In this embodiment, it can be understood that when the processor 3 performs the second nonlinear activation processing, it can be implemented by using a nonlinear activation function, such as a Sigmoid function, a Tanh function, a ReLU function, etc., and this embodiment does not limit this.

[0229] In this embodiment, the optical filter layer 1 corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the image sensor 2 corresponds to a part of the nonlinear layer of the artificial neural network, that is, the square detection response of the image sensor 2 corresponds to the first nonlinear activation function of the artificial neural network, the image sensor 2 is used to perform nonlinear activation processing on the information carried by the incident light at different positions in space through the square detection response and then convert it into an electrical signal, the processor 3 corresponds to the remaining layers of the artificial neural network, fully connects the electrical signals at different positions, and can further obtain the output signal of the artificial neural network through the second nonlinear activation function, so as to realize intelligent perception, recognition and / or decision-making of specific targets.

[0230] like Figure 2 As shown on the left, the optical artificial neural network smart chip includes an optical filter layer 1, an image sensor 2 and a processor 3. Figure 2 In the embodiment, the processor 3 is implemented by a signal readout circuit and a computer. Figure 2 As shown on the right, the optical filter layer 1 in the optical artificial neural network smart chip corresponds to the input layer and linear layer of the artificial neural network, the image sensor 2 corresponds to a part of the nonlinear layer of the artificial neural network, the processor 3 corresponds to the other part of the nonlinear layer and the output layer of the artificial neural network, the filtering effect of the optical filter layer 1 on the incident light entering the optical filter layer 1 corresponds to the connection weight from the input layer to the linear layer, and the square detection response of the image sensor 2 corresponds to the first nonlinear activation function of the artificial neural network. It can be seen that the optical filter layer and the image sensor in the smart chip provided in this embodiment realize the related functions of the input layer, the linear layer and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the smart chip is used for intelligent processing later, there is no need to perform complex signal processing and algorithm processing corresponding to the input layer and the linear layer (for example, calculations such as the connection weight from the input layer to the linear layer are omitted), which can greatly reduce the power consumption and delay during artificial neural network processing.

[0231] like Figure 2 As shown on the right, the optical filter layer 1 has different broadband spectrum modulation effects on the incident light, and the incident light spectrum P at the corresponding unit position λ Projection / connection to the outgoing light field E N The square detection response of the image sensor 2 corresponds to the nonlinear activation function of the optical artificial neural network, and the outgoing light field E of the optical filter layer 1 N Converted to the photocurrent response I of the image sensor N The processor 3 includes a signal readout circuit and a computer. The signal readout circuit in the processor 3 reads out the photocurrent response I NThe signal is then transmitted to the computer, which performs full connection processing on the electrical signal or performs nonlinear activation processing again, and finally outputs the result.

[0232] like Figure 3 As shown, the light modulation structure on the optical filter layer 1 is integrated on the surface of the image sensor 2 to modulate the incident light, and the spectrum information of the incident light is projected / connected to different pixel points of the image sensor 2 to obtain an electrical signal containing the information carried by the incident light. That is, after the incident light passes through the optical filter layer 1, it is converted into an electrical signal after nonlinear activation by the square detection response of the image sensor 2 to form an image containing the spectrum information of the incident light. Finally, the processor 3 connected to the image sensor 2 processes the electrical signal containing the spectrum information and image information of the incident light to obtain an output result.

[0233] In this embodiment, the information carried by the incident light may include one or more (including two) of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0234] For example, in one implementation, the information carried by the incident light may include light intensity distribution information. In other implementations, the target object can be identified by simultaneously utilizing multiple information including the target object's image information, spectral information, angle of the incident light, and phase information of the incident light, thereby more accurately realizing intelligent identification of the target object.

[0235] It can be seen that the optical artificial neural network chip provided in this embodiment can actually simultaneously utilize the image information, spectral information, angle of incident light and phase information of incident light of the target object, that is, the information carried by the incident light at different points in space, and embed the artificial neural network in the hardware. The material composition, image shape, three-dimensional depth and other information can be further extracted from the spatial image, spectrum, angle, and phase information, thereby solving the problem mentioned in the background technology part that it is difficult to ensure the accuracy of recognition using the two-dimensional image information of the target object, for example, it is difficult to distinguish between a real person and a picture, and realizes intelligent perception, recognition and / or decision-making functions for different application fields, and realizes a spectrum optical artificial neural network intelligent chip with low power consumption, low latency and high accuracy.

[0236] The optical artificial neural network smart chip and preparation method provided in the embodiment of the present invention realize a new smart chip capable of realizing the function of an artificial neural network. In the smart chip, the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network, the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to the other part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, which is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In the embodiment of the present invention, the optical modulation structure on the optical filter layer modulates the incident light equivalent to the connection weight from the input layer to the linear layer. At the same time, in an embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in this intelligent chip, the optical filter layer corresponds to the input layer, linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network. , the second nonlinear activation function in the nonlinear layer and the output layer, that is, the optical filter layer and the image sensor in the smart chip realize the related functions of the input layer, the linear layer and some nonlinear activation functions in the artificial neural network, that is, the embodiment of the present invention removes the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the smart chip is used for artificial neural network intelligent processing in the future, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions. The processor in the smart chip only needs to perform full connection processing with the electrical signal or full connection and the second nonlinear activation processing, which can greatly reduce the power consumption and delay during artificial neural network processing.

[0237] In addition, it should be noted that in the prior art, when identifying a person or an object, only the two-dimensional image information of the person or the object is used. However, it is difficult to ensure the accuracy of the recognition with the two-dimensional image information, for example, it is difficult to distinguish between a real face and a face photo. Therefore, based on this, in one implementation, the incident light carries information that may include light intensity distribution information and spectral information, so that when the optical artificial neural network intelligent chip provided by the present application is used to perform the intelligent recognition task, the light intensity distribution information and spectral information of the object to be recognized can be used simultaneously. It can be seen that since the incident light carries information that covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, when the identification process is performed based on the incident light carrying information at different points in the target object space, the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the target object can be covered, thereby solving the problem that it is difficult to ensure the accuracy of the recognition using the two-dimensional image information of the target object, for example, it is difficult to distinguish between a real person and a picture, thereby enabling more accurate recognition of the object to be recognized. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object can be captured more comprehensively, so that the identification of the object to be identified can be achieved more accurately. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light, so that the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object can be captured more comprehensively, so that the identification of the object to be identified can be achieved more accurately. The optical artificial neural network smart chip provided by an embodiment of the present invention, in which the optical filter layer corresponds to the input layer and linear layer of the artificial neural network, the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes a light modulation structure. The light modulation structure is used to perform different spectral modulations on the incident light entering different positions of the light modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In an embodiment of the present invention, the modulation effect of the light modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer.At the same time, in an embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in this intelligent chip, the optical filter layer corresponds to the input layer, linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network. , the second nonlinear activation function in the nonlinear layer and the output layer, that is, the optical filter layer and the image sensor in the smart chip realize the related functions of the input layer, the linear layer and some nonlinear activation functions in the artificial neural network, that is, the embodiment of the present invention removes the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, the linear layer and some or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the smart chip is used for artificial neural network intelligent processing in the future, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions. The processor in the smart chip only needs to perform full connection processing with the electrical signal or full connection and the second nonlinear activation processing, which can greatly reduce the power consumption and delay during artificial neural network processing.It can be seen that the embodiment of the present invention uses the optical filter layer as the input layer, linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, and uses the square detection response of the image sensor as the first nonlinear activation function in the nonlinear layer of the artificial neural network; uses the processor as the fully connected and output layer of the artificial neural network, or, uses the processor to correspond to the fully connected, second nonlinear activation function in the nonlinear layer of the artificial neural network and the output layer. It can be seen that the embodiment of the present invention can not only save the complex signal processing and algorithm processing corresponding to the input layer, linear layer and part of the nonlinear activation function in the prior art, but also the embodiment of the present invention actually uses the image information, spectral information, angle of incident light and phase information of incident light of the target object at the same time, that is, the target object at different points in space. It can be seen that since the information carried by the incident light at different points in the target object space covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, when the identification and processing is performed based on the information carried by the incident light at different points in the target object space, the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the target object can be covered, thereby solving the problem mentioned in the background technology part that it is difficult to ensure the accuracy of recognition using the two-dimensional image information of the target object, for example, it is difficult to distinguish between a real person and a picture. It can be seen that the optical artificial neural network chip provided by the embodiment of the present invention can not only achieve the effects of low power consumption and low latency, but also achieve the effect of high accuracy, thereby being prepared for application in intelligent processing tasks such as intelligent perception, recognition and / or decision-making.

[0238] Based on the content of the above embodiment, in this embodiment, the optical artificial neural network intelligent chip is used for intelligent processing tasks of the target object; the intelligent processing tasks include at least one or more of intelligent perception, intelligent recognition and intelligent decision-making tasks;

[0239] The reflected light, transmitted light and / or radiated light of the target object enters the trained optical artificial neural network intelligent chip to obtain the intelligent processing result of the target object; the intelligent processing result at least includes one or more of the intelligent perception result, the intelligent recognition result and / or the intelligent decision result;

[0240] The trained optical artificial neural network intelligent chip refers to an optical artificial neural network intelligent chip including a trained optical modulation structure, an image sensor and a processor; the trained optical modulation structure, image sensor and processor refer to the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors with different fully connected parameters, which are obtained by training the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors with different fully connected parameters using the input training samples and output training samples corresponding to the intelligent processing task; or, the trained optical modulation structure, image sensor and processor refer to the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, which are obtained by training the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using the input training samples and output training samples corresponding to the intelligent processing task, which are obtained by training the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters.

[0241] In this embodiment, the input training samples include incident light reflected, transmitted and / or radiated by the target object in the corresponding intelligent processing task; the output training samples include the intelligent processing results of the target object (such as recognition results, perception results, decision results or qualitative analysis results, etc.).

[0242] In this embodiment, the optical artificial neural network intelligent chip can be used for intelligent processing tasks of the target object, for example, including one or more tasks of intelligent perception, intelligent recognition and intelligent decision-making tasks.

[0243] In this embodiment, it can be understood that intelligent perception refers to mapping the signals of the physical world to the digital world through hardware devices such as cameras, microphones or other sensors, with the help of cutting-edge technologies such as voice recognition and image recognition, and then further elevating these digital information to a cognitive level, such as memory, understanding, planning, decision-making, etc. Intelligent recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different modes. At present, intelligent recognition technology is generally divided into face recognition and commodity recognition. Face recognition is mainly used in security inspections, identity verification and mobile payments, and commodity recognition is mainly used in the process of commodity circulation, especially in unmanned retail fields such as unmanned shelves and smart retail cabinets. Intelligent decision-making refers to solving the problem of automatically organizing and coordinating the operation of multiple models by computers, accessing and processing data in a large number of databases, and performing corresponding data processing and numerical calculations.

[0244] In this embodiment, the reflected light, transmitted light and / or radiated light of the target object enters the trained optical artificial neural network intelligent chip to obtain the intelligent processing result of the target object.

[0245] In this embodiment, the target object recognition task is taken as an example for explanation. It can be understood that when using the smart chip to perform the recognition task, the optical artificial neural network smart chip needs to be trained first. Training the optical artificial neural network smart chip here means determining the optical modulation structure suitable for the current recognition task through training, as well as the fully connected parameters and nonlinear activation parameters suitable for the current recognition task.

[0246] It can be understood that since the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight from the input layer to the linear layer of the artificial neural network, during training, changing the light modulation structure in the optical filter layer is equivalent to changing the connection weight from the input layer to the linear layer of the artificial neural network. Through the training convergence conditions, the light modulation structure suitable for the current recognition task is determined, as well as the fully connected parameters and nonlinear activation parameters suitable for the current recognition task, thereby completing the training of the smart chip.

[0247] It is understandable that after the smart chip is trained, the smart chip can be used to perform recognition tasks. Specifically, after the incident light carrying the image information and spatial spectrum information of the target object enters the optical filter layer 1 of the trained smart chip, the light modulation structure in the optical filter layer 1 modulates the incident light, and the intensity of the modulated light signal is detected by the image sensor 2 and converted into an electrical signal, and then the processor 3 performs full connection processing or performs full connection and second nonlinear activation processing at the same time, and the recognition result of the target object can be obtained.

[0248] like Figure 4 As shown, the complete process for target object recognition is: the broadband light source 100 irradiates the target object 200, and then the reflected light or transmitted light of the target object is collected by the optical artificial neural network smart chip 300, or the light directly radiated outward by the target object is collected by the optical artificial neural network smart chip 300, and the recognition result can be obtained after being processed by the optical filter layer, image sensor and processor in the smart chip.

[0249] Among them, the trained optical artificial neural network intelligent chip refers to an optical artificial neural network intelligent chip including a trained optical modulation structure, an image sensor and a processor; the trained optical modulation structure, image sensor and processor refer to the optical artificial neural network intelligent chip including different optical modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, which are trained by using input training samples and output training samples corresponding to the intelligent processing task, and the optical modulation structure, image sensor and processor that meet the training convergence conditions are obtained.

[0250] For example, for intelligent recognition tasks, the input training samples corresponding to the intelligent recognition tasks are recognition object samples, and the output training samples corresponding to the intelligent recognition tasks are the recognition results of the recognition object samples. It can be understood that for recognition tasks, since the advantage of the smart chip provided in this embodiment is that it can obtain image information, spectral information, angle information of incident light, and phase information of incident light at different points in the recognition object space, in order to make full use of this advantage, for recognition object samples as input training samples, real recognition objects are preferred, rather than two-dimensional images of recognition objects. Of course, this does not mean that two-dimensional images cannot be used as recognition object samples.

[0251] In addition, the optical artificial neural network intelligent chip provided in this embodiment can also be used for other intelligent processing tasks of the target object, such as intelligent perception, intelligent decision-making and other tasks.

[0252] In this embodiment, the optical filter layer 1 serves as the input layer and linear layer of the neural network, and the image sensor 2 is used as a part of the nonlinear layer of the neural network (that is, the square detection response of the image sensor 2 is used as the first nonlinear activation function of the neural network). In order to minimize the loss function of the neural network, the modulation intensity of the different wavelength components in the incident light of the target object by the light modulation structure in the optical filter layer is used as the connection weight from the input layer to the linear layer of the neural network. By adjusting the structure of the filter, the modulation intensity of the different wavelength components in the incident light of the target object can be adjusted, thereby realizing the adjustment of the connection weight from the input layer to the linear layer, thereby optimizing the training of the neural network.

[0253] Therefore, in this embodiment, the optical modulation structure is obtained based on neural network training. The training samples are optically simulated by a computer to obtain the sample modulation intensity of the optical modulation structure in the training samples for different wavelength components of the incident light of the target object in the intelligent processing task. The sample modulation intensity is used as the connection weight from the input layer to the linear layer of the neural network to perform nonlinear activation, and the neural network is trained using the training samples corresponding to the intelligent processing task until the neural network converges and the corresponding training sample optical modulation structure is used as the optical filter layer corresponding to the intelligent processing task.

[0254] It can be seen that this embodiment can not only save the complex signal processing and algorithm processing corresponding to the input layer, linear layer (optical filter layer) and part of the nonlinear layer (the square detection response of image sensor 2 serves as the first nonlinear activation function of the neural network) in the prior art by realizing the input layer and linear layer (optical filter layer) of the neural network and part of the nonlinear layer (the square detection response of image sensor 2 serves as the first nonlinear activation function of the neural network) at the physical layer. At the same time, the embodiment of the present invention actually simultaneously utilizes the image information, spectral information, angle information of the incident light and phase information of the incident light at different points in the target object space, that is, the information carried by the incident light at different points in the target object space. It can be seen that since the information carried by the incident light at different points in the target object space covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, when the recognition processing is performed based on the information carried by the incident light at different points in the target object space, the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the target object can be covered, thereby solving the problem mentioned in the background technology part that it is difficult to ensure the accuracy of recognition using the two-dimensional image information of the target object, for example, it is difficult to distinguish between a real person and a picture. It can be seen that the optical artificial neural network chip provided by the embodiment of the present invention can not only achieve the effects of low power consumption and low latency, but also achieve the effect of high accuracy, thereby being prepared for application in intelligent processing tasks such as intelligent perception, recognition and / or decision-making.

[0255] Based on the contents of the above embodiments, in this embodiment, when training an optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or training an optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0256] In this embodiment, optical simulation allows users to experience products through a digital environment before making physical prototypes. For example, for cars, because illumination and reflected light can interfere with the driver's attention, especially when driving at night, a suitable optical simulation solution can not only effectively help users improve design efficiency, but also simulate the interaction between light and materials to understand the display effect of the product under real conditions. Therefore, in this embodiment, the light modulation structure is designed through computer optical simulation, and the light modulation structure is adjusted through optical simulation until the corresponding light modulation structure is determined to be the size of the light modulation structure that needs to be produced when the neural network converges, saving prototype production time and cost, improving product efficiency, and easily solving complex optical problems.

[0257] For example, FDTD software can be used to simulate and design the light modulation structure. By changing the light modulation structure in the optical simulation, the modulation intensity of the light modulation structure for different incident lights can be accurately predicted, and used as the connection weight between the input layer and the linear layer of the neural network to train the optical artificial neural network intelligent chip and accurately obtain the light modulation structure.

[0258] It can be seen that this embodiment designs the light modulation structure by adopting the computer optical simulation design method, which saves the time and cost of prototyping the light modulation structure and improves product efficiency.

[0259] Based on the contents of the above embodiments, in the present embodiment, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or, the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0260] In this embodiment, the light modulation structure in the optical filter layer may include only regular structures, only irregular structures, or both regular structures and irregular structures.

[0261] In this embodiment, the optical modulation structure includes a regular structure, which may refer to: the minimum modulation unit included in the optical modulation structure is a regular structure, such as the minimum modulation unit can be a regular shape such as a rectangle, a square, and a circle. In addition, the optical modulation structure includes a regular structure, which may refer to: the arrangement of the minimum modulation unit included in the optical modulation structure is regular, such as the arrangement can be a regular array form, a circular form, a trapezoidal form, a polygonal form, etc. In addition, the optical modulation structure includes a regular structure, which may refer to: the minimum modulation unit included in the optical modulation structure is a regular structure, and the arrangement of the minimum modulation unit is also regular, etc.

[0262] In this embodiment, the light modulation structure includes an irregular structure, which may refer to: the smallest modulation unit included in the light modulation structure is an irregular structure, such as the smallest modulation unit may be an irregular polygon, a random shape, or other irregular graphics. In addition, the light modulation structure includes an irregular structure, which may refer to: the arrangement of the smallest modulation unit included in the light modulation structure is irregular, such as the arrangement may be an irregular polygon, a random arrangement, etc. In addition, the light modulation structure includes an irregular structure, which may refer to: the smallest modulation unit included in the light modulation structure is an irregular structure, and the arrangement of the smallest modulation unit is also irregular, etc.

[0263] In this embodiment, the light modulation structure in the optical filter layer may include a discrete structure, a continuous structure, or both a discrete structure and a continuous structure.

[0264] In this embodiment, the light modulation structure includes a continuous structure, which may mean that the light modulation structure is composed of a continuous modulation pattern; the light modulation structure includes a discrete structure, which may mean that the light modulation structure is composed of a discrete modulation pattern.

[0265] It can be understood that the continuous modulation pattern here may refer to a straight line pattern, a wavy line pattern, a broken line pattern, etc.

[0266] It can be understood that the discrete modulation pattern here may refer to a modulation pattern formed by discrete graphics (such as discrete points, discrete triangles, discrete stars, etc.).

[0267] In this embodiment, it should be noted that the light modulation structure has different modulation effects on light of different wavelengths, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different filter structures, the corresponding transmission spectra of light passing through different groups of filter structures are different.

[0268] Based on the contents of the above embodiments, in this embodiment, the optical filter layer is a single-layer structure or a multi-layer structure.

[0269] In this embodiment, it should be noted that the optical filter layer may be a single-layer filter structure or a multi-layer filter structure, for example, a two-layer, three-layer, four-layer or other multi-layer structure.

[0270] In this embodiment, if Figure 1 As shown, the optical filter layer 1 is a single-layer structure, and the thickness of the optical filter layer 1 is related to the target wavelength range. For a wavelength of 400 nm to 10 μm, the thickness of the grating structure can be 50 nm to 5 μm.

[0271] It can be understood that since the function of the optical filter layer 1 is to perform spectral modulation on the incident light, it is preferably prepared from materials with high refractive index and low loss. For example, silicon, germanium, germanium silicon materials, silicon compounds, germanium compounds, III-V materials, etc. can be selected for preparation, wherein silicon compounds include but are not limited to silicon nitride, silicon dioxide, silicon carbide, etc.

[0272] In addition, it should be noted that in order to form more or more complex connection weights between the input layer and the linear layer, preferably, the optical filter layer 1 can be set to a multi-layer structure, and the optical modulation structure corresponding to each layer can be set to a different structure, thereby increasing the spectrum modulation capability of the optical filter layer to the incident light, so that more or more complex connection weights can be formed between the input layer and the linear layer, thereby improving the accuracy of the smart chip when processing smart tasks.

[0273] In addition, it should be noted that, for a filter layer including a multi-layer structure, the material of each layer can be the same or different. For example, for an optical filter layer 1 having two layers, the first layer can be a silicon layer and the second layer can be a silicon nitride layer.

[0274] It should be noted that the thickness of the optical filter layer 1 is related to the target wavelength range. For a wavelength of 400 nm to 10 μm, the total thickness of the multilayer structure can be 50 nm to 5 μm.

[0275] Based on the contents of the above embodiments, in this embodiment, the light modulation structure in the optical filter layer includes a unit array composed of multiple micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of each micro-nano unit are the same or different.

[0276] In this embodiment, in order to obtain connection weights distributed in an array (connection weights for connecting the input layer and the linear layer) so that the processor can perform subsequent full connection and nonlinear activation processing, preferably, in this embodiment, the light modulation structure is in the form of an array structure. Specifically, the light modulation structure includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixels on the image sensor. It should be noted that the structures of each micro-nano unit may be the same or different. In addition, it should be noted that the structure of each micro-nano unit may be periodic or non-periodic. In addition, it should be noted that each micro-nano unit may further include a plurality of groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays may be the same or different.

[0277] Combine the following Figures 5 to 9 For example, in this embodiment, Figure 5 As shown, the optical filter layer 1 includes a plurality of repeated continuous or discrete micro-nano units, such as 11, 22, 33, 44, 55, 66, each micro-nano unit has the same structure (and each micro-nano unit is a non-periodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2; Figure 6 As shown, the optical filter layer 1 comprises a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66, each of which has the same structure (with Figure 5 The difference is Figure 6 Each micro-nano unit is a periodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2; Figure 7 As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each of which has the same structure (and each of which is a periodic structure), and each of which corresponds to one or more pixels on the image sensor 2. Figure 6 The difference is Figure 7The unit shape of the periodic array in each micro-nano unit has four-fold rotational symmetry; Figure 8 As shown, the optical filter layer 1 includes a plurality of micro-nano units, such as 11, 22, 33, 44, 55, 66, and Figure 6 The difference is that each micro-nano unit has a different structure, and each micro-nano unit corresponds to one or more pixels on the image sensor 2. In this embodiment, the optical filter layer 1 includes multiple different micro-nano units, that is, different areas on the smart chip have different modulation effects on the incident light, thereby increasing the degree of freedom of design and further improving the accuracy of recognition. Fig. 9 As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each of which has the same structure. Figure 5 The difference is that each micro-nano unit is composed of a discrete non-periodic array structure, and each micro-nano unit corresponds to one or more pixel points on the image sensor 2.

[0278] In this embodiment, the micro-nano unit has different modulation effects on light of different wavelengths, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different filter structures, the corresponding transmission spectra of light passing through different groups of filter structures are different.

[0279] Based on the contents of the above embodiments, in this embodiment, the micro-nano unit includes a regular structure and / or an irregular structure; and / or, the micro-nano unit includes a discrete structure and / or a continuous structure.

[0280] In this embodiment, the micro-nano unit may include only regular structures, only irregular structures, or both regular structures and irregular structures.

[0281] In this embodiment, the micro-nano unit includes a regular structure, which may refer to: the minimum modulation unit included in the micro-nano unit is a regular structure, such as the minimum modulation unit can be a regular shape such as a rectangle, a square, and a circle. In addition, the micro-nano unit includes a regular structure, which may refer to: the arrangement of the minimum modulation unit included in the micro-nano unit is regular, such as the arrangement can be a regular array form, a circular form, a trapezoidal form, a polygonal form, etc. In addition, the micro-nano unit includes a regular structure, which may refer to: the minimum modulation unit included in the micro-nano unit is a regular structure, and the arrangement of the minimum modulation unit is also regular, etc.

[0282] In this embodiment, the micro-nano unit includes an irregular structure, which may refer to: the minimum modulation unit included in the micro-nano unit is an irregular structure, such as the minimum modulation unit may be an irregular polygon, a random shape, or other irregular graphics. In addition, the micro-nano unit includes an irregular structure, which may refer to: the arrangement of the minimum modulation unit included in the micro-nano unit is irregular, such as the arrangement may be an irregular polygon, a random arrangement, etc. In addition, the micro-nano unit includes an irregular structure, which may refer to: the minimum modulation unit included in the micro-nano unit is an irregular structure, and the arrangement of the minimum modulation unit is also irregular, etc.

[0283] In this embodiment, the micro-nano unit in the optical filter layer may include a discrete structure, a continuous structure, or both a discrete structure and a continuous structure.

[0284] In this embodiment, the micro-nano unit including a continuous structure may refer to that the micro-nano unit is composed of a continuous modulation pattern; the micro-nano unit including a discrete structure may refer to that the micro-nano unit is composed of a discrete modulation pattern.

[0285] It can be understood that the continuous modulation pattern here may refer to a straight line pattern, a wavy line pattern, a broken line pattern, etc.

[0286] It can be understood that the discrete modulation pattern here may refer to a modulation pattern formed by discrete graphics (such as discrete points, discrete triangles, discrete stars, etc.).

[0287] In this embodiment, it should be noted that different micro-nano units have different modulation effects on light of different wavelengths, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different micro-nano units, the corresponding transmission spectra of light passing through different groups of micro-nano units are different.

[0288] Based on the contents of the above embodiments, in this embodiment, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0289] In this embodiment, if Figure 5 As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each of which includes a plurality of micro-nano structure arrays, such as the micro-nano unit 11 includes four different micro-nano structure arrays 110, 111, 112, and 113, and the filter unit 44 includes four different micro-nano structure arrays 440, 441, 442, and 443. Fig.10As shown, the optical filter layer 1 includes multiple micro-nano units, such as 11, 22, 33, 44, 55, and 66. Each micro-nano unit includes multiple groups of micro-nano structure arrays. For example, the micro-nano unit 11 includes 4 identical micro-nano structure arrays 110, 111, 112, and 113.

[0290] It should be noted that the micro-nano unit including four groups of micro-nano structure arrays is only used as an example here, which does not serve as a limitation. In practical applications, the micro-nano unit can also be provided with six groups, eight groups or other number of groups of micro-nano structure arrays as needed.

[0291] In this embodiment, each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on light of different wavelengths, and each group of filter structures has different modulation effects on input light, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different micro-nano structure arrays, the corresponding transmission spectra of light passing through different groups of micro-nano structure arrays are different.

[0292] Based on the contents of the above embodiments, in this embodiment, each group of micro-nano structure arrays has the function of broadband filtering or narrowband filtering.

[0293] In this embodiment, in order to obtain the modulation intensity of different wavelength components of the incident light of the target object as the connection weights of the neural network input layer and the linear layer, broadband filtering and narrowband filtering are achieved by using different micro-nano structure arrays. Therefore, in this embodiment, the micro-nano structure array performs broadband filtering or narrowband filtering on the incident light of the target object to obtain the modulation intensity of different wavelength components of the incident light of the target object. Fig.11 and Fig.12 As shown, each group of micro-nano structure arrays in the optical filter layer has the function of broadband filtering or narrowband filtering.

[0294] It is understandable that, for each group of micro-nano structure arrays, they may all have broadband filtering effects, or they may all have narrowband filtering effects, or they may partially have broadband filtering effects and partially have narrowband filtering effects. In addition, the broadband filtering range and narrowband filtering range of each group of micro-nano structure arrays may also be the same or different. For example, by designing the period, duty cycle, radius, side length and other dimensional parameters of each group of micro-nano structures in the micro-nano unit, it has a narrowband filtering effect, that is, only one (or fewer) wavelengths of light can pass through. For another example, by designing the period, duty cycle, radius, side length and other dimensional parameters of each group of micro-nano structures in the micro-nano unit, it has a broadband filtering effect, that is, allowing more wavelengths or all wavelengths of light to pass through.

[0295] It is understandable that, in specific use, the filtering state of each group of micro-nano structure arrays can be determined by performing broadband filtering, narrowband filtering or a combination thereof according to the application scenario.

[0296] Based on the contents of the above embodiments, in this embodiment, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0297] In this embodiment, each group of micro-nano structure arrays can be periodic structure arrays, non-periodic structure arrays, or partly periodic structure arrays and partly non-periodic structure arrays. Among them, the periodic structure array is easy to perform optical simulation design, and the non-periodic structure array can achieve more complex modulation effects.

[0298] In this embodiment, if Figure 5 As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66. Each micro-nano unit is composed of a plurality of micro-nano structure arrays. The structures of each micro-nano structure array are different from each other, and the micro-nano structure array is a non-periodic structure. The non-periodic structure refers to that the shapes of the modulated holes on the micro-nano structure array are arranged in a non-periodic arrangement. Figure 5 As shown, the micro-nano unit 11 includes four different non-periodic structure arrays 110, 111, 112 and 113, and the micro-nano unit 44 includes four different non-periodic structure arrays 440, 441, 442 and 443. The micro-nano structure arrays with non-periodic structures are designed by neural network data training for intelligent processing tasks in the early stage, and are usually irregularly shaped structures. Figure 6 As shown, the optical filter layer 1 comprises a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66. Each micro-nano unit is composed of a plurality of micro-nano structure arrays. The structures of the micro-nano structure arrays are different from each other. Figure 5 The difference is that the micro-nano structure array is a periodic structure. The periodic structure refers to the arrangement of the modulated holes on the micro-nano structure array in a periodic arrangement, and the period size is usually 20nm to 50μm. Figure 6 As shown, the micro-nano unit 11 includes four different periodic structure arrays 110, 111, 112 and 113, and the micro-nano unit 44 includes four different periodic structure arrays 440, 441, 442 and 443. The filter structure of the periodic structure is obtained by training and designing the neural network data for intelligent processing tasks in the early stage, and is usually an irregular structure. Figure 7 As shown, the optical filter layer 1 includes a plurality of different micro-nano units, such as 11, 22, 33, 44, 55, and 66. Each micro-nano unit is composed of a plurality of micro-nano structure arrays. The micro-nano structure arrays are different from each other, and the micro-nano structure arrays are periodic structures. The periodic structure refers to the arrangement of the shapes on the filter structure in a periodic arrangement, and the size of the period is usually 20nm to 50μm. Figure 7As shown, the micro-nano structure arrays of the micro-nano unit 11 and the micro-nano unit 12 are different. The micro-nano unit 11 includes 4 different periodic structure arrays 110, 111, 112 and 113, and the micro-nano unit 44 includes 4 different periodic structure arrays 440, 441, 442 and 443. The micro-nano structure arrays with periodic structures are obtained through early training and design of neural network data for intelligent processing tasks, and are usually irregularly shaped structures.

[0299] It should be noted that Figures 5 to 9 Each micro-nano unit includes four groups of micro-nano structure arrays, and the four groups of micro-nano structure arrays are formed by modulation holes of four different shapes, and the four groups of micro-nano structure arrays are used to have different modulation effects on the incident light. It should be noted that here, only the micro-nano unit including four groups of micro-nano structure arrays is used as an example, which does not play a limiting role. In practical applications, micro-nano units including six groups, eight groups or other number groups of micro-nano structure arrays can also be set as needed. In this embodiment, the four different shapes can be circular, cross, regular polygon and rectangular (not limited to this).

[0300] In this embodiment, each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on light of different wavelengths, and each group of micro-nano structure arrays has different modulation effects on input light, and specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different micro-nano structure arrays, the corresponding transmission spectra of light passing through different groups of micro-nano structure arrays are different.

[0301] Based on the contents of the above embodiments, in this embodiment, the micro-nano unit includes a plurality of micro-nano structure arrays, and there is one or more groups of empty structures.

[0302] Combine the following Fig. 9 The example shown is used for illustration. In this embodiment, Fig. 9 As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each of which is composed of a plurality of micro-nano structure arrays, the structures corresponding to the plurality of micro-nano structure arrays are different from each other, and the micro-nano structure arrays are periodic structures. Different from the above-mentioned embodiments, any micro-nano unit includes one or more groups of empty structures, and the empty structures are used to pass incident light directly. It can be understood that when a plurality of micro-nano structure arrays include one or more groups of empty structures, a richer spectrum modulation effect can be formed, thereby meeting the spectrum modulation requirements in a specific scenario (or meeting the specific connection weight requirements between the input layer and the linear layer in a specific scenario).

[0303] like Fig. 9As shown, each micro-nano unit includes a group of micro-nano structure arrays and three groups of empty structures, micro-nano unit 11 includes a non-periodic structure array 111, micro-nano unit 22 includes a non-periodic structure array 221, micro-nano unit 33 includes a non-periodic structure array 331, micro-nano unit 44 includes a non-periodic structure array 441, micro-nano unit 55 includes a non-periodic structure array 551, and micro-nano unit 66 includes a non-periodic structure array 661, wherein the micro-nano structure array is used to modulate the incident light in different ways. It should be noted that here, only one group of micro-nano structure arrays and three groups of empty structures are used as an example, which does not play a limiting role. In practical applications, a micro-nano unit including a group of micro-nano structure arrays and five groups of empty structures or other number groups of micro-nano structure arrays can also be set according to needs. In this embodiment, the micro-nano structure array can be made of circular, cross-shaped, regular polygonal and rectangular modulation holes (not limited to this).

[0304] It should be noted that the multiple groups of micro-nano structure arrays included in the micro-nano unit may not include any empty structures, that is, the multiple groups of micro-nano structure arrays may be non-periodic structure arrays or periodic structure arrays.

[0305] Based on the contents of the above embodiments, in this embodiment, the micro-nano unit has polarization-independent characteristics.

[0306] In this embodiment, since the micro-nano unit has polarization-independent characteristics, the optical filter layer is insensitive to the polarization of the incident light, thereby realizing an optical artificial neural network intelligent chip that is insensitive to both the incident angle and polarization. The optical artificial neural network intelligent chip provided by the embodiment of the present invention is insensitive to the incident angle and polarization characteristics of the incident light, that is, the measurement result will not be affected by the incident angle and polarization characteristics of the incident light, thereby ensuring the stability of the spectral measurement performance, and further ensuring the stability of intelligent processing, such as the stability of intelligent perception, the stability of intelligent identification, the stability of intelligent decision-making, etc. It should be noted that the micro-nano unit may also have polarization-related characteristics.

[0307] Based on the contents of the above embodiments, in this embodiment, the micro-nano unit has four-fold rotational symmetry.

[0308] In this embodiment, it should be noted that four-fold rotational symmetry is a specific case of polarization-independent characteristics. By designing the micro-nano unit to have a structure with four-fold rotational symmetry, the requirements of polarization-independent characteristics can be met.

[0309] Combine the following Figure 7 The example shown is used for illustration. In this embodiment, Figure 7As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, and 66, each of which is composed of a plurality of groups of micro-nano structure arrays. The structures corresponding to the plurality of groups of micro-nano structure arrays are different from each other. The micro-nano structure arrays are periodic structures. Different from the above-mentioned embodiments, the structure corresponding to each group of micro-nano structure arrays can be a structure with four-fold rotational symmetry, such as a circle, a cross, a regular polygon, a rectangle, etc., that is, after the structure is rotated 90°, 180°, and 270°, it overlaps with the original structure, so that the structure has the characteristic of being independent of polarization, so that the same intelligent recognition effect can be achieved when different polarized light is incident.

[0310] Based on the contents of the above embodiments, in this embodiment, the optical filter layer is composed of one or more layers;

[0311] The filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer made of one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP (Surface Plasmon Polaritons, SPP) micro-nano structures, and adjustable Fabry-Perot cavity (Fabry-perot Cavity, FP cavity).

[0312] The semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0313] Among them, photonic crystals, as well as the combination of metasurfaces and random structures are compatible with CMOS technology and can have better modulation effects. Other materials can also be filled in the micropores of the micro-nano modulation structure to smooth the surface. Quantum dots and perovskites can utilize the spectral modulation characteristics of the material itself to minimize the volume of a single modulation structure. SPP has a small volume and can achieve polarization-dependent light modulation. Liquid crystals can be dynamically controlled by voltage to improve spatial resolution. Adjustable Fabry-Perot cavities can be dynamically controlled to improve spatial resolution.

[0314] Based on the content of the above embodiment, in this embodiment, the thickness of the optical filter layer is 0.1λ to 10λ, where λ represents the central wavelength of the incident light.

[0315] In this embodiment, it should be noted that if the thickness of the optical filter layer is much smaller than the central wavelength of the incident light, it cannot play an effective spectrum modulation role; if the thickness of the optical filter layer is much larger than the central wavelength of the incident light, it is difficult to prepare in terms of technology, and will introduce large optical losses. Therefore, in this embodiment, in order to reduce optical losses, facilitate preparation, and ensure effective spectrum modulation, the overall size (area) of each micro-nano unit in the optical filter layer 1 is usually λ 2 ~10 5 λ 2 , the thickness is usually 0.1λ~10λ (λ represents the central wavelength of the incident light of the target object). Figure 5 As shown, the overall size of each micro-nano unit is 0.5 μm 2 ~40000μm 2 The dielectric material in the optical filter layer 1 is polysilicon with a thickness of 50nm to 2μm.

[0316] Based on the contents of the above embodiments, in this embodiment, the image sensor is any one or more of the following:

[0317] CMOS image sensor (Contact Image Sensor, CIS), Charge Coupled Device (Charge Coupled Device, CCD), Single Photon Avalanche Diode (Single Photon Avalanche Diode, SPAD) array and focal plane photodetector array.

[0318] In this embodiment, it should be noted that the use of a wafer-level CMOS image sensor CIS and monolithic integration at the wafer level can minimize the distance between the image sensor and the optical filter layer, which is beneficial for reducing the size of the unit, reducing the device volume and packaging cost. SPAD can be used for weak light detection, and CCD can be used for strong light detection.

[0319] In this embodiment, the optical filter layer and the image sensor can be manufactured by a complementary metal oxide semiconductor (CMOS) integrated process, which is conducive to reducing device failure rate, improving device yield and reducing cost. For example, the optical filter layer can be prepared by directly growing one or more layers of dielectric material on the image sensor, then etching, depositing metal material before removing the sacrificial layer used for etching, and finally removing the sacrificial layer.

[0320] Based on the contents of the above embodiments, in this embodiment, the types of the artificial neural network include: feedforward neural network.

[0321] In this embodiment, the feedforward neural network (FNN), also known as the deep feedforward network (DFN) and the multi-layer perceptron (MLP), is the simplest neural network, in which neurons are arranged in layers. Each neuron is only connected to the neurons in the previous layer. It receives the output of the previous layer and outputs it to the next layer, and there is no feedback between the layers. The feedforward neural network has a simple structure, is easy to implement on hardware, and is widely used. It can approximate any continuous function and square integrable function with arbitrary precision. And it can accurately implement any finite training sample set. The feedforward network is a static nonlinear mapping. Complex nonlinear processing capabilities can be obtained through the composite mapping of simple nonlinear processing units.

[0322] Based on the contents of the above embodiments, in this embodiment, a light-transmitting medium layer is arranged between the optical filter layer and the image sensor.

[0323] In this embodiment, it should be noted that a light-transmitting medium layer is provided between the optical filter layer and the image sensor, so as to effectively separate the optical filter layer from the image sensor layer and avoid mutual interference between the two.

[0324] Based on the contents of the above embodiments, in this embodiment, the image sensor is front-illuminated, comprising: a metal wire layer and a light detection layer arranged from top to bottom, and the optical filter layer is integrated on a side of the metal wire layer away from the light detection layer; or,

[0325] The image sensor is back-illuminated and comprises: a light detection layer and a metal wire layer arranged from top to bottom, and the optical filter layer is integrated on a side of the light detection layer away from the metal wire layer.

[0326] In this embodiment, if Fig.13 The image sensor shown is a front-illuminated image sensor, in which the silicon detection layer 21 is below the metal wire layer 22 , and the optical filter layer 1 is directly integrated onto the metal wire layer 22 .

[0327] In this embodiment, Fig.13 The difference is, Fig.14 The image sensor shown is a back-illuminated one, in which the silicon detection layer 21 is above the metal line layer 22 , and the optical filter layer 1 is directly integrated onto the silicon detection layer 21 .

[0328] It should be noted that, for a back-illuminated image sensor, the silicon detection layer 21 is above the metal wire layer 22, which can reduce the influence of the metal wire layer on the incident light, thereby improving the quantum efficiency of the device.

[0329] According to the above content, the present embodiment uses the optical filter layer as the input layer and linear layer of the artificial neural network, uses the image sensor as a part of the nonlinear layer of the artificial neural network (using the square detection response of the image sensor as the first nonlinear activation function of the artificial neural network), and uses the filtering effect of the optical filter layer on the incident light entering the optical filter layer as the connection weight from the input layer to the linear layer. The optical filter layer and the image sensor in the optical artificial neural network intelligent chip provided by the present embodiment realize the related functions of the input layer, the linear layer and the part of the nonlinear activation function in the artificial neural network by hardware, so that when the intelligent chip is used for intelligent processing later, it is not necessary to perform the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and the part of the nonlinear activation function, which can greatly reduce the power consumption and delay of the artificial neural network processing. In addition, the present embodiment simultaneously utilizes the image information, the spectrum information, the angle information of the incident light and the phase information of the incident light at different points in the target object space, so that the intelligent processing of the target object can be realized more accurately.

[0330] It can be seen that in the embodiment of the present invention, the optical filter layer is used as the input layer and linear layer of the artificial neural network, and the image sensor is used as part of the nonlinear layer of the artificial neural network. The spatial spectrum information of the object is projected into the photocurrent response of the detector, and the full connection and secondary nonlinear activation of the electrical signal are realized in the processor, so as to realize the functions of intelligent perception, recognition and / or decision-making with low power consumption, low latency and high accuracy. The optical artificial neural network intelligent chip based on the optical filter and the image sensor in the embodiment of the present invention has the following effects: the artificial neural network is partially embedded in the image sensor containing various optical filter layers, so as to realize fast and accurate intelligent perception, recognition and / or decision-making functions. In addition, the embodiment of the present invention can also realize monolithic integration at the wafer level, so as to minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing the device volume and packaging cost.

[0331] Based on the same inventive concept, another embodiment of the present invention provides an intelligent device, including: the optical artificial neural network intelligent chip as described in the above embodiment. The intelligent device includes one or more of a smart phone, a smart computer, an intelligent recognition device, an intelligent perception device, and an intelligent decision-making device.

[0332] Since the smart device provided in this embodiment includes the optical artificial neural network smart chip described in the above embodiments, the smart device provided in this embodiment has all the beneficial effects of the optical artificial neural network smart chip described in the above embodiments. Since the above embodiments have already described this in detail, this embodiment will not repeat it.

[0333] Based on the same inventive concept, another embodiment of the present invention provides a method for preparing an optical artificial neural network intelligent chip as described in the above embodiment, such as Fig.15 As shown, the specific steps include:

[0334] Step 1510, preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0335] Step 1520: Generate a processor capable of performing full connection processing on a signal or a processor capable of performing full connection processing on a signal and performing a second nonlinear activation processing;

[0336] Step 1530: connecting the image sensor and the processor;

[0337] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light carrying information includes light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light;

[0338] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0339] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an output signal of the artificial neural network.

[0340] In this embodiment, a training process for the optical artificial neural intelligence chip is also included, which specifically includes:

[0341] Using input training samples and output training samples corresponding to the intelligent processing task, an optical artificial neural intelligent chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0342] Alternatively, using input training samples and output training samples corresponding to the intelligent processing task, an optical artificial neural intelligent chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0343] It can be understood that when training an optical artificial neural intelligence chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural intelligence chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0344] In this embodiment, a light filter layer including a light modulation structure is prepared on the surface of the photosensitive area of ​​the image sensor, including:

[0345] Growing one or more layers of a preset material on the surface of the image sensor;

[0346] Dry-etching the one or more layers of the preset material to form a light modulation structure pattern to obtain an optical filter layer including the light modulation structure;

[0347] or performing embossing transfer on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure;

[0348] Or by externally dynamically regulating the one or more layers of the preset material, an optical filter layer including a light modulation structure is obtained;

[0349] or performing zone printing on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure;

[0350] or performing partitioned material growth on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure;

[0351] Alternatively, quantum dots are transferred to the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure.

[0352] When the optical artificial neural network smart chip is used for an intelligent processing task of a target object, the optical artificial neural network smart chip including different light modulation structures, image sensors and processors with different full connection parameters is trained using input training samples and output training samples corresponding to the intelligent processing task to obtain a light modulation structure, image sensor and processor that meet the training convergence conditions; or, the optical artificial neural network smart chip including different light modulation structures, image sensors and processors with different full connection parameters and different second nonlinear activation parameters is trained to obtain a light modulation structure, image sensor and processor that meet the training convergence conditions.

[0353] In this embodiment, it should be noted that Figure 1 As shown, the optical filter layer 1 can be prepared by directly growing one or more layers of dielectric material on the image sensor 2, then etching, depositing metal material before removing the sacrificial layer used for etching, and finally removing the sacrificial layer. By designing the size parameters of the optical modulation structure, each unit can have different modulation effects on light of different wavelengths within the target range, and the modulation effect is insensitive to the incident angle and polarization. Each unit in the optical filter layer 1 corresponds to one or more pixels on the image sensor 2. 1 is prepared directly on 2.

[0354] In this embodiment, it should be noted that Fig.14 As shown, assuming that the image sensor 2 is a back-illuminated structure, the optical filter layer 1 can be directly etched on the silicon detector layer 21 of the back-illuminated image sensor, and then prepared by depositing metal.

[0355] In addition, it should be noted that the light modulation structure on the optical filter layer can be obtained by dry etching the light modulation structure pattern of one or more layers of preset materials. Dry etching is to directly remove the unnecessary parts of one or more layers of preset materials on the surface of the photosensitive area of ​​the image sensor to obtain the optical filter layer containing the light modulation structure; or one or more layers of preset materials are subjected to imprint transfer. Imprint transfer is to prepare the required structure on other substrates by etching, and then transfer the structure to the photosensitive area of ​​the image sensor through materials such as PDMS to obtain the optical filter layer containing the light modulation structure; or by external dynamic regulation of one or more layers of preset materials. The external dynamic regulation is to use active materials, and then the external electrodes regulate the light modulation characteristics of the corresponding areas by changing the voltage to obtain the optical filter layer containing the light modulation structure; or one or more layers of preset materials are subjected to zone printing. Zone printing is to use printing technology in zones to obtain the optical filter layer containing the light modulation structure; or one or more layers of preset materials are subjected to zone material growth to obtain the optical filter layer containing the light modulation structure; or one or more layers of preset materials are subjected to quantum dot transfer to obtain the optical filter layer containing the light modulation structure.

[0356] In addition, it should be noted that since the preparation method provided in this embodiment is the preparation method of the optical artificial neural network intelligent chip in the above-mentioned embodiment, for detailed contents on some principles and structures, etc., please refer to the introduction of the above-mentioned embodiment, and this embodiment will not go into details.

[0357] Based on this, an embodiment of the present invention provides an optical artificial neural network smart chip, in which an optical filter layer corresponds to an input layer, a linear layer and a connection weight from the input layer to the linear layer of the artificial neural network, and a square detection response of an image sensor corresponds to a first nonlinear activation function in a nonlinear layer of the artificial neural network; a processor corresponds to a full connection and an output layer of the artificial neural network, or the processor corresponds to a full connection, a second nonlinear activation function in a nonlinear layer and an output layer of the artificial neural network. An embodiment of the present invention uses an optical filter layer and an image sensor to project spatial spectral information of a target object into an electrical signal, and then implements full connection processing or full connection processing and a second nonlinear activation processing of the electrical signal in the processor. It can be seen that the embodiment of the present invention can omit the complex signal processing and algorithm processing corresponding to the input layer, the linear layer and some or all of the nonlinear activation functions in the prior art. The embodiment of the present invention removes the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the smart chip is used for artificial neural network intelligent processing in the future, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and part or all of the nonlinear activation functions. The processor in the smart chip only needs to perform full connection processing with the electrical signal or full connection and a second nonlinear activation processing. This can greatly reduce the power consumption and delay during artificial neural network processing. In addition, the embodiment of the present invention can also simultaneously utilize the image information, Spectral information, the angle of the incident light and the phase information of the incident light, that is, the information carried by the incident light at different points in the target object space. It can be seen that since the information carried by the incident light at different points in the target object space covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, when identification and processing are performed based on the information carried by the incident light at different points in the target object space, multi-dimensional information such as the image, composition, shape, three-dimensional depth, structure and the like of the target object can be covered, thereby improving the accuracy of intelligent processing (such as intelligent recognition). It can be seen that the optical artificial neural network chip provided in the embodiment of the present invention can not only achieve the effects of low power consumption and low latency, but also improve the accuracy of intelligent processing, so that it can be better used in intelligent processing fields such as intelligent perception, recognition and / or decision-making.

[0358] Environmental problems are the main problems facing my country's sustainable development. Environmental testing can achieve effective protection and governance of the environment. Environmental testing includes real-time monitoring of air, water quality, soil, etc., providing scientific, accurate and effective monitoring for environmental management, and formulating reasonable solutions based on this. Traditional environmental pollution monitoring is based on wet chemical technology and experimental analysis after air sampling. In recent years, the rapid development of analytical instruments can meet the needs of many environmental pollution monitoring, but these instruments are usually limited to single-point measurements. In comparison, optical and spectroscopy technologies have become ideal tools for environmental pollution monitoring with their large range, multi-component detection, good time resolution, and continuous real-time monitoring. Therefore, it is of great significance to realize large-scale, high-resolution, small volume, low-cost, safe and reliable real-time environmental pollution detection chips for environmental governance and protection.

[0359] To this end, based on the optical artificial neural network intelligent chip introduced in the above embodiment, this embodiment provides a new optoelectronic chip for multi-component environmental detection, the chip is composed of an optical filter layer to form the input layer and linear layer of the optical artificial neural network, and an image sensor to form the first nonlinear activation function of the optical artificial neural network, by collecting image information and spectral information of the detected environmental samples, it can realize rapid, accurate, safe and reliable identification and qualitative analysis of environmental pollutants in samples such as air, water, and soil. The content of this embodiment is explained and described in detail below.

[0360] The embodiment of the present invention further provides an optical artificial neural network environmental monitoring chip, which is used for environmental monitoring intelligent processing tasks, and includes: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, and the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0361] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of environmental pollutants;

[0362] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0363] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain intelligent processing results of environmental monitoring;

[0364] Among them, the environmental monitoring intelligent processing tasks include the identification and / or qualitative analysis of environmental pollutants; the environmental monitoring intelligent processing results include the identification results of environmental pollutants and / or the qualitative analysis results of environmental pollution.

[0365] In this embodiment, the information carried by the incident light includes image information and / or various optical spatial information of the target object (such as environmental pollutants) to be processed by the optical artificial neural network intelligent chip. For example, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0366] This embodiment realizes a new optical artificial neural network environmental monitoring chip that can realize the function of artificial neural network, which is used for environmental monitoring tasks. The embodiment of the present invention embeds an artificial neural network on the hardware chip, uses the optical filter layer on the hardware chip as the input layer and linear layer of the artificial neural network, uses the filtering effect of the optical filter layer on the hardware chip on the incident light as the connection weight from the input layer to the linear layer, and uses the square detection response of the image sensor on the hardware chip as the first nonlinear activation function in the nonlinear layer of the artificial neural network. The embodiment of the present invention injects the spatial spectral information of environmental pollutants into a pre-trained hardware chip, and uses the hardware chip to perform artificial neural network analysis on the information carried by the incident light at different points in the space of environmental pollutants, such as image information, spectral information, angle information of the incident light, and phase information of the incident light, and then obtains the environmental monitoring result. It should be noted that the embodiment of the present invention realizes low-power, safe and reliable, fast and accurate environmental monitoring.

[0367] It can be understood that in the optical artificial neural network environmental monitoring chip, the hardware structure thereon - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure thereon - the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In the embodiment of the present invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. At the same time, in the embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in the optical artificial neural network environmental monitoring chip, the optical filter layer and the image sensor implemented in hardware replace or implement the related functions of the input layer, linear layer and part of the nonlinear activation function in the existing artificial neural network. That is, the embodiment of the present invention strips the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the optical artificial neural network environmental monitoring chip is used for artificial neural network intelligent processing in the future, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and part or all of the nonlinear activation functions. The processor in the optical artificial neural network environmental monitoring chip only needs to perform full connection processing with the electrical signal or full connection and a second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing.It can be seen that the embodiment of the present invention uses the optical filter layer as the input layer, linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, and uses the square detection response of the image sensor as the first nonlinear activation function in the nonlinear layer of the artificial neural network; uses the processor as the fully connected and output layer of the artificial neural network, or, uses the processor to correspond to the fully connected, second nonlinear activation function in the nonlinear layer of the artificial neural network and the output layer. It can be seen that the embodiment of the present invention can omit the complex signal processing and algorithm processing corresponding to the input layer, linear layer and part of the nonlinear activation function in the prior art, thereby greatly reducing the power consumption and delay during artificial neural network processing.

[0368] Furthermore, the incident light carries information including at least one of light intensity distribution information, spectral information, angle information of the incident light, and phase information of the incident light. It can be seen that the embodiments of the present invention can utilize one or more of the image information, spectral information, angle of the incident light, and phase information of the incident light of environmental pollutants, thereby improving the accuracy of recognition. It should be noted that in the prior art, when identifying environmental pollutants, only two-dimensional image information of environmental pollutants is used, but two-dimensional image information is difficult to guarantee the accuracy of recognition. Therefore, based on this, in one implementation, the incident light carries information that may include light intensity distribution information and spectral information, so that when the optical artificial neural network smart chip provided by the present application is used to perform intelligent recognition tasks, the light intensity distribution information and spectral information of the object to be recognized can be used simultaneously. It can be seen that since the incident light carries information that covers the image, composition, shape, three-dimensional depth, structure and other information of environmental pollutants (hereinafter also referred to as target objects), when the identification process is performed based on the incident light carrying information at different points in the target object space, the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the target object can be covered, thereby solving the problem that it is difficult to ensure the accuracy of recognition by using the two-dimensional image information of the target object, such as it is difficult to distinguish between a real object and a picture, thereby enabling more accurate recognition of the object to be recognized. In addition, in another implementation, the incident light carries information that may further include light intensity distribution information, spectral information, and angle information of the incident light, thereby enabling more comprehensive capture of the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object, thereby enabling more accurate recognition of the object to be recognized. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectral information, angle information of the incident light and phase information of the incident light, so as to more comprehensively capture the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object, thereby more accurately realizing the identification of the object to be identified.

[0369] It can be seen that the embodiments of the present invention can make full use of the information carried by the incident light at different points in space of environmental pollutants. Since the information carried by the incident light at different points in space of environmental pollutants covers the image, composition, shape, three-dimensional depth, structure and other information of the environmental pollutants, when identification and processing are performed based on the information carried by the incident light at different points in space of environmental pollutants, multi-dimensional information such as the image, composition, shape, three-dimensional depth, structure and the like of the environmental pollutants can be covered, so that the information of the environmental pollutants can be accurately measured. In addition, due to the advantages of spectral information such as large range, multi-component detection, good time resolution, and continuous real-time monitoring, the chip can become an ideal tool for environmental pollution monitoring, thereby solving the problem that the current environmental monitoring instruments are only suitable for single-point monitoring. The chip provided in this embodiment can realize large-scale, high-resolution, small volume, low-cost, safe and reliable real-time environmental pollution detection, which is of great significance to environmental governance and protection.

[0370] Furthermore, the optical artificial neural network environmental monitoring chip includes a trained light modulation structure, an image sensor and a processor;

[0371] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network environmental monitoring chip containing different light modulation structures, image sensors and processors with different fully connected parameters using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network environmental monitoring chip containing different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using the input training samples and output training samples corresponding to the environmental monitoring intelligent processing task;

[0372] Wherein, the input training samples include incident light reflected, transmitted and / or radiated by different environmental pollutants; the output training samples include corresponding environmental pollutant identification results; and / or, the input training samples include incident light reflected, transmitted and / or radiated by different environmental pollutants; the output training samples include corresponding environmental pollution qualitative analysis results.

[0373] Furthermore, when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0374] In this embodiment, a large number of environmental pollutant samples can be collected first, and the weight of the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, so that the required optical filter layer can be reversely designed and integrated above the image sensor. In the actual environmental detection process, the output of the optical filter layer is used to further train and optimize the weight of the fully connected layer of the electrical signal, so that a high-accuracy optical artificial neural network can be realized, and the rapid and accurate identification and qualitative analysis of environmental pollutant samples can be completed. It can be seen that the chip actually uses the image information, spectral information, angle information of the incident light, and phase information of the incident light at different points in the space of environmental pollutants at the same time, improves the accuracy and diversity of environmental pollution detection, and partially realizes the artificial neural network in hardware, which improves the speed of environmental pollution detection. In addition, the chip solution can be mass-produced using existing CMOS technology, reducing the size, power consumption and cost of the device.

[0375] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0376] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0377] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0378] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0379] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0380] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0381] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0382] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0383] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0384] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0385] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0386] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0387] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0388] The embodiment of the present invention further provides an environmental monitoring device, including the optical artificial neural network environmental monitoring chip described in the above embodiment. The environmental monitoring device can be an environmental condition detector, a pollutant content analyzer, and the like.

[0389] The embodiment of the present invention also provides a method for preparing the optical artificial neural network environmental monitoring chip as described above, comprising:

[0390] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0391] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0392] connecting the image sensor and the processor;

[0393] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0394] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0395] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an intelligent processing result of environmental monitoring.

[0396] Furthermore, the method for preparing the optical artificial neural network environmental monitoring chip further includes: a training process for the optical artificial neural network environmental monitoring chip, specifically including:

[0397] Using input training samples and output training samples corresponding to the environmental monitoring intelligent processing task, an optical artificial neural network environmental monitoring chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0398] Alternatively, using input training samples and output training samples corresponding to the environmental monitoring intelligent processing task, an optical artificial neural network environmental monitoring chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0399] Furthermore, when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network environmental monitoring chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0400] In this embodiment, it should be noted that the chip directly prepares a micro-nano modulation structure on the surface of the photosensitive area of ​​the image sensor, and a plurality of discrete or continuous micro-nano structures constitute a unit. The micro-nano modulation structures at different positions have different spectrum modulation effects on the incident light, and together constitute an optical filter layer. The modulation intensity of these micro-nano modulation structures on different wavelength components of the incident light corresponds to the connection strength (linear layer weight) of the artificial neural network. The optical filter layer constitutes the input layer and the linear layer of the artificial neural network and the connection weight from the input layer to the linear layer, wherein the optical filter layer weights the input signal in the spectrum, and then the image sensor converts the weighted signal into an electrical signal (the part processed by the image sensor is equivalent to the first nonlinear activation function), and then the electrical signals output by the image sensors at different positions are fully connected through the processor, and a complete optical artificial neural network is realized through the second nonlinear activation function. For environmental pollution detection, a large number of environmental pollutant samples can be collected first, and the weights of the linear layer, that is, the system function of the optical filter layer, can be obtained through data training. Then the required optical filter layer can be reversely designed and integrated above the image sensor. In the actual environmental detection process, the output of the manufactured optical filter layer is used to further train and optimize the weights of the fully connected layer of the electrical signal, so as to realize a high-accuracy optical artificial neural network and complete the rapid and accurate identification and qualitative analysis of environmental pollutant samples.

[0401] It can be understood that for environmental pollution detection, by performing optical simulation of the micro-nano modulation structure on a computer, the modulation intensity (transmittance) of the modulation structure for different wavelength components of the incident light can be obtained, which can be used as the connection weight from the input layer to the linear layer of the artificial neural network, and the nonlinear activation function can be implemented in the processor. By collecting a large number of environmental pollutant samples in advance and conducting data training, the required micro-nano modulation structure can be designed and prepared, and the input layer, linear layer and part of the nonlinear activation function of the artificial neural network can be implemented on the chip.

[0402] Furthermore, it is understandable that see Figure 5 The image sensor 2 can use a CIS wafer, and the optical filter layer 1 is directly prepared on the CIS wafer. The optical filter layer 1 includes multiple repeated modulation units, each of which includes 4 different continuous non-periodic structure arrays. The basic unit of the non-periodic structure array is obtained by collecting a large number of different environmental pollutant samples in the early stage and training and designing the artificial neural network data. It is usually an irregularly shaped structure. Each non-periodic structure array has a different wide-spectrum modulation effect on the incident light, and the overall size of each modulation unit is 0.5μm. 2 ~40000μm 2The dielectric material in the optical filter layer 1 is polysilicon with a thickness of 50nm to 2μm. It can be understood that the CIS wafer includes a silicon detector layer and a metal wire layer, and the response range is from visible to near-infrared band; the CIS wafer is bare and the Bayer filter array and microlens array are not prepared. Each modulation unit corresponds to multiple sensor units on the CIS wafer.

[0403] The complete process of detecting pollutants with environmental detection chips is as follows: Fig.16 As shown, the light source under the detection instrument is irradiated onto the detection sample, and then the reflected light is collected by the chip, and the recognition result can be obtained after processing. Among them, the optical filter layer and the image sensor can be manufactured by the semiconductor CMOS integration process, and the monolithic integration is realized at the wafer level, which is conducive to reducing the distance between the sensor and the optical filter layer, reducing the size of the device, reducing the packaging cost, and realizing portable detection.

[0404] The environmental monitoring chip of the optical artificial neural network based on the micro-nano modulation structure and the image sensor provided in this embodiment has the following effects:

[0405] A. Embedding artificial neural network into image sensors containing various optical filter layers to achieve safe, reliable, fast and accurate environmental pollution detection; B. Detectable samples include but are not limited to air, water, and soil. Introducing artificial neural network training to identify pollutants has a large detection range, rich samples, high recognition accuracy, and precise qualitative analysis; C. The chip can be prepared in one-time tape-out through CMOS process, which is conducive to reducing device failure rate, improving device finished product yield, and reducing costs. D. Realizing monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing device volume and packaging costs.

[0406] It should be noted that for the detailed structural description of the optical artificial neural network environmental monitoring chip provided in this embodiment, please refer to the introduction of the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here to avoid redundancy. In addition, for the detailed introduction of the preparation method of the optical artificial neural network environmental monitoring chip, please refer to the introduction of the preparation method of the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here.

[0407] It is understandable that fingerprint recognition technology is a biometric recognition technology that is widely used in the fields of smartphone unlocking, access control systems, bank password verification, etc. The main process of fingerprint recognition includes fingerprint collection, fingerprint preprocessing, fingerprint feature extraction and comparison.

[0408] Current fingerprint recognition tasks generally rely on neural network recognition models, that is, current fingerprint recognition tasks require fingerprint imaging first and then transmission to a computer for subsequent neural network recognition model algorithm processing. The transmission and processing of large amounts of data causes greater power consumption and delays.

[0409] In addition, the acquisition of fingerprint images is the key to fingerprint recognition. The main ways to obtain fingerprint images are optical acquisition, capacitive sensor acquisition, thermal sensor acquisition, ultrasonic acquisition, etc. Among them, the capacitive sensor is usually placed on the back of the mobile phone to collect fingerprints. When combined with the thermal sensor, it can also realize liveness detection, but it cannot be used for under-screen fingerprint recognition. This is because the thickness of the screen module limits the signal acquisition of the capacitive sensor. The existing under-screen fingerprint recognition technology mainly includes two solutions: optical acquisition and ultrasonic acquisition. However, these two solutions only collect the texture image information of the fingerprint, which limits the accuracy of fingerprint recognition, and the size and power consumption of the device are still large. In addition, the fingerprint features of some people are less and difficult to identify; the fingerprints of relatives are similar, which can easily lead to recognition errors; the fingerprint information left on the surface of the object may be stolen and the security is not high. Therefore, it is necessary to combine fingerprint information with other information to improve the accuracy and security of recognition. In summary, it is of great significance to realize a fast under-screen fingerprint recognition device with high accuracy, small size, low power consumption, safety and reliability.

[0410] To this end, based on the optical artificial neural network smart chip introduced in the previous embodiment, this embodiment provides a new optoelectronic chip for accurate fingerprint recognition, the chip consists of an optical filter layer to form the input layer and linear layer of the optical artificial neural network, and an image sensor to form the first nonlinear activation function of the optical artificial neural network, and can achieve fast, accurate, safe and reliable fingerprint recognition by collecting image information and spectral information of the fingerprint to be recognized. The content of this embodiment is explained and illustrated in detail below.

[0411] The embodiment of the present invention further provides an optical artificial neural network fingerprint recognition chip for fingerprint recognition processing tasks, comprising: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0412] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of the user's fingerprint;

[0413] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0414] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a fingerprint recognition processing result.

[0415] In this embodiment, the information carried by the incident light includes image information and / or various optical spatial information of the target object (fingerprint) to be processed by the optical artificial neural network intelligent chip. For example, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0416] It can be seen that this embodiment realizes a new optical artificial neural network fingerprint recognition chip that can realize the function of artificial neural network, which is used for fingerprint recognition tasks. The embodiment of the present invention embeds an artificial neural network on the hardware chip, uses the optical filter layer on the hardware chip as the input layer and linear layer of the artificial neural network, uses the filtering effect of the optical filter layer on the hardware chip on the incident light as the connection weight from the input layer to the linear layer, and uses the square detection response of the image sensor on the hardware chip as the first nonlinear activation function in the nonlinear layer of the artificial neural network. The embodiment of the present invention allows the incident light at different points in the fingerprint space to carry information and enter the pre-trained hardware chip, and uses the hardware chip to perform artificial neural network analysis on the spatial spectrum information of the fingerprint to obtain the fingerprint recognition result. It should be noted that the embodiment of the present invention realizes low-power, safe and reliable fast and accurate fingerprint recognition.

[0417] It can be understood that in the optical artificial neural network fingerprint recognition chip, the hardware structure thereon - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure thereon - the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In the embodiment of the present invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. At the same time, in the embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in the optical artificial neural network fingerprint recognition chip, the optical filter layer and the image sensor implemented in hardware replace or implement the related functions of the input layer, linear layer and part of the nonlinear activation function in the existing artificial neural network. That is, the embodiment of the present invention strips the input layer, linear layer, and part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, linear layer, and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the optical artificial neural network fingerprint recognition chip is used for artificial neural network intelligent processing, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer, and part or all of the nonlinear activation functions. The processor in the optical artificial neural network fingerprint recognition chip only needs to perform full connection processing with the electrical signal or full connection and a second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing. It can be seen that the embodiment of the present invention can save the complex signal processing and algorithm processing corresponding to the input layer, linear layer, and part of the nonlinear activation function in the prior art, thereby greatly reducing the power consumption and delay of the artificial neural network processing.

[0418] Further, the incident light carries information including at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light. It can be seen that the embodiment of the present invention can utilize one or more information of the image information, spectrum information, angle of the incident light, and phase information of the incident light of the fingerprint, so as to improve the accuracy of recognition. In this embodiment, in one implementation, the incident light carries information including light intensity distribution information and spectrum information, so that when the optical artificial neural network intelligent chip provided by the present application is used to perform intelligent recognition tasks, the light intensity distribution information and spectrum information of the object to be recognized can be used at the same time. It can be seen that since the incident light carries information covering the image, component, shape, three-dimensional depth, structure and other information of the fingerprint (hereinafter also referred to as the target object), when the identification processing is performed based on the incident light carrying information at different points in the target object space, it can cover the image, component, shape, three-dimensional depth, structure and other multi-dimensional information of the target object, so that the recognition of the object to be recognized can be realized more accurately. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object can be more comprehensively captured, so that the identification of the object to be identified can be more accurately achieved. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectrum information, angle information of the incident light and phase information of the incident light, so that the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object can be more comprehensively captured, so that the identification of the object to be identified can be more accurately achieved.

[0419] It can be seen that the embodiments of the present invention can make full use of the information carried by the incident light at different points in the fingerprint space. Since the information carried by the incident light at different points in the fingerprint space covers the image, composition, shape, three-dimensional depth, structure and other information of the fingerprint, when the identification and processing is performed based on the information carried by the incident light at different points in the fingerprint space, the multi-dimensional information of the fingerprint, such as the image, composition, shape, three-dimensional depth, structure, etc., can be covered, so that the fingerprint identification can be accurately performed.

[0420] Furthermore, the optical artificial neural network fingerprint recognition chip includes a trained light modulation structure, an image sensor and a processor;

[0421] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters using the input training samples and output training samples corresponding to the fingerprint recognition processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training the optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using the input training samples and output training samples corresponding to the fingerprint recognition processing task;

[0422] The input training samples include incident light reflected, transmitted and / or radiated by different human fingerprints; and the output training samples include corresponding fingerprint recognition results.

[0423] In this embodiment, for under-screen fingerprint recognition, fingerprints of a large number of people can be collected first, and the weights from the input layer to the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and the required optical filter layer can be reversely designed and integrated above the image sensor. In actual use, when the user enters the fingerprint, the output of the manufactured optical filter layer is used to further train and optimize the weights of the fully connected layer of the electrical signal, so that a high-accuracy optical artificial neural network can be realized to quickly and accurately recognize the user's fingerprint.

[0424] It is understandable that the specific modulation pattern of the modulation structure on the optical filter layer is obtained by collecting fingerprints of a large number of people in the early stage and training and designing with artificial neural network data. It is usually an irregularly shaped structure, but it may also be a regular shaped structure.

[0425] like Fig.17 As shown in the figure, the complete process of under-screen fingerprint recognition is as follows: the light source under the mobile phone screen shines on the user's finger, and then the reflected light is collected by the chip, and the recognition result is obtained after internal processing.

[0426] It is understandable that the chip actually uses both the image information and the spectrum information of the fingerprint, which improves the accuracy and security of fingerprint recognition. At the same time, the chip partially implements the artificial neural network in hardware, which improves the speed of fingerprint recognition. In addition, the chip solution can be mass-produced using the existing CMOS process, reducing the size, power consumption and cost of the device.

[0427] Furthermore, when training an optical artificial neural network fingerprint recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network fingerprint recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0428] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0429] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0430] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0431] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0432] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0433] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0434] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0435] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0436] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0437] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0438] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0439] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0440] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0441] The embodiment of the present invention further provides a fingerprint recognition device, comprising the optical artificial neural network fingerprint recognition chip described in the above embodiment. The fingerprint recognition device can be a portable fingerprint recognition device or a fingerprint recognition device installed in a fixed position.

[0442] An embodiment of the present invention further provides a method for preparing an optical artificial neural network fingerprint recognition chip as described in any one of the above items, comprising:

[0443] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0444] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0445] connecting the image sensor and the processor;

[0446] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0447] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0448] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a fingerprint recognition processing result.

[0449] Furthermore, the method for preparing the optical artificial neural network fingerprint recognition chip further includes: a training process for the optical artificial neural network fingerprint recognition chip, specifically including:

[0450] Using input training samples and output training samples corresponding to the fingerprint recognition processing task, an optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0451] Alternatively, using input training samples and output training samples corresponding to the fingerprint recognition processing task, an optical artificial neural network fingerprint recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors.

[0452] It should be noted that the optical artificial neural network under-screen fingerprint recognition chip based on micro-nano modulation structure and image sensor provided in this embodiment has the following effects: A. The artificial neural network part is embedded in the hardware chip to achieve safe, reliable, fast and accurate under-screen fingerprint recognition. B. The preparation of the chip can be completed in one-time tape-out through the CMOS process, which is conducive to reducing the device failure rate, improving the device yield rate, and reducing costs. C. Realizing monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing the device volume and packaging cost.

[0453] It should be noted that for the detailed structural description of the optical artificial neural network fingerprint recognition chip provided in this embodiment, please refer to the introduction of the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here to avoid redundancy. In addition, for the detailed introduction of the preparation method of the optical artificial neural network fingerprint recognition chip, please refer to the introduction of the preparation method of the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here.

[0454] It is understandable that face recognition technology is a biometric recognition technology that is widely used in access control and attendance systems, criminal investigation systems, e-commerce, etc. The main process of face recognition includes face image acquisition, preprocessing, feature extraction, matching and recognition.

[0455] Current face recognition tasks generally rely on neural network recognition models, that is, current face recognition tasks require face imaging first and then transmission to a computer for subsequent neural network recognition model algorithm processing. The transmission and processing of large amounts of data causes greater power consumption and delays.

[0456] In addition, when performing face recognition, only the image information of the face is generally used. However, it is difficult to ensure the accuracy of recognition by using only the image information of the face. For example, it is difficult to distinguish between a real face and a face photo; even if the depth information is combined, it is difficult to distinguish between a face model and a real face. Therefore, it is of great significance to mine more face information and achieve faster face recognition with higher accuracy, safety and reliability.

[0457] To this end, based on the optical artificial neural network smart chip introduced in the previous embodiment, this embodiment provides a new optoelectronic chip for accurate face recognition, the chip consists of an optical filter layer to form the input layer and linear layer of the optical artificial neural network, and an image sensor to form the first nonlinear activation function of the optical artificial neural network, and can achieve fast, accurate, safe and reliable face recognition by collecting information carried by incident light at different points in the face space to be recognized. The content of this embodiment is explained and illustrated in detail below.

[0458] The embodiment of the present invention also provides an optical artificial neural network face recognition chip for face recognition processing tasks, comprising: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0459] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of the user's face;

[0460] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0461] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a face recognition processing result.

[0462] In this embodiment, the information carried by the incident light includes image information of the target object (such as a human face) to be processed by the optical artificial neural network intelligent chip and / or various optical spatial information. For example, the information carried by the incident light includes at least one of light intensity distribution information, spectral information, angle information of the incident light, and phase information of the incident light.

[0463] It can be seen that this embodiment realizes a new optical artificial neural network face recognition chip that can realize the function of an artificial neural network, which is used for face recognition tasks. The embodiment of the present invention embeds an artificial neural network on the hardware chip, uses the optical filter layer on the hardware chip as the input layer and linear layer of the artificial neural network, uses the filtering effect of the optical filter layer on the hardware chip on the incident light as the connection weight from the input layer to the linear layer, and uses the square detection response of the image sensor on the hardware chip as the first nonlinear activation function in the nonlinear layer of the artificial neural network. The embodiment of the present invention injects the spatial spectral information of the face into the pre-trained hardware chip, and uses the hardware chip to perform artificial neural network analysis on the information carried by the incident light at different points in the face space, such as image information, spectral information, angle information of the incident light, and phase information of the incident light, and then obtains the face recognition result. It should be noted that the embodiment of the present invention realizes low-power, safe and reliable fast and accurate face recognition.

[0464] It can be understood that in the optical artificial neural network face recognition chip, the hardware structure thereon - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network, and the hardware structure thereon - the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In the embodiment of the present invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. At the same time, in the embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in the optical artificial neural network face recognition chip, the optical filter layer and the image sensor implemented in hardware replace or implement the related functions of the input layer, linear layer and part of the nonlinear activation function in the existing artificial neural network. That is, the embodiment of the present invention strips off the input layer, linear layer, and part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, linear layer, and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the optical artificial neural network face recognition chip is used for artificial neural network intelligent processing, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer, and part or all of the nonlinear activation functions. The processor in the optical artificial neural network face recognition chip only needs to perform full connection processing with the electrical signal or full connection and a second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing. It can be seen that the embodiment of the present invention can save the complex signal processing and algorithm processing corresponding to the input layer, linear layer, and part of the nonlinear activation function in the prior art, thereby greatly reducing the power consumption and delay of the artificial neural network processing.

[0465] Furthermore, the information carried by the incident light includes at least one of light intensity distribution information, spectral information, angle information of the incident light, and phase information of the incident light. It can be seen that the embodiments of the present invention can utilize one or more information of face image information, spectral information, angle of incident light, and phase information of incident light, so as to improve the accuracy of recognition. It should be noted that in the prior art, when recognizing a face, only two-dimensional image information of the face is used, but two-dimensional image information is difficult to guarantee the accuracy of recognition (such as using a face picture). Therefore, based on this, in one implementation, the incident light carries information that may include light intensity distribution information and spectral information, so that when the optical artificial neural network smart chip provided by the present application is used to perform intelligent recognition tasks, the light intensity distribution information and spectral information of the object to be recognized can be used simultaneously. It can be seen that since the incident light carries information that covers the image, component, shape, three-dimensional depth, structure and other information of the face (hereinafter also referred to as the target object), when the identification process is performed based on the incident light carrying information at different points in the target object space, the image, component, shape, three-dimensional depth, structure and other multi-dimensional information of the target object can be covered, thereby solving the problem that it is difficult to ensure the accuracy of recognition by using the two-dimensional image information of the target object, such as it is difficult to distinguish between a real person and a picture, thereby enabling more accurate recognition of the object to be recognized. In addition, in another implementation, the incident light carries information that may further include light intensity distribution information, spectral information, and angle information of the incident light, thereby enabling more comprehensive capture of the image, component, shape, three-dimensional depth, three-dimensional structure and other information of the target object, thereby enabling more accurate recognition of the object to be recognized. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectral information, angle information of the incident light and phase information of the incident light, so as to more comprehensively capture the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object, thereby more accurately realizing the identification of the object to be identified.

[0466] It can be seen that the embodiments of the present invention can make full use of the information carried by the incident light at different points in the face space. Since the information carried by the incident light at different points in the face space covers the image, composition, shape, three-dimensional depth, structure and other information of the face, when identification processing is performed based on the information carried by the incident light at different points in the face space, multi-dimensional information such as the image, composition, shape, three-dimensional depth, structure and other information of the face can be covered, thereby accurately performing face recognition.

[0467] Furthermore, the optical artificial neural network face recognition chip includes a trained light modulation structure, an image sensor and a processor;

[0468] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network face recognition chip that includes different light modulation structures, image sensors and processors with different fully connected parameters using input training samples and output training samples corresponding to the face recognition processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network face recognition chip that includes different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using input training samples and output training samples corresponding to the face recognition processing task;

[0469] The input training samples include incident light reflected, transmitted and / or radiated by different faces; and the output training samples include corresponding face recognition results.

[0470] Furthermore, when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0471] In this embodiment, due to the addition of spectrum modulation of light, an artificial neural network optoelectronic chip whose input is an object image and its spectrum can be implemented to achieve fast, accurate, safe and reliable living face recognition.

[0472] In this embodiment, for face recognition, faces of a large number of people can be collected first, and the weights from the input layer to the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and the required optical filter layer can be reversely designed and integrated above the image sensor. During actual training, the weights of the fully connected layer of the electrical signal are further trained and optimized using the face samples to be recognized and the output of the manufactured optical filter layer, so that a high-accuracy optical artificial neural network can be realized to quickly and accurately recognize the user's face.

[0473] It is understandable that the specific modulation pattern of the modulation structure on the optical filter layer is obtained by collecting faces of a large number of people in the early stage and training and designing with artificial neural network data. It is usually an irregularly shaped structure, but it may also be a regular shaped structure.

[0474] like Fig.18As shown, the complete process of face recognition is: ambient light or other light sources shine onto the user's face, and then the reflected light is collected by the chip, and the recognition result is obtained after internal processing.

[0475] It is understandable that the chip actually uses both the image information and spectral information of the face, improving the accuracy and security of face recognition, especially for non-living face models, and can accurately exclude them. At the same time, the chip partially implements artificial neural networks in hardware, which improves the speed of fingerprint recognition. In addition, the chip solution can be mass-produced using existing CMOS processes, reducing the size, power consumption and cost of the device.

[0476] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0477] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0478] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0479] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0480] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0481] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0482] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0483] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0484] Furthermore, the micro-nano unit has a polarization-independent characteristic, and in particular, the micro-nano unit has a four-fold rotational symmetry.

[0485] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0486] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0487] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0488] The embodiment of the present invention further provides a face recognition device, including the optical artificial neural network face recognition chip described in the above embodiment. The face recognition device can be a portable face recognition device, or a face recognition device installed in a fixed position. Since the face recognition device has similar beneficial effects as the above optical artificial neural network face recognition chip, it will not be described in detail here.

[0489] The embodiment of the present invention further provides a method for preparing the optical artificial neural network face recognition chip as described above, comprising:

[0490] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0491] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0492] connecting the image sensor and the processor;

[0493] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0494] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0495] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a face recognition processing result.

[0496] Furthermore, the method for preparing the optical artificial neural network face recognition chip further includes: a training process for the optical artificial neural network face recognition chip, specifically including:

[0497] Using input training samples and output training samples corresponding to the face recognition processing task, an optical artificial neural network face recognition chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0498] Alternatively, using input training samples and output training samples corresponding to the face recognition processing task, an optical artificial neural network face recognition chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0499] Furthermore, when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network face recognition chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0500] It should be noted that the optical artificial neural network face recognition chip based on micro-nano modulation structure and image sensor provided in this embodiment has the following effects: A. The artificial neural network part is embedded in the hardware chip to achieve safe, reliable, fast and accurate face recognition. B. The preparation of the chip can be completed in one-time tape-out through the CMOS process, which is conducive to reducing the device failure rate, improving the device's finished product yield, and reducing costs. C. Realizing monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing the device volume and packaging costs.

[0501] It should be noted that for the detailed structural description of the optical artificial neural network face recognition chip provided in this embodiment, please refer to the introduction of the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here to avoid redundancy. In addition, for the detailed introduction of the method for preparing the optical artificial neural network face recognition chip, please refer to the introduction of the method for preparing the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here.

[0502] Machine vision technology is a branch of artificial intelligence technology. It uses machines to replace human eyes for observation and judgment. It is widely used in industrial production, quality inspection, express sorting, unmanned driving and other fields. A typical machine vision system includes an imaging system, an image processing system, a communication and IO system, and a linkage mechanism. Among them, the imaging system is responsible for collecting image information of the target object and is the key to machine vision technology.

[0503] The image processing system of current machine vision systems generally relies on neural network recognition models. That is, the current machine vision intelligent recognition tasks require imaging first and then transmission to a computer for subsequent neural network recognition model algorithm processing. The transmission and processing of large amounts of data results in high power consumption and delays.

[0504] In addition, the current machine vision technology only uses the image information of the object, and the accuracy and reliability of measurement and recognition need to be improved. Therefore, it is of great significance to use the information of other dimensions of the object to achieve enhanced machine vision with higher accuracy and reliability.

[0505] To this end, based on the optical artificial neural network intelligent chip introduced in the previous embodiment, this embodiment provides a new optoelectronic chip for enhancing machine vision, the chip is composed of an optical filter layer to form the input layer and linear layer of the optical artificial neural network, and the image sensor is used to form the first nonlinear activation function of the optical artificial neural network. By collecting the image information and spectral information of the object to be identified, it can achieve fast, accurate, safe and reliable object target recognition, thereby achieving enhanced machine vision with higher accuracy and reliability. The content of this embodiment is specifically explained and illustrated in detail below.

[0506] The embodiment of the present invention further provides an optical artificial neural network machine vision enhancement chip, which is used for machine vision intelligent processing tasks, and includes: an optical filter layer, an image sensor and a processor; the optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, and the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection of the artificial neural network, the second nonlinear activation function in the nonlinear layer and the output layer;

[0507] The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light includes reflected light, transmitted light and / or radiated light of the target object in the machine vision scene;

[0508] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0509] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result;

[0510] Among them, the machine vision intelligent processing task includes the recognition and / or qualitative analysis of the target object in the machine vision scene; the machine vision intelligent processing result includes the recognition result and / or qualitative analysis result of the target object in the machine vision scene.

[0511] In this embodiment, the information carried by the incident light includes image information and / or various optical spatial information of the target object (target object in the machine vision scene) to be processed by the optical artificial neural network intelligent chip. For example, the information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

[0512] It can be seen that this embodiment realizes a new optical artificial neural network enhanced machine vision recognition chip that can realize the function of an artificial neural network, which is used for target object recognition tasks in various machine vision application scenarios. The embodiment of the present invention embeds an artificial neural network on the hardware chip, uses the optical filter layer on the hardware chip as the input layer and linear layer of the artificial neural network, uses the filtering effect of the optical filter layer on the hardware chip on the incident light as the connection weight from the input layer to the linear layer, and uses the square detection response of the image sensor on the hardware chip as the first nonlinear activation function in the nonlinear layer of the artificial neural network. The embodiment of the present invention injects the image information, spectral information, angle information of the incident light and phase information of the incident light at different points in the target object space in the machine vision application scenario into the pre-trained hardware chip, and uses the hardware chip to perform artificial neural network analysis on the image information, spectral information, angle information of the incident light and phase information of the incident light at different points in the target object space to obtain the recognition result of the target object. It should be noted that the embodiment of the present invention realizes low-power, safe and reliable fast and accurate object recognition.

[0513] It can be understood that in the optical artificial neural network enhanced machine vision chip, the hardware structure thereon - the optical filter layer corresponds to the input layer and the linear layer of the artificial neural network, the hardware structure thereon - the image sensor corresponds to a part of the nonlinear layer of the artificial neural network; the processor corresponds to another part of the nonlinear layer of the artificial neural network and the output layer. Specifically, the optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the information carried by the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor. In the embodiment of the present invention, the modulation effect of the optical modulation structure on the optical filter layer on the incident light is equivalent to the connection weight from the input layer to the linear layer. At the same time, in the embodiment of the present invention, the image sensor converts the information carried by the incident light corresponding to the different position points after being modulated by the optical filter layer into electrical signals corresponding to the different position points after the first nonlinear activation processing through the square detection response, and sends the electrical signals corresponding to the different position points to the processor, and the processor performs full connection processing on the electrical signals corresponding to the different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to the different position points to obtain the output signal of the artificial neural network. It can be seen that in the optical artificial neural network enhanced machine vision chip, the optical filter layer and the image sensor implemented in hardware replace or implement the related functions of the input layer, the linear layer and some nonlinear activation functions in the existing artificial neural network. That is, the embodiment of the present invention strips the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network implemented by software in the prior art, and implements the input layer, linear layer and part or all of the nonlinear activation functions in the artificial neural network by hardware, so that when the chip is used for machine vision intelligent processing based on artificial neural networks in the future, it is no longer necessary to perform complex signal processing and algorithm processing corresponding to the input layer, linear layer and part or all of the nonlinear activation functions. It is only necessary for the processor in the optical artificial neural network enhanced machine vision chip to perform full connection processing with the electrical signal or full connection and a second nonlinear activation processing, which can greatly reduce the power consumption and delay of the artificial neural network processing. It can be seen that the embodiment of the present invention can omit the complex signal processing and algorithm processing corresponding to the input layer, linear layer and part of the nonlinear activation function in the prior art, thereby greatly reducing the power consumption and delay of the artificial neural network processing.

[0514] Further, the incident light carries information including at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light. It can be seen that the embodiment of the present invention can utilize one or more information of the image information, spectrum information, angle of the incident light, and phase information of the incident light of the target object in the machine vision scene, so as to improve the accuracy of recognition. In one implementation of this embodiment, the incident light carries information including light intensity distribution information and spectrum information, so that when the optical artificial neural network intelligent chip provided by the present application is used to perform intelligent recognition tasks, the light intensity distribution information and spectrum information of the object to be recognized can be used at the same time. It can be seen that since the incident light carries information covering the image, composition, shape, three-dimensional depth, structure and other information of the target object, when the identification processing is performed based on the incident light carrying information at different points in the target object space, it can cover the image, composition, shape, three-dimensional depth, structure and other multi-dimensional information of the target object, so that the recognition of the object to be recognized can be realized more accurately. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectrum information, and angle information of the incident light, so that the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object can be more comprehensively captured, so that the target object can be more accurately identified. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectrum information, angle information of the incident light and phase information of the incident light, so that the image, composition, shape, three-dimensional depth, three-dimensional structure and other information of the target object can be more comprehensively captured, so that the target object can be more accurately identified.

[0515] It can be seen that the embodiments of the present invention can make full use of the information carried by the incident light at different points in the space of the target object in the machine vision application scenario. Since the information carried by the incident light at different points in the space of the target object covers the image, composition, shape, three-dimensional depth, structure and other information of the target object, when identification and processing are performed based on the information carried by the incident light at different points in the space of the target object, multi-dimensional information such as the image, composition, shape, three-dimensional depth, structure and the like of the target object can be covered, thereby accurately identifying the target object.

[0516] Further, the optical artificial neural network machine vision enhancement chip includes a trained light modulation structure, an image sensor and a processor;

[0517] The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network machine vision enhancement chip containing different light modulation structures, image sensors and processors with different fully connected parameters using input training samples and output training samples corresponding to the machine vision intelligent processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network machine vision enhancement chip containing different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using input training samples and output training samples corresponding to the machine vision intelligent processing task;

[0518] Wherein, the input training samples include incident light reflected, transmitted and / or radiated by a target object in a specific machine vision scene, and the output training samples include target object recognition results in the specific machine vision scene; and / or, the input training samples include incident light reflected, transmitted and / or radiated by a target object in a specific machine vision scene, and the output training samples include qualitative analysis results of the target object in the specific machine vision scene.

[0519] In this embodiment, due to the addition of spectrum modulation of light, an artificial neural network optoelectronic chip whose input is an object image and its spectrum can be realized, so that accurate, safe and reliable machine vision target object recognition can be achieved.

[0520] In this embodiment, for a specific machine vision application scenario, the connection weights from the input layer to the linear layer, that is, the system function of the optical filter layer, can be obtained through data training, and the required optical filter layer can be reversely designed and integrated above the image sensor, thereby preparing an optical artificial neural network enhanced machine vision chip that can quickly and accurately identify and judge in machine vision application scenarios.

[0521] It can be understood that the specific modulation pattern of the modulation structure on the optical filter layer is obtained by collecting a large number of target objects in the corresponding machine vision application scenarios in the early stage and training and designing with artificial neural network data. It is usually an irregularly shaped structure, but it may also be a regular shaped structure.

[0522] like Fig.19 As shown, the complete process for enhanced machine vision applications is: the light source is irradiated onto the object to be detected, and then the reflected light is collected by the chip, and then the processor performs algorithm processing to obtain the recognition result, and finally the control mechanism makes the corresponding operation.

[0523] It is understandable that the chip actually uses both the image information and spectral information of the detected object, improving the accuracy and security of the detected object recognition. At the same time, the chip partially implements the artificial neural network in hardware, which improves the speed of detecting the object recognition. In addition, the chip solution can be mass-produced using the existing CMOS process, reducing the size, power consumption and cost of the device.

[0524] Furthermore, when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0525] Furthermore, the light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0526] Furthermore, the optical filter layer is a single-layer structure or a multi-layer structure.

[0527] Furthermore, the light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each micro-nano unit corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

[0528] Furthermore, the micro-nano unit includes a regular structure and / or an irregular structure; and / or the micro-nano unit includes a discrete structure and / or a continuous structure.

[0529] Furthermore, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0530] Furthermore, each group of micro-nano structure arrays has a broadband filtering or narrowband filtering function.

[0531] Furthermore, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0532] Furthermore, the micro-nano unit includes one or more groups of empty structures in the multiple groups of micro-nano structure arrays.

[0533] Furthermore, the micro-nano unit has polarization-independent characteristics.

[0534] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0535] Furthermore, the filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

[0536] Furthermore, the semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

[0537] Furthermore, the thickness of the optical filter layer is 0.1λ to 10λ, wherein λ represents the central wavelength of the incident light.

[0538] The embodiment of the present invention also provides an enhanced machine vision system, including a control mechanism and an optical artificial neural network enhanced machine vision chip as described in the above embodiment. The control mechanism is connected to the optical artificial neural network enhanced machine vision chip, and the control mechanism performs corresponding control according to the recognition result of the artificial neural network enhanced machine vision chip, thereby completing the application goal in the machine vision scene. Here, the control mechanism may be a manipulator, a robotic arm, or an intelligent control button, etc., which is not limited in this embodiment. The control structure can perform corresponding control according to the recognition result of the optical artificial neural network enhanced machine vision chip according to the predetermined control logic.

[0539] It is understandable that machine vision technology is a branch technology of artificial intelligence. It uses machines to replace human eyes for observation and judgment, and is widely used in industrial production, quality inspection, express sorting, unmanned driving and other fields. A typical machine vision system includes an imaging system, an image processing system, a communication and IO system and a linkage mechanism. Among them, the imaging system is responsible for collecting the image information of the target object, which is the key to machine vision technology. At present, machine vision technology only uses the image information of the object, and the accuracy and reliability of measurement and identification need to be improved. The present embodiment makes comprehensive use of the spectral information of the object, so that an enhanced machine vision system with higher accuracy and reliability can be achieved.

[0540] The embodiment of the present invention further provides a method for preparing an optical artificial neural network machine vision enhancement chip as described in any one of the above items, comprising:

[0541] Preparing an optical filter layer including a light modulation structure on the surface of the image sensor;

[0542] Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing;

[0543] connecting the image sensor and the processor;

[0544] The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain information carried by the incident light corresponding to different positions on the surface of the image sensor;

[0545] The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor;

[0546] The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and a second nonlinear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result.

[0547] Furthermore, the method for preparing the optical artificial neural network machine vision enhancement chip further includes: a training process for the optical artificial neural network machine vision enhancement chip, specifically including:

[0548] Using input training samples and output training samples corresponding to the machine vision intelligent processing task, an optical artificial neural network machine vision enhancement chip including different light modulation structures, image sensors and processors with different fully connected parameters is trained to obtain light modulation structures, image sensors and processors that meet the training convergence conditions, and the light modulation structures, image sensors and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors and processors;

[0549] Alternatively, using input training samples and output training samples corresponding to the machine vision intelligent processing task, an optical artificial neural network machine vision enhancement chip including different light modulation structures, image sensors, and processors with different fully connected parameters and different second nonlinear activation parameters is trained to obtain light modulation structures, image sensors, and processors that meet the training convergence conditions, and the light modulation structures, image sensors, and processors that meet the training convergence conditions are used as the trained light modulation structures, image sensors, and processors.

[0550] Furthermore, when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network machine vision enhancement chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

[0551] For example, for the unmanned driving machine vision system, it is necessary to automatically identify the type of obstacles in front of the vehicle, and then perform accurate and fast automatic control. For another example, for the quality inspection machine vision system, it is necessary to be able to accurately and quickly identify the quality defects of the target object, so as to ensure the quality of quality inspection and avoid missed inspections or wrong inspections. For another example, for the surgical navigation machine vision system, the real-time requirements are very high, and the chip provided in this embodiment can well solve the real-time problem.

[0552] It is understandable that when designing the structure of the optical filter layer on the chip, it is necessary to first collect a large number of target objects in the corresponding machine vision application scenarios, and obtain the weight of the linear layer through data training, that is, the system function of the optical filter layer. Then, the required optical filter layer can be reversely designed and integrated above the image sensor.

[0553] It should be noted that the optical artificial neural network enhanced machine vision chip based on micro-nano modulation structure and image sensor provided in this embodiment has the following effects: A. The artificial neural network is partially embedded in the hardware chip, which improves the real-time performance and reliability of machine vision related applications. B. The preparation of the chip can be completed in one-time tape-out through the CMOS process, which is conducive to reducing the device failure rate, improving the device yield rate, and reducing costs. C. Realizing monolithic integration at the wafer level can minimize the distance between the sensor and the optical filter layer, which is conducive to reducing the size of the unit, reducing the device volume and packaging costs.

[0554] It should be noted that for the detailed structural description of the optical artificial neural network enhanced machine vision chip provided in this embodiment, please refer to the introduction of the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here to avoid redundancy. In addition, for the detailed introduction of the method for preparing the optical artificial neural network enhanced machine vision chip, please refer to the introduction of the method for preparing the optical artificial neural network chip in the aforementioned embodiment, and it will not be described here.

[0555] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optical artificial neural network intelligent chip, characterized in that: include: optical filter layer, image sensor and processor; The optical filter layer corresponds to the input layer, the linear layer and the connection weight from the input layer to the linear layer of the artificial neural network, the square detection response of the image sensor corresponds to the first nonlinear activation function in the nonlinear layer of the artificial neural network; the processor corresponds to the full connection and the output layer of the artificial neural network, or the processor corresponds to the full connection, the second nonlinear activation function in the nonlinear layer and the output layer of the artificial neural network; The optical filter layer is arranged on the surface of the image sensor, and the optical filter layer includes an optical modulation structure, and the optical modulation structure is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure, so as to obtain the incident light carrying information corresponding to different positions on the surface of the image sensor; The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor; The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an output signal of the artificial neural network; The optical artificial neural network intelligent chip is used for intelligent processing tasks of target objects; The intelligent processing tasks include at least one or more of intelligent perception, intelligent recognition and intelligent decision-making tasks; The reflected light, transmitted light and / or radiated light of the target object enters the trained optical artificial neural network intelligent chip to obtain the intelligent processing result of the target object; the intelligent processing result at least includes one or more of the intelligent perception result, the intelligent recognition result and / or the intelligent decision result; The trained optical artificial neural network smart chip refers to an optical artificial neural network smart chip including a trained optical modulation structure, an image sensor and a processor; The trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network intelligent chip containing different light modulation structures, image sensors and processors with different fully connected parameters using input training samples and output training samples corresponding to the intelligent processing task; or, the trained light modulation structure, image sensor and processor refer to the light modulation structure, image sensor and processor that meet the training convergence conditions by training an optical artificial neural network intelligent chip containing different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters using input training samples and output training samples corresponding to the intelligent processing task.

2. The optical artificial neural network intelligent chip according to claim 1, characterized in that: The information carried by the incident light includes at least one of light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light.

3. The optical artificial neural network intelligent chip according to claim 1, characterized in that: When training an optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters, or when training an optical artificial neural network smart chip comprising different light modulation structures, image sensors and processors with different fully connected parameters and different second nonlinear activation parameters, the different light modulation structures are designed and implemented by adopting computer optical simulation design.

4. The optical artificial neural network intelligent chip according to any one of claims 1 to 3, characterized in that: The light modulation structure in the optical filter layer includes a regular structure and / or an irregular structure; and / or, the light modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

5. The optical artificial neural network intelligent chip according to any one of claims 1 to 3, characterized in that: The optical filter layer is a single-layer structure or a multi-layer structure.

6. The optical artificial neural network intelligent chip according to any one of claims 1 to 3, characterized in that: The light modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, each of which corresponds to one or more pixel points on the image sensor; and the structures of the various micro-nano units are the same or different.

7. The optical artificial neural network intelligent chip according to claim 6, characterized in that: The micro-nano unit includes a regular structure and / or an irregular structure; and / or, the micro-nano unit includes a discrete structure and / or a continuous structure.

8. The optical artificial neural network intelligent chip according to claim 6, characterized in that: The micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

9. The optical artificial neural network intelligent chip according to claim 8, characterized in that: Each group of micro-nano structure arrays has the function of broadband filtering or narrowband filtering.

10. The optical artificial neural network intelligent chip according to claim 8, characterized in that: Each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

11. The optical artificial neural network intelligent chip according to claim 8, characterized in that: The micro-nano unit includes a plurality of micro-nano structure arrays, and there is one or more groups of empty structures.

12. The optical artificial neural network intelligent chip according to claim 8, characterized in that: The micro-nano unit has polarization-independent characteristics.

13. The optical artificial neural network intelligent chip according to claim 12, characterized in that: The micro-nano unit has four-fold rotational symmetry.

14. The optical artificial neural network intelligent chip according to claim 1, characterized in that: The optical filter layer is composed of one or more layers; The filter layer is made of one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, and perovskite materials; and / or, the filter layer is made of one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, and adjustable Fabry-Perot resonant cavities.

15. The optical artificial neural network intelligent chip according to claim 14, characterized in that: The semiconductor material includes one or more of silicon, silicon oxide, silicon nitride, titanium oxide, a composite material mixed in a preset proportion, and a direct bandgap compound semiconductor material; and / or, the nanostructure includes one or more of nanodot two-dimensional materials, nanocolumn two-dimensional materials, and nanowire two-dimensional materials.

16. The optical artificial neural network intelligent chip according to claim 1, characterized in that: The thickness of the optical filter layer is 0.1λ~10λ, where λ represents the central wavelength of the incident light.

17. The optical artificial neural network intelligent chip according to claim 1, characterized in that: The image sensor is any one or more of the following: CMOS image sensor CIS, charge coupled device CCD, single photon avalanche diode SPAD array and focal plane photodetector array.

18. A smart device, characterized in that: include: An optical artificial neural network intelligent chip as described in any one of claims 1 to 17.

19. The smart device according to claim 18, characterized in that: The smart device includes one or more of a smart phone, a smart computer, a smart identification device, a smart sensing device and a smart decision-making device.

20. A method for preparing an optical artificial neural network intelligent chip according to any one of claims 1 to 17, characterized in that: include: Preparing an optical filter layer including a light modulation structure on the surface of the image sensor; Generate a processor with a function of fully connecting the signal or generate a processor with a function of fully connecting the signal and performing a second nonlinear activation processing; connecting the image sensor and the processor; The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the optical modulation structure through the optical modulation structure, so as to obtain incident light carrying information corresponding to different positions on the surface of the image sensor; the incident light carrying information includes light intensity distribution information, spectrum information, angle information of the incident light, and phase information of the incident light; The image sensor converts the information carried by the incident light corresponding to different positions after being modulated by the optical filter layer into electrical signals corresponding to different positions through the square detection response, and sends the electrical signals corresponding to different positions to the processor; The processor performs full connection processing on the electrical signals corresponding to different position points, or the processor performs full connection processing and second nonlinear activation processing on the electrical signals corresponding to different position points to obtain an output signal of the artificial neural network; When the optical artificial neural network smart chip is used for an intelligent processing task of a target object, the optical artificial neural network smart chip including different light modulation structures, image sensors and processors with different full connection parameters is trained using input training samples and output training samples corresponding to the intelligent processing task to obtain a light modulation structure, image sensor and processor that meet the training convergence conditions; or, the optical artificial neural network smart chip including different light modulation structures, image sensors and processors with different full connection parameters and different second nonlinear activation parameters is trained to obtain a light modulation structure, image sensor and processor that meet the training convergence conditions.

21. The method for preparing an optical artificial neural network intelligent chip according to claim 20, characterized in that: An optical filter layer including a light modulation structure is prepared on the surface of the image sensor, comprising: Growing one or more layers of a preset material on the surface of the image sensor; Etching the one or more layers of the preset material into a light modulation structure pattern to obtain a light filter layer including the light modulation structure; or performing embossing transfer on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure; or obtaining an optical filter layer including an optical modulation structure by externally dynamically modulating the one or more layers of the preset material; or performing zone printing on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure; or performing partitioned growth on the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure; Alternatively, quantum dots are transferred to the one or more layers of the preset material to obtain an optical filter layer including a light modulation structure.

Images