Optical Artificial Neural Network Enhanced Machine Vision Chip and Preparation Method

Abstract

The present invention provides an optical artificial neural network enhanced machine vision chip and a preparation method thereof. The chip provided by the present invention simulates an artificial neural network in a hardware manner and is used for the recognition of target objects in a machine vision scenario. The present invention uses an optical filter layer as the input layer of the artificial neural network, an image sensor as the linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the incident light entering the optical filter layer as the connection weight from the input layer to the linear layer, so that when the enhanced machine vision chip is subsequently used for machine vision intelligent processing, complex signal processing and algorithm processing corresponding to the input layer and the linear layer do not need to be carried out anymore, and the power consumption and delay during the processing of the artificial neural network can be greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optical artificial neural network enhanced machine vision chip and a preparation method thereof. Background Art

[0002] Machine vision technology is a branch technology of artificial intelligence. It uses machines to replace human eyes for observation and judgment, and is widely used in industrial production, quality inspection, express sorting, driverless and other fields.

[0003] Existing machine vision systems include an imaging system, an image processing system, a communication and IO system, and a linkage mechanism. Among them, the imaging system is responsible for collecting image information of the target object to achieve the recognition of the target object. Since the existing machine vision technology needs to first image the target object and then transmit it to the computer for subsequent neural network recognition model algorithm processing, the transmission and processing of a large amount of data cause relatively large power consumption and delay. Summary of the Invention

[0004] In view of the problems existing in the prior art, an embodiment of the present invention provides an optical artificial neural network enhanced machine vision chip and a preparation method thereof.

[0005] Specifically, the embodiment of the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides an optical artificial neural network enhanced machine vision chip, including: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to the input layer of the artificial neural network and the connection weights from the input layer to the linear layer, and the image sensor corresponds to the linear layer of the artificial neural network; the processor corresponds to the non-linear layer and the output layer of the artificial neural network;

[0007] The optical filter layer is disposed on the surface of the photosensitive area of the image sensor. The optical filter layer includes an optical modulation structure. The optical filter layer is used to perform different spectral modulations on the incident light entering different position points of the optical modulation structure through the optical modulation structure, so as to obtain the incident light carrying information corresponding to different position points on the surface of the photosensitive area; the incident light includes reflected light, transmitted light, and / or radiation light of the target object in the machine vision scene;

[0008] The image sensor is used to convert the incident light carrying information corresponding to different position points modulated by the optical filter layer into electrical signals corresponding to different position points, and send the electrical signals corresponding to different position points to the processor; the electrical signal is an image signal modulated by the optical filter layer;

[0009] The processor is used to perform full-connection processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain the machine vision intelligent processing result;

[0010] Among them, the machine vision intelligent processing task includes the recognition, and / or measurement, and / or control of the target object in the machine vision scenario; the machine vision intelligent processing result includes the recognition result, and / or measurement result, and / or control result of the target object in the machine vision scenario.

[0011] Further, the information carried by the incident light includes at least one of light intensity distribution information, spectral information, the angle information of the incident light, and the phase information of the incident light.

[0012] Further, the optical artificial neural network enhanced machine vision chip includes a trained optical modulation structure, an image sensor, and a processor;

[0013] The trained optical modulation structure, image sensor, and processor refer to the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different full-connection parameters and different non-linear activation parameters, which is trained using the input training samples and output training samples corresponding to the machine vision intelligent processing task to obtain an optical modulation structure, image sensor, and processor that meet the training convergence conditions;

[0014] The input training samples include the incident light reflected, transmitted, and / or radiated by the target objects with different machine vision scenarios; the output training samples include the recognition result, and / or measurement result, and / or control result of the target object.

[0015] Further, when training the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different full-connection parameters and non-linear activation parameters, the different optical modulation structures are designed and implemented by means of computer optical simulation design.

[0016] Further, the machine vision scenario includes autonomous driving, robotic surgical navigation, express package sorting, robotic measurement, and beer production line detection.

[0017] Further, when the machine vision scenario is autonomous driving, the input training samples include the incident light reflected, transmitted, and / or radiated by the roads with different road conditions and the front obstacles; the output training samples include the road condition recognition result and the front obstacle recognition result;

[0018] When the machine vision scenario is robot surgical navigation, the input training samples include incident light reflected, transmitted, and / or radiated by the surgical operation object and surgical auxiliary tools; the output training samples include the recognition results of the relative positions of the corresponding surgical operation object and surgical auxiliary tools.

[0019] When the machine vision scenario is express package sorting, the input training samples include incident light reflected, transmitted, and / or radiated by different express packages; the output training samples include the recognition results of the volume, weight, and recipient information of the express packages.

[0020] When the machine vision scenario is robot measurement, the input training samples include incident light reflected, transmitted, and / or radiated by different measurement objects; the output training samples include the recognition results of the positions, shapes, and sizes of the measurement objects.

[0021] When the machine vision scenario is beer production line detection, the input training samples include incident light reflected, transmitted, and / or radiated by different production lines; the output training samples include the working states of the corresponding beer production lines.

[0022] Further, the light modulation structure in the optical filter layer includes regular structures and / or irregular structures; and / or, the light modulation structure in the optical filter layer includes discrete structures and / or continuous structures.

[0023] Further, the light modulation structure in the optical filter layer includes a unit array composed of multiple micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the respective micro-nano units are the same or different.

[0024] Further, the micro-nano unit includes regular structures and / or irregular structures; and / or, the micro-nano unit includes discrete structures and / or continuous structures.

[0025] Further, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of the respective groups of micro-nano structure arrays are the same or different.

[0026] Further, each group of micro-nano structure arrays has the function of broadband filtering or narrowband filtering.

[0027] Further, each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

[0028] Further, one or more groups of the multiple groups of micro-nano structure arrays included in the micro-nano unit are empty structures.

[0029] Further, the micro-nano unit has polarization-independent characteristics.

[0030] Furthermore, the micro-nano unit has four-fold rotational symmetry.

[0031] Furthermore, the optical filter layer is composed of one or more filter layers;

[0032] The filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon SPP micro-nano structures, tunable Fabry-Perot resonators.

[0033] Furthermore, the semiconductor materials include one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and / or, the nanostructures include one or more of two-dimensional nanodot materials, two-dimensional nanocolumn materials, and two-dimensional nanowire materials.

[0034] Furthermore, the thickness of the optical filter layer is 0.1λ - 10λ, where λ represents the central wavelength of the incident light.

[0035] In a second aspect, an embodiment of the present invention provides a machine vision device, including: a control mechanism and the optical artificial neural network enhanced machine vision chip as described above;

[0036] Wherein, the control mechanism is connected to the optical artificial neural network enhanced machine vision chip, and the control mechanism is configured to perform corresponding control according to the machine vision intelligent processing result of the artificial neural network enhanced machine vision chip.

[0037] In a third aspect, an embodiment of the present invention provides a method for preparing an optical artificial neural network enhanced machine vision chip, including:

[0038] Preparing an optical filter layer including an optical modulation structure on the surface of the photosensitive area of the image sensor;

[0039] Generating a processor with the function of performing fully connected processing and non-linear activation processing on signals;

[0040] Connecting the image sensor and the processor;

[0041] Wherein, the optical filter layer is configured to perform different spectral modulations on the incident light entering different position points of the optical modulation structure through the optical modulation structure, so as to obtain incident light carrying information corresponding to different position points on the surface of the photosensitive area; the incident light carrying information includes at least one of light intensity distribution information, spectral information, angle information of the incident light, and phase information of the incident light;

[0042] The image sensor is used to convert the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer into electrical signals corresponding to different position points, and send the electrical signals corresponding to different position points to the processor; the processor is used to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result; the electrical signal is an image signal modulated by the optical filter layer, and the incident light includes reflected light, transmitted light, and / or radiation light of a target object in a machine vision scene.

[0043] Further, an optical filter layer including an optical modulation structure is prepared on the surface of the photosensitive area of the image sensor, including:

[0044] Growing one or more layers of a preset material on the surface of the photosensitive area of the image sensor;

[0045] Etching a pattern of the optical modulation structure on the one or more layers of the preset material to obtain an optical filter layer including the optical modulation structure;

[0046] Or performing imprint transfer on the one or more layers of the preset material to obtain an optical filter layer including the optical modulation structure;

[0047] Or obtaining an optical filter layer including the optical modulation structure by applying external dynamic modulation to the one or more layers of the preset material;

[0048] Or performing zone printing on the one or more layers of the preset material to obtain an optical filter layer including the optical modulation structure;

[0049] Or performing zone growth on the one or more layers of the preset material to obtain an optical filter layer including the optical modulation structure;

[0050] Or performing quantum dot transfer on the one or more layers of the preset material to obtain an optical filter layer including the optical modulation structure.

[0051] Further, it also includes: training the optical artificial neural network enhanced machine vision chip, specifically including:

[0052] Using input training samples and output training samples corresponding to the machine vision intelligent processing task to train an optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and different non-linear activation parameters to obtain an optical modulation structure, image sensor, and processor that meet the training convergence conditions, and using the optical modulation structure, image sensor, and processor that meet the training convergence conditions as the trained optical modulation structure, image sensor, and processor.

[0053] The optical artificial neural network enhanced machine vision chip and preparation method provided in the embodiment of the present invention simulate artificial neural networks in a hardware manner and are used for online identification, measurement or control in machine vision scenarios. That is, the embodiment of the present invention realizes a new enhanced machine vision chip that can realize the function of artificial neural networks. In the enhanced machine vision chip, an optical filter layer is arranged on the surface of the photosensitive area of ​​the image sensor, and the optical filter layer includes a light modulation structure. The optical filter layer is used to perform different spectrum modulations on the incident light entering different positions of the light modulation structure through the light modulation structure, so as to obtain the incident light carrying information corresponding to the different positions on the surface of the photosensitive area. Accordingly, the image sensor is used to convert the incident light carrying information corresponding to the different positions into electrical signals corresponding to the different positions. At the same time, the processor connected to the image sensor is used to perform full connection processing and nonlinear activation processing on the electrical signals corresponding to the different positions to obtain the output signal of the artificial neural network. It can be seen that in the enhanced machine vision chip, the optical filter layer serves as the input layer of the artificial neural network, and the image sensor serves as the linear layer of the artificial neural network. At the same time, the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight from the input layer to the linear layer, that is, the optical filter layer and the image sensor in the enhanced machine vision chip realize the related functions of the input layer and the linear layer in the artificial neural network, that is, the embodiment of the present invention peels off the input layer and the linear layer in the artificial neural network implemented by software in the prior art, and implements the two-layer structure of the input layer and the linear layer in the artificial neural network by hardware, so that when the enhanced machine vision chip is used for the subsequent intelligent processing of the artificial neural network, there is no need to perform complex signal processing and algorithm processing corresponding to the input layer and the linear layer. The processor in the enhanced machine vision only needs to perform related processing with full connection and nonlinear activation of the electrical signal, which can greatly reduce the power consumption and delay of the artificial neural network processing. It can be seen that the embodiment of the present invention uses the optical filter layer as the input layer of the artificial neural network, the image sensor as the linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the incident light entering the optical filter layer as the connection weight from the input layer to the linear layer. The optical filter layer and the image sensor are used to project the information carried by the incident light at different points in the space of the target object into an electrical signal, and then the full connection processing and nonlinear activation processing of the electrical signal are implemented in the processor. It can be seen that the embodiment of the present invention can omit the complex signal processing and algorithm processing corresponding to the input layer and the linear layer in the prior art.

[0054] It can be seen that the enhanced machine vision chip provided by the embodiment of the present invention embeds the artificial neural network part into the image sensor containing various optical filter layers, so as to achieve safe, reliable, fast and accurate recognition of the target object, and then perform machine vision intelligent processing according to the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0056] Figure 1 It is a schematic structural diagram of an optical artificial neural network enhanced machine vision chip provided by the first embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the recognition principle of an optical artificial neural network enhanced machine vision chip provided by an embodiment of the present invention;

[0058] Figure 3 It is a schematic disassembly diagram of an optical artificial neural network enhanced machine vision chip provided by an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of the target object control process in a machine vision scenario provided by an embodiment of the present invention;

[0060] Figure 5 It is a top view of an optical filter layer provided by an embodiment of the present invention;

[0061] Figure 6 It is a top view of another optical filter layer provided by an embodiment of the present invention;

[0062] Figure 7 It is a top view of yet another optical filter layer provided by an embodiment of the present invention;

[0063] Figure 8 It is a top view of yet another optical filter layer provided by an embodiment of the present invention;

[0064] Figure 9 It is a top view of yet another optical filter layer provided by an embodiment of the present invention;

[0065] Figure 10 It is a top view of still another optical filter layer provided by an embodiment of the present invention;

[0066] Figure 11 It is a schematic diagram of the broadband filtering effect of a micro-nano structure provided by an embodiment of the present invention;

[0067] Figure 12 It is a schematic diagram of the narrowband filtering effect of a micro-nano structure provided by an embodiment of the present invention;

[0068] Figure 13It is a schematic flow chart of a method for manufacturing an optical artificial neural network enhanced machine vision chip provided by the third embodiment of the present invention. Detailed implementation manners

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0070] Existing machine vision systems include an imaging system, an image processing system, a communication and IO system, and a linkage mechanism. Since existing machine vision technologies need to first image the target object and then transmit it to a computer for subsequent neural network recognition model algorithm processing, the transmission and processing of a large amount of data result in relatively high power consumption and latency. Based on this, the embodiments of the present invention provide an optical artificial neural network enhanced machine vision chip. The optical filter layer in this enhanced machine vision chip corresponds to the input layer of the artificial neural network, the image sensor corresponds to the linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weights from the input layer to the linear layer. The embodiments of the present invention use the optical filter layer and the image sensor to project the information carried by the incident light at different points in the target object space into electrical signals, and then perform fully connected processing and non-linear activation processing of the electrical signals in the processor. It can be seen that the embodiments of the present invention can eliminate the complex signal processing and algorithm processing corresponding to the input layer and the linear layer in the prior art. The embodiments of the present invention separate the input layer and the linear layer in the artificial neural network implemented by software in the prior art and implement these two layers of structures in the artificial neural network in a hardware manner. As a result, when using this intelligent chip for artificial neural network intelligent processing later, there is no need to perform complex signal processing and algorithm processing corresponding to the input layer and the linear layer, and only the relevant processing of fully connecting the electrical signals and non-linear activation needs to be performed by the processor in the intelligent chip. This can significantly reduce the power consumption and latency during artificial neural network processing. The content provided by the present invention will be explained and described in detail below through specific embodiments.

[0071] As Figure 1 shown, the optical artificial neural network enhanced machine vision chip provided by the first embodiment of the present invention includes: an optical filter layer 1, an image sensor 2, and a processor 3; the optical filter layer 1 corresponds to the input layer of the artificial neural network and the connection weights from the input layer to the linear layer, and the image sensor 2 corresponds to the linear layer of the artificial neural network; the processor 3 corresponds to the non-linear layer and the output layer of the artificial neural network;

[0072] The optical filter layer 1 is disposed on the surface of the photosensitive area of the image sensor. The optical filter layer 1 includes an optical modulation structure. The optical filter layer 1 is configured to perform spectral modulation on the incident light entering different position points of the optical modulation structure by varying the intensity with wavelength, that is, perform different intensity modulations on incident light of different wavelengths, so as to obtain the information carried by the incident light corresponding to different position points on the surface of the photosensitive area; the information carried by the incident light includes the image information of the target object to be processed by the optical artificial neural network enhanced machine vision chip and / or various optical space information. For example, the information carried by the incident light includes at least one of light intensity distribution information, spectral information, the angle information of the incident light, and the phase information of the incident light; the incident light includes the reflected light, transmitted light, and / or radiation light of the target object in the machine vision scenario.

[0073] The image sensor 2 is configured to convert the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer 1 into electrical signals corresponding to different position points, and send the electrical signals corresponding to different position points to the processor 3; the electrical signals are image signals after being modulated by the optical filter layer.

[0074] The processor 3 is configured to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain the output signal of the artificial neural network.

[0075] Wherein, the machine vision intelligent processing task includes the recognition result, and / or measurement result, and / or control result of the target object in the machine vision scenario; the machine vision intelligent processing result includes the recognition result, and / or measurement result, and / or control result of the target object in the machine vision scenario.

[0076] In this embodiment, the optical filter layer 1 is disposed on the surface of the photosensitive area of the image sensor. The optical filter layer 1 includes an optical modulation structure. The optical filter layer 1 is configured to perform different spectral modulations on the incident light entering different position points of the optical modulation structure, so as to obtain the information carried by the modulated incident light corresponding to different position points on the surface of the photosensitive area. Correspondingly, the image sensor 2 is configured to convert the information carried by the incident light corresponding to different position points into electrical signals corresponding to different position points, that is, the image signals after being modulated by the optical filter layer. At the same time, the processor 3 connected to the image sensor 2 is configured to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain the output signal of the artificial neural network.

[0077] In this embodiment, the optical filter layer 1 includes an optical modulation structure. The incident light (such as the reflected light, transmitted light, radiation light, etc. of the target to be recognized) entering different position points of the optical modulation structure is subjected to spectral modulation with different intensities, so as to obtain the information carried by the incident light corresponding to different position points on the surface of the photosensitive area of the image sensor 2.

[0078] In this embodiment, it can be understood that the modulation intensity is related to the specific structural form of the optical modulation structure. For example, different modulation intensities can be achieved by designing different optical modulation structures (such as changing the shape and / or size parameters of the optical modulation structure).

[0079] In this embodiment, it can be understood that the optical modulation structures at different positions on the optical filter layer 1 have different spectral modulation effects on the incident light. The modulation intensity of different wavelength components of the incident light by the optical modulation structure corresponds to the connection strength of the artificial neural network, that is, the connection weights corresponding to the input layer and the connection from the input layer to the linear layer. It should be noted that the optical filter layer 1 is composed of multiple optical filter units. The optical modulation structures at different positions within each optical filter unit are different, so they have different spectral modulation effects on the incident light; the optical modulation structures at different positions between the optical filter units can be the same or different, so they have the same or different spectral modulation effects on the incident light.

[0080] In this embodiment, the image sensor 2 converts the information carried by the incident light corresponding to different position points into electrical signals corresponding to different position points, and sends the electrical signals corresponding to different position points to the processor 3. The image sensor 2 corresponds to the linear layer of the neural network.

[0081] In this embodiment, the processor 3 performs a fully connected process and a non-linear activation process on the electrical signals at different position points, and then obtains the output signal of the artificial neural network.

[0082] It can be understood that the processor 3 corresponds to the non-linear layer and the output layer of the neural network, and can also be understood as corresponding to the remaining layers (all other layers) of the neural network except the input layer and the linear layer.

[0083] In addition, it should be supplemented that the processor 3 can be arranged in the enhanced machine vision chip, that is, the processor 3 can be arranged in the enhanced machine vision chip together with the filter layer 1 and the image sensor 2, or can be separately arranged outside the enhanced machine vision chip and connected to the image sensor 2 in the enhanced machine vision through a data line or a connecting device. This embodiment does not limit this.

[0084] In addition, it should be noted that the processor 3 can be implemented by a computer, or by an ARM or FPGA circuit board with certain computing capabilities, or by a microprocessor. This embodiment does not limit this. In addition, as described above, the processor 3 can be integrated in the enhanced machine vision chip or can be provided independently outside the enhanced machine vision chip. When the processor 3 is provided independently outside the enhanced machine vision chip, the electrical signals in the image sensor 2 can be read out to the processor 3 through a signal readout circuit, and then the processor 3 performs a fully connected process and a non-linear activation process on the read electrical signals.

[0085] In this embodiment, it can be understood that when the processor 3 performs a non-linear activation process, it can be implemented by a non-linear activation function. For example, it can use a Sigmoid function, a Tanh function, a ReLU function, etc. This embodiment does not limit this.

[0086] In this embodiment, the optical filter layer 1 corresponds to the input layer of the artificial neural network and the connection weights from the input layer to the linear layer. The image sensor 2 corresponds to the linear layer of the artificial neural network, converting the information carried by the incident light at different spatial positions into electrical signals. The processor 3 corresponds to the non-linear layer and the output layer of the artificial neural network, fully connecting the electrical signals at different positions, and obtaining the machine vision intelligent processing result through a non-linear activation function.

[0087] For example, the enhanced machine vision chip controls the actions of on-site devices by identifying the target object (identifying the size, shape, color, etc. of the target object), and then based on the identification result. Such as visual inspection of printed circuit boards, automatic flaw detection of steel plate surfaces, measurement of the parallelism and perpendicularity of large workpieces, detection of the volume or impurities of containers, automatic identification and classification of mechanical parts, and measurement of geometric dimensions.

[0088] Such as Figure 2 shown on the left, the optical artificial neural network enhanced machine vision chip includes an optical filter layer 1, an image sensor 2, and a processor 3. In Figure 2 it, the processor 3 is implemented using a signal readout circuit and a computer. Such as Figure 2As shown on the right side, the optical filter layer 1 in the optical artificial neural network enhanced machine vision chip corresponds to the input layer of the artificial neural network, the image sensor 2 corresponds to the linear layer of the artificial neural network, and the processor 3 corresponds to the non-linear layer and the output layer of the artificial neural network. The filtering effect of the optical filter layer 1 on the incident light entering the optical filter layer 1 corresponds to the connection weight from the input layer to the linear layer. It can be seen that the optical filter layer and the image sensor in the enhanced machine vision chip provided in this embodiment implement the related functions of the input layer and the linear layer in the artificial neural network in a hardware manner, so that subsequent complex signal processing and algorithm processing corresponding to the input layer and the linear layer are not required when using this enhanced machine vision chip for machine vision intelligent processing, which can greatly reduce the power consumption and delay during the processing of the artificial neural network.

[0089] As Figure 2 shown on the right side, project / connect the incident light spectrum P at different positions of the optical filter layer 1 λ onto the photocurrent response I of the image sensor N . The processor 3 includes a signal readout circuit and a computer. The signal readout circuit in the processor 3 reads out the photocurrent response and transmits it to the computer, and the computer performs full connection processing and non-linear activation processing on the electrical signal, and finally outputs the machine vision intelligent processing result.

[0090] As Figure 3 shown, the optical modulation structure on the optical filter layer 1 is integrated above the image sensor 2 to modulate the incident light, project / connect the spectral information of the incident light onto different pixel points of the image sensor 2, and obtain an electrical signal containing the spectral information of the incident light and the image information. That is, after the incident light passes through the optical filter layer 1, it is converted into an electrical signal by the image sensor 2 to form an image containing the spectral information of the incident light, and finally the processor 3 connected to the image sensor 2 processes the electrical signal containing the spectral information of the incident light and the image information.

[0091] In this embodiment, the information carried by the incident light may include one or more (including two) of the light intensity distribution information, spectral information, the angle information of the incident light, and the phase information of the incident light.

[0092] For example, in one implementation, the information carried by the incident light may include the light intensity distribution information. In other implementations, multiple information such as the image information of the target object, spectral information, the angle of the incident light, and the phase information of the incident light can be used simultaneously to identify the target object, so that the identification of the target object can be achieved more accurately.

[0093] It can be seen that the optical artificial neural network enhanced machine vision chip provided in this embodiment can simultaneously utilize the image information, spectral information, incident light angle, and incident light phase information of the target object, that is, the incident light at different points in space carries information, and an artificial neural network is embedded in the hardware. Information such as substance composition, image shape, and three-dimensional depth can be further extracted from the spatial image, spectrum, angle, and phase information, thereby improving the recognition accuracy. It can be seen that the embodiment of the present invention realizes a spectrum optical artificial neural network enhanced machine vision chip with low power consumption, low latency, and high accuracy.

[0094] The optical artificial neural network enhanced machine vision chip provided by the embodiment of the present invention realizes a brand-new intelligent chip capable of implementing the functions of an artificial neural network. In this intelligent chip, the optical filter layer serves as the input layer of the artificial neural network, and the image sensor serves as the linear layer of the artificial neural network. At the same time, the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight from the input layer to the linear layer. That is, the optical filter layer and the image sensor in this intelligent chip implement the relevant functions of the input layer and the linear layer in the artificial neural network. That is, the embodiment of the present invention separates the input layer and the linear layer in the artificial neural network implemented by software in the prior art and realizes these two-layer structures of the input layer and the linear layer in the artificial neural network in a hardware manner. As a result, when using this intelligent chip for artificial neural network intelligent processing subsequently, there is no need to perform complex signal processing and algorithm processing corresponding to the input layer and the linear layer. Only the relevant processing of full connection and non-linear activation with electrical signals needs to be performed by the processor in the intelligent chip. In this way, the power consumption and latency during artificial neural network processing can be significantly reduced.

[0095] In addition, it should be noted that in the prior art, only two-dimensional image information of the target object is utilized during the identification of the target object. However, it is difficult to ensure the accuracy of identification with two-dimensional image information. Therefore, based on this, in one implementation, the information carried by the incident light may include light intensity distribution information and spectral information. As a result, when using the optical artificial neural network provided by the present application to enhance the intelligent identification task of the machine vision chip, the light intensity distribution information and spectral information of the target object can be utilized simultaneously. Thus, it can be seen that since the information carried by the incident light covers information such as the image, composition, shape, three-dimensional depth, and structure of the target object, when performing identification processing based on the information carried by the incident light at different points in the space of the target object, multi-dimensional information such as the image, composition, shape, three-dimensional depth, and structure of the target object can be covered, thereby enabling more accurate identification of the target object. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectral information, and the angle information of the incident light, so as to be able to capture more comprehensive information such as the image, composition, shape, three-dimensional depth, and three-dimensional structure of the target object, thereby enabling more accurate identification of the target object. In addition, in another implementation, the information carried by the incident light may further include light intensity distribution information, spectral information, the angle information of the incident light, and the phase information of the incident light, so as to be able to capture more comprehensive information such as the image, composition, shape, three-dimensional depth, and three-dimensional structure of the target object, thereby enabling more accurate identification of the target object.

[0096] The optical artificial neural network enhanced machine vision chip provided by the embodiment of the present invention includes an optical filter layer, an image sensor, and a processor. The optical filter layer is disposed on the surface of the photosensitive area of the image sensor. The optical filter layer includes an optical modulation structure. The optical filter layer is used to perform different spectral modulations on the incident light entering different position points of the optical modulation structure respectively, so as to obtain the information carried by the incident light corresponding to different position points on the surface of the photosensitive area. Correspondingly, the image sensor is used to convert the information carried by the incident light corresponding to different position points into electrical signals corresponding to different position points. At the same time, the processor connected to the image sensor is used to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain the output signal of the artificial neural network. It can be seen that in this enhanced machine vision chip, the optical filter layer serves as the input layer of the artificial neural network, and the image sensor serves as the linear layer of the artificial neural network. At the same time, the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight from the input layer to the linear layer. That is to say, the optical filter layer and the image sensor in this enhanced machine vision chip implement the related functions of the input layer and the linear layer in the artificial neural network. That is, the embodiment of the present invention separates the input layer and the linear layer in the artificial neural network implemented by software in the prior art and implements these two layers of structures in the artificial neural network in a hardware manner. As a result, when using this enhanced machine vision chip for artificial neural network intelligent processing subsequently, there is no need to perform complex signal processing and algorithm processing corresponding to the input layer and the linear layer. It only needs to perform relevant processing of full connection and non-linear activation of electrical signals by the processor in the enhanced machine vision chip. In this way, the power consumption and delay during artificial neural network processing can be significantly reduced.As can be seen, in the embodiment of the present invention, the optical filter layer is used as the input layer of the artificial neural network, and the image sensor is used as the linear layer of the artificial neural network. The filtering effect of the optical filter layer on the incident light entering the optical filter layer is used as the connection weight from the input layer to the linear layer. The spatial spectral information of the target object is projected into an electrical signal by using the optical filter layer and the image sensor, and then the fully connected processing and non-linear activation processing of the electrical signal are implemented in the processor. As can be seen, the embodiment of the present invention can not only eliminate the complex signal processing and algorithm processing corresponding to the input layer and the linear layer in the prior art, but also actually utilize the image information, spectral information, incident light angle and incident light phase information of the target object at the same time, that is, the incident light at different points in the space of the target object carries information. As can be seen, since the incident light at different points in the space of the target object carries information covering the image, composition, shape, three-dimensional depth, structure, etc. of the target object, when the recognition process is carried out based on the information carried by the incident light at different points in the space of the target object, multi-dimensional information such as the image, composition, shape, three-dimensional depth, structure, etc. of the target object can be covered, so that the recognition accuracy can be improved. As can be seen, the optical artificial neural network enhanced machine vision chip provided by the embodiment of the present invention can not only achieve the effects of low power consumption and low latency, but also achieve the effect of high accuracy.

[0097] Based on the content of the above embodiment, in this embodiment, the optical artificial neural network enhanced machine vision chip includes a trained optical modulation structure, an image sensor, and a processor;

[0098] The trained optical modulation structure, image sensor, and processor refer to the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and different non-linear activation parameters, which is trained by using the input training samples and output training samples corresponding to the machine vision intelligent processing task to obtain the optical modulation structure, image sensor, and processor that meet the training convergence conditions;

[0099] The input training samples include the incident light reflected, transmitted, and / or radiated by the target object with different machine vision scenarios; the output training samples include the recognition result, and / or measurement result, and / or control result of the target object.

[0100] In this embodiment, the optical artificial neural network enhanced machine vision chip can be used for the machine vision intelligent processing task of the target object. In this embodiment, it can be understood that machine vision mainly uses a computer to simulate the visual function of a human, extract information from objective things, process and understand it, and finally use it for actual detection, measurement, and control.

[0101] In this embodiment, the reflected light, transmitted light, and / or radiated light of the target object enter the trained optical artificial neural network enhanced machine vision chip to obtain the machine vision intelligent processing result of the target object.

[0102] It can be understood that since the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weights from the input layer to the linear layer of the artificial neural network, during training, changing the optical modulation structure in the optical filter layer is equivalent to changing the connection weights from the input layer to the linear layer of the artificial neural network. Through the training convergence condition, the optical modulation structure suitable for the current machine vision intelligent processing task, as well as the fully connected parameters and non-linear activation parameters suitable for the current machine vision intelligent processing task, are determined, thereby completing the training of the optical artificial neural network enhanced machine vision chip.

[0103] For example, when using this enhanced machine vision chip to perform machine vision intelligent processing tasks, it is first necessary to train the optical artificial neural network enhanced machine vision chip. Here, training the optical artificial neural network enhanced machine vision chip means determining the optical modulation structure suitable for the current machine vision intelligent processing task, as well as the fully connected parameters and non-linear activation parameters suitable for the current machine vision intelligent processing task by collecting a large number of machine vision application scenarios as the training set in the early stage.

[0104] It can be understood that since the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weights from the input layer to the linear layer of the artificial neural network, during training, changing the optical modulation structure in the optical filter layer is equivalent to changing the connection weights from the input layer to the linear layer of the artificial neural network. Through the training convergence condition, the optical modulation structure suitable for the current machine vision intelligent processing task, as well as the fully connected parameters and non-linear activation parameters suitable for the current machine vision intelligent processing task, are determined, thereby completing the training of the enhanced machine vision chip.

[0105] It can be understood that after training the enhanced machine vision chip, this enhanced machine vision chip can be used to perform machine vision intelligent processing tasks. Specifically, after the incident light carrying the image information and spatial spectral information of the target object enters the optical filter layer 1 of the trained enhanced machine vision chip, the optical modulation structure in the optical filter layer 1 modulates the incident light. The intensity of the modulated optical signal is detected by the image sensor 2 and converted into an electrical signal, and then the processor 3 performs fully connected processing and non-linear activation processing to obtain the machine vision intelligent processing result of the target object.

[0106] Such as Figure 4As shown in the figure, the complete process for controlling a target object in a machine vision application scenario is as follows: The light source 200 irradiates the target object 300, and then the reflected light or transmitted light of the target object is collected by the optical artificial neural network enhanced machine vision chip 100, or the light directly radiated outward by the target object is collected by the optical artificial neural network enhanced machine vision chip 100. After being processed by the optical filter layer, image sensor, and processor in the enhanced machine vision, the machine vision intelligent processing result can be obtained. Finally, the control mechanism makes corresponding operations based on the intelligent processing result.

[0107] Among them, the trained optical artificial neural network enhanced machine vision chip refers to the optical artificial neural network enhanced machine vision chip including a trained optical modulation structure, an image sensor, and a processor; the trained optical modulation structure, image sensor, and processor refer to the optical artificial neural network enhanced machine vision chip containing different optical modulation structures, image sensors, and processors with different fully connected parameters and non-linear activation parameters, which is trained using the input training samples and output training samples corresponding to the machine vision intelligent processing task to obtain an optical modulation structure, image sensor, and processor that meet the training convergence conditions.

[0108] For example, for the machine vision intelligent processing task, the input training sample corresponding to the machine vision intelligent processing task is the target object sample in the machine vision scenario, and the output training sample corresponding to the machine vision intelligent processing task is the machine vision intelligent processing result of the target object sample in the machine vision scenario. It can be understood that for the machine vision intelligent processing task, since the advantage of the enhanced machine vision chip provided in this embodiment is also that it can obtain image information, spectral information, incident light angle information, and incident light phase information at different points in the target object space, therefore, to make full use of this advantage, the real target object is preferably used as the target object sample for the input training sample, rather than the two-dimensional image of the target object. Of course, this does not mean that the two-dimensional image cannot be used as the target object sample.

[0109] In this embodiment, the optical filter layer 1 serves as the input layer of the neural network, and the image sensor 2 serves as the linear layer of the neural network. To minimize the loss function of the neural network, the modulation intensity of different wavelength components in the incident light of the target object by the optical modulation structure in the optical filter layer is used as the connection weight from the input layer to the linear layer of the neural network. By adjusting the structure of the filter, the modulation intensity of different wavelength components in the incident light of the target object can be adjusted, thereby realizing the adjustment of the connection weight from the input layer to the linear layer, and further optimizing the training of the neural network.

[0110] Therefore, in this embodiment, the optical modulation structure is obtained based on neural network training. The computer performs optical simulation on the training samples to obtain the sample modulation intensities of different wavelength components of the incident light of the target object by the optical modulation structure in the training samples. The sample modulation intensities are used as the connection weights from the input layer to the linear layer of the neural network for non-linear activation, and the neural network is trained using the training samples corresponding to the intelligent processing tasks until the neural network converges. At this time, the corresponding training sample optical modulation structure is used as the optical filter layer for the corresponding machine vision intelligent processing task.

[0111] It can be seen that in this embodiment, by implementing the input layer (optical filter layer) and the linear layer (image sensor) of the neural network at the physical layer, not only can the complex signal processing and algorithm processing corresponding to the input layer and the linear layer in the prior art be omitted, but in fact, the image information, spectral information, incident light angle, and incident light phase information of the target object are simultaneously utilized in this embodiment of the present invention, that is, the incident light at different points in the space of the target object carries information. It can be seen that since the incident light at different points in the space of the target object carries information covering the image, composition, shape, three-dimensional depth, structure, etc. of the target object, when performing recognition processing based on the information carried by the incident light at different points in the space of the target object, multi-dimensional information such as the image, composition, shape, three-dimensional depth, and structure of the target object can be covered, thereby improving the recognition accuracy. It can be seen that the optical artificial neural network enhanced machine vision chip provided by this embodiment of the present invention can not only achieve the effects of low power consumption and low latency, but also achieve the effect of high accuracy.

[0112] Based on the content of the above embodiment, in this embodiment, when training an optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and non-linear activation parameters, the different optical modulation structures are designed and implemented by means of computer optical simulation design.

[0113] In this embodiment, the optical modulation structure is designed through computer optical simulation, and the optical modulation structure is adjusted through optical simulation until the neural network converges, and then the corresponding optical modulation structure is determined as the size of the optical modulation structure that finally needs to be fabricated, saving prototype production time and cost, improving product efficiency, and easily solving complex optical problems. For example, the optical modulation structure can be simulated and designed by FDTD software, and the optical modulation structure can be changed in the optical simulation, so that the modulation intensity of the optical modulation structure for different incident lights can be accurately predicted and used as the connection weight between the input layer and the linear layer of the neural network to train the optical artificial neural network enhanced machine vision chip to accurately obtain the optical modulation structure.

[0114] It can be seen that in this embodiment, by adopting the method of computer optical simulation design for the optical modulation structure, the prototype production time and cost of the optical modulation structure are saved, and the product efficiency is improved.

[0115] Based on the above embodiment, the machine vision scenarios include autonomous driving, robotic surgical navigation, express package sorting, robotic measurement, and beer production line detection;

[0116] When the machine vision scenario is autonomous driving, the input training samples include incident light reflected, transmitted, and / or radiated by roads with different road conditions and front obstacles; the output training samples include road condition recognition results and front obstacle recognition results;

[0117] When the machine vision scenario is robotic surgical navigation, the input training samples include incident light reflected, transmitted, and / or radiated by surgical operation objects and surgical auxiliary tools; the output training samples include relative position recognition results of the corresponding surgical operation objects and surgical auxiliary tools;

[0118] When the machine vision scenario is express package sorting, the input training samples include incident light reflected, transmitted, and / or radiated by different express packages; the output training samples include recognition results of the volume, weight, and recipient information of the express packages;

[0119] When the machine vision scenario is robotic measurement, the input training samples include incident light reflected, transmitted, and / or radiated by different measurement objects; the output training samples include recognition results of the position, shape, and size of the measurement objects;

[0120] When the machine vision scenario is beer production line detection, the input training samples include incident light reflected, transmitted, and / or radiated by different production lines; the output training samples include the working status of the corresponding beer production line.

[0121] In this embodiment, the machine vision scenarios can include autonomous driving, robotic surgical navigation, express package sorting, robotic measurement, and beer production line detection.

[0122] When the machine vision scenario is autonomous driving, in this embodiment, the incident light of roads with different road conditions and front obstacles is used as the input training samples of the chip, and the corresponding road condition recognition results and front obstacle recognition results are used as the training output samples, so that the trained chip can automatically recognize roads with different road conditions, and then can be used for autonomous driving to plan corresponding driving paths.

[0123] When the machine vision scenario is robotic surgical navigation, in this embodiment, the incident light of the surgical operation object and the surgical auxiliary tool is used as the input training sample of the chip, and the recognition result of the relative position of the corresponding surgical operation object and the surgical auxiliary tool is used as the training output sample, so that the trained chip can navigate and locate the corresponding lesion during the operation, improving the success rate of the operation.

[0124] When the machine vision scenario is express package sorting, in this embodiment, the incident light of different express packages is used as the input training sample, and the recognition results of the volume, weight and recipient information of the corresponding express packages are used as the training output samples, so that the trained chip can accurately and in real time sort different express packages.

[0125] When the machine vision scenario is robotic measurement, in this embodiment, the incident light of different measurement objects is used as the input training sample, and the recognition results of the position, shape and size of the corresponding measurement objects are used as the training output samples, so that the trained chip can accurately measure different target objects, improving the measurement accuracy.

[0126] When the machine vision scenario is beer production line detection, in this embodiment, the incident light of different production lines is used as the input training sample, and the working state of the corresponding beer production line is used as the output training sample, so that the trained chip can perform corresponding operation processing according to the states of different production lines, improving the production efficiency of the production line.

[0127] It can be seen that machine vision technology is a branch technology of artificial intelligence. By using machines to replace human eyes for observation and judgment, it is widely used in industrial production, quality inspection, express sorting, driverless and other fields. The optical artificial neural network enhanced machine vision chip provided in this embodiment can accurately identify, measure and / or control target objects in different machine vision scenarios, realizing enhanced machine vision with higher accuracy and reliability.

[0128] Based on the content of the above embodiments, in this embodiment, the optical modulation structure in the optical filter layer includes a regular structure and / or an irregular structure.

[0129] In this embodiment, it can be understood that the optical modulation structure including a regular structure may include: the smallest modulation unit included in the optical modulation structure is a regular structure, the arrangement mode of the smallest modulation units included in the optical modulation structure is regular, and the smallest modulation unit included in the optical modulation structure is a regular structure, and at the same time, the arrangement mode of the smallest modulation units is also regular, etc.

[0130] Among them, the minimum modulation unit included in the optical modulation structure can be a regular shape such as a rectangle, a square, or a circle. The arrangement of the minimum modulation units included in the optical modulation structure can be that the minimum modulation units are arranged in a regular array form, a regular circular form, a regular trapezoidal form, a regular polygonal form, etc.

[0131] In this embodiment, the fact that the optical modulation structure includes an irregular structure here can mean: the minimum modulation unit included in the optical modulation structure is an irregular structure, such as an irregular polygon, a random shape, or other irregular shapes. In addition, the fact that the optical modulation structure includes an irregular structure here can also mean: the arrangement of the minimum modulation units included in the optical modulation structure is irregular, such as an irregular polygonal form, a random arrangement form, etc. In addition, the fact that the optical modulation structure includes an irregular structure here can also mean: the minimum modulation unit included in the optical modulation structure is an irregular structure, and at the same time, the arrangement of the minimum modulation units is also irregular, etc.

[0132] Based on the content of the above embodiments, in this embodiment, the optical modulation structure in the optical filter layer includes a discrete structure and / or a continuous structure.

[0133] In this embodiment, the optical modulation structure in the optical filter layer can include a discrete structure, can also include a continuous structure, or can include both a discrete structure and a continuous structure.

[0134] In this embodiment, the fact that the optical modulation structure includes a continuous structure here can mean: the optical modulation structure is composed of continuous modulation patterns; the fact that the optical modulation structure includes a discrete structure here can mean: the optical modulation structure is composed of discrete modulation patterns.

[0135] It can be understood that the continuous modulation patterns here can refer to linear patterns, wavy line patterns, broken line patterns, etc.

[0136] It can be understood that the discrete modulation patterns here can refer to modulation patterns formed by discrete graphics (such as discrete points, discrete triangles, discrete stars, etc.).

[0137] In this embodiment, it should be noted that the optical modulation structure has different modulation effects on lights of different wavelengths. The specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different filter structures, the corresponding transmission spectra are different after light passes through different groups of filter structures.

[0138] Based on the content of the above embodiments, in this embodiment, the optical filter layer is a single-layer structure or a multi-layer structure.

[0139] In this embodiment, it should be noted that the optical filter layer can be a single-layer filter structure or a multi-layer filter structure. For example, it can be a multi-layer structure such as two layers, three layers, four layers, etc.

[0140] In this embodiment, as Figure 1 shown, the optical filter layer 1 is a single-layer structure. The thickness of the optical filter layer 1 is related to the target wavelength range. For wavelengths from 400 nm to 10 μm, the thickness of the grating structure can be from 50 nm to 5 μm.

[0141] It can be understood that since the function of the optical filter layer 1 is to perform spectral modulation on the incident light, preferably, materials with high refractive index and low loss are selected for preparation. For example, silicon, germanium, germanium-silicon materials, silicon compounds, germanium compounds, group III-V materials, etc. can be selected for preparation. Among them, silicon compounds include but are not limited to silicon nitride, silicon dioxide, silicon carbide, etc.

[0142] In addition, it should be noted that in order to form more or more complex connection weights between the input layer and the linear layer, preferably, the optical filter layer 1 can be set as a multi-layer structure, and the corresponding optical modulation structures of each layer can be set as different structures, so as to increase the spectral modulation ability of the optical filter layer for the incident light, so that more or more complex connection weights can be formed between the input layer and the linear layer, and further improve the accuracy of the enhanced machine vision chip when processing machine vision intelligent processing tasks.

[0143] In addition, it should be noted that for a filter layer including a multi-layer structure, the materials of each layer structure can be the same or different. For example, for the two-layer optical filter layer 1, the first layer can be a silicon layer and the second layer can be a silicon nitride layer.

[0144] It should be noted that the thickness of the optical filter layer 1 is related to the target wavelength range. For wavelengths from 400 nm to 10 μm, the total thickness of the multi-layer structure can be from 50 nm to 5 μm.

[0145] Based on the content of the above embodiment, in this embodiment, the optical modulation structure in the optical filter layer includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the respective micro-nano units are the same or different.

[0146] In this embodiment, in order to obtain connection weights distributed in an array (the connection weights for connecting the input layer and the linear layer) to facilitate subsequent fully connected and non-linear activation processing by the processor, preferably, in this embodiment, the optical modulation structure is in the form of an array structure. Specifically, the optical modulation structure includes a unit array composed of a plurality of micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor. It should be noted that the structures of the respective micro-nano units may be the same or different. In addition, it should be noted that the structures of the respective micro-nano units may be periodic or aperiodic. In addition, it should be noted that each micro-nano unit may further include multiple groups of micro-nano structure arrays, and the structures of the respective groups of micro-nano structure arrays may be the same or different, etc. It should be noted that the structure of each micro-nano unit is a structure designed according to the corresponding machine vision intelligent processing task. For example, if the machine vision intelligent processing task is to plan the path of autonomous driving, then the structure of each micro-nano unit is trained based on the incident light reflected, transmitted, or radiated by roads with different road conditions as the input and the corresponding driving path as the output. Thus, it can be seen that in this embodiment, different micro-nano modulation structures can be designed corresponding to different machine vision intelligent processing tasks, and then the corresponding machine vision applications can be accurately and quickly completed.

[0147] The following is an example in conjunction with Figures 5 - 9 In this embodiment, as Figure 5 shown, the optical filter layer 1 includes a plurality of repeated continuous or discrete micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit has the same structure (and each micro-nano unit is an aperiodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2; as Figure 6 shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit has the same structure (the difference from Figure 5 is that Figure 6 each micro-nano unit in Figure 7 is a periodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2; as Figure 6 shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit has the same structure (and each micro-nano unit is a periodic structure), and each micro-nano unit corresponds to one or more pixel points on the image sensor 2. The difference from Figure 7 is that Figure 8 the unit shape of the periodic array in each micro-nano unit in Figure 6The difference lies in that each micro-nano unit structure is different from each other, and each micro-nano unit corresponds to one or more pixel points on the image sensor 2. In this embodiment, the optical filter layer 1 includes a plurality of different micro-nano units, that is, the modulation effects of different regions on the incident light in machine vision are different, thereby increasing the design freedom, and further improving the recognition accuracy. For example, Figure 9 As shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit has the same structure, and is different from Figure 5 in that each micro-nano unit is composed of a discrete aperiodic array structure, and each micro-nano unit corresponds to one or more pixel points on the image sensor 2.

[0148] In this embodiment, the micro-nano unit has different modulation effects on lights of different wavelengths. The specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different filter structures, the corresponding transmission spectra are different after light passes through different groups of filter structures.

[0149] Based on the content of the above embodiments, in this embodiment, the micro-nano unit includes a regular structure and / or an irregular structure; and / or, the micro-nano unit includes a discrete structure and / or a continuous structure.

[0150] In this embodiment, the micro-nano unit may only include a regular structure, or only include an irregular structure, or may include both a regular structure and an irregular structure.

[0151] In this embodiment, the micro-nano unit including a regular structure here may mean that: the smallest modulation unit included in the micro-nano unit is a regular structure, the arrangement manner of the smallest modulation units included in the micro-nano unit is regular, the smallest modulation unit included in the micro-nano unit is a regular structure, and at the same time, the arrangement manner of the smallest modulation units is also regular, etc. Among them, the smallest modulation unit may be a regular figure such as a rectangle, a square, and a circle, and the arrangement manner of the smallest modulation units included in the micro-nano unit may be a regular array form, a circular form, a trapezoidal form, a polygonal form, etc.

[0152] In this embodiment, the micro-nano unit including an irregular structure here may mean that: the smallest modulation unit included in the micro-nano unit is an irregular structure, the arrangement manner of the smallest modulation units included in the micro-nano unit is irregular, the smallest modulation unit included in the micro-nano unit is an irregular structure, and at the same time, the arrangement manner of the smallest modulation units is also irregular, etc.

[0153] Among them, the smallest modulation unit included in the micro-nano unit may be an irregular figure such as an irregular polygon, a random shape, etc., and the arrangement manner of the smallest modulation units included in the micro-nano unit may be an irregular polygonal form, a random arrangement form, etc.

[0154] In this embodiment, the micro-nano units in the optical filter layer may include a discrete structure, a continuous structure, or both a discrete structure and a continuous structure.

[0155] In this embodiment, that the micro-nano unit includes a continuous structure may mean that the micro-nano unit is composed of a continuous modulation pattern; that the micro-nano unit includes a discrete structure may mean that the micro-nano unit is composed of a discrete modulation pattern.

[0156] It can be understood that the continuous modulation pattern here may refer to a linear pattern, a wavy pattern, a broken-line pattern, etc.

[0157] It can be understood that the discrete modulation pattern here may refer to a modulation pattern formed by discrete graphics (such as discrete points, discrete triangles, discrete stars, etc.).

[0158] In this embodiment, it should be noted that different micro-nano units have different modulation effects on light of different wavelengths. The specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different micro-nano units, the corresponding transmission spectra are different after light passes through different groups of micro-nano units.

[0159] Based on the content of the above embodiment, in this embodiment, the micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of each group of micro-nano structure arrays are the same or different.

[0160] In this embodiment, as Figure 5 shown, the optical filter layer 1 includes multiple repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit includes multiple groups of micro-nano structure arrays. For example, the micro-nano unit 11 includes 4 different micro-nano structure arrays 110, 111, 112, and 113, and the filtering unit 44 includes 4 different micro-nano structure arrays 440, 441, 442, and 443. As Figure 10 shown, the optical filter layer 1 includes multiple micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit includes multiple groups of micro-nano structure arrays. For example, the micro-nano unit 11 includes 4 identical micro-nano structure arrays 110, 111, 112, and 113.

[0161] It should be noted that only the micro-nano unit including four groups of micro-nano structure arrays is taken as an example here, which does not play a restrictive role. In actual applications, micro-nano units including six groups, eight groups or other numbers of micro-nano structure arrays can also be set according to needs.

[0162] In this embodiment, each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on lights of different wavelengths, and the modulation effects of each group of filtering structures on the input light are also different. The specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different micro-nano structure arrays, the corresponding transmission spectra are different after the light passes through different groups of micro-nano structure arrays.

[0163] Based on the content of the above embodiment, in this embodiment, each group of micro-nano structure arrays has the function of broadband filtering or narrowband filtering.

[0164] In this embodiment, in order to obtain the modulation intensity of different wavelength components of the incident light of the target object as the connection weight between the input layer and the linear layer of the neural network, broadband filtering and narrowband filtering are realized by adopting different micro-nano structure arrays. Therefore, in this embodiment, the micro-nano structure array obtains the modulation intensity of different wavelength components of the incident light of the target object by performing broadband filtering or narrowband filtering on the incident light of the target object. As Figure 11 and Figure 12 shown, each group of micro-nano structure arrays in the optical filter layer has the function of broadband filtering or narrowband filtering.

[0165] It can be understood that for each group of micro-nano structure arrays, they can all have the function of broadband filtering, or they can all have the function of narrowband filtering, or some can have the function of broadband filtering and some can have the function of narrowband filtering. In addition, the broadband filtering range and narrowband filtering range of each group of micro-nano structure arrays can be the same or different. For example, by designing the size parameters such as the period, duty cycle, radius, side length, etc. of each group of micro-nano structures in the micro-nano unit, it has the function of narrowband filtering, that is, only light of one (or a few) wavelengths can pass through. Another example is that by designing the size parameters such as the period, duty cycle, radius, side length, etc. of each group of micro-nano structures in the micro-nano unit, it has the function of broadband filtering, that is, light of more wavelengths or all wavelengths is allowed to pass through.

[0166] It can be understood that in specific use, the filtering state of each group of micro-nano structure arrays can be determined by means of broadband filtering, narrowband filtering or a combination thereof according to the application scenario.

[0167] Based on the content of the above embodiment, in this embodiment, each group of micro-nano structure arrays is a periodic structure array or an aperiodic structure array.

[0168] In this embodiment, each group of micro-nano structure arrays can all be periodic structure arrays, or they can all be aperiodic structure arrays, or some can be periodic structure arrays and some can be aperiodic structure arrays. Among them, the periodic structure array is easy to perform optical simulation design, and the aperiodic structure array can achieve more complex modulation effects.

[0169] In this embodiment, as Figure 5 shown, the optical filter layer 1 includes a plurality of repeating micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit is composed of multiple groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays are different from each other, and the micro-nano structure arrays are aperiodic structures. Among them, the aperiodic structure means that the shapes of the modulation holes on the micro-nano structure array are arranged in an aperiodic arrangement. As Figure 5 shown, the micro-nano unit 11 includes 4 different aperiodic structure arrays 110, 111, 112, and 113, and the micro-nano unit 44 includes 4 different aperiodic structure arrays 440, 441, 442, and 443. The micro-nano structure arrays with aperiodic structures are designed through neural network data training for intelligent processing tasks in the early stage, and are usually structures with irregular shapes. As Figure 6 shown, the optical filter layer 1 includes a plurality of repeating micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit is composed of multiple groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays are different from each other. Different from Figure 5 this, the micro-nano structure arrays are periodic structures. Among them, the periodic structure means that the shapes of the modulation holes on the micro-nano structure array are arranged in a periodic arrangement, and the size of the period is usually 20 nm to 50 μm. As Figure 6 shown, the micro-nano unit 11 includes 4 different periodic structure arrays 110, 111, 112, and 113, and the micro-nano unit 44 includes 4 different periodic structure arrays 440, 441, 442, and 443. The filter structures with periodic structures are designed through neural network data training for intelligent processing tasks in the early stage, and are usually structures with irregular shapes. As Figure 7 shown, the optical filter layer 1 includes a plurality of different micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit is composed of multiple groups of micro-nano structure arrays, and the structures of the micro-nano structure arrays are different from each other, and the micro-nano structure arrays are periodic structures. Among them, the periodic structure means that the shapes on the filter structure are arranged in a periodic arrangement, and the size of the period is usually 20 nm to 50 μm. As Figure 7 shown, the micro-nano structure arrays of the micro-nano unit 11 and the micro-nano unit 12 are different from each other. The micro-nano unit 11 includes 4 different periodic structure arrays 110, 111, 112, and 113, and the micro-nano unit 44 includes 4 different periodic structure arrays 440, 441, 442, and 443. The micro-nano structure arrays with periodic structures are designed through neural network data training for intelligent processing tasks in the early stage, and are usually structures with irregular shapes.

[0170] It should be noted that Figures 5 - 9Each micro-nano unit includes four groups of micro-nano structure arrays, which are respectively formed by four different-shaped modulation holes, and the four groups of micro-nano structure arrays are used to have different modulation effects on incident light. It should be noted that here only a micro-nano unit including four groups of micro-nano structure arrays is used as an example for illustration, which does not play a restrictive role. In actual applications, micro-nano units including six groups, eight groups or other numbers of micro-nano structure arrays can also be set according to needs. In this embodiment, the four different shapes can be circular, cross-shaped, regular polygon and rectangular (not limited to this).

[0171] In this embodiment, each group of micro-nano structure arrays in the micro-nano unit has different modulation effects on lights of different wavelengths, and the modulation effects on the input light between the groups of micro-nano structure arrays are also different. The specific modulation methods include but are not limited to scattering, absorption, interference, surface plasmon, resonance enhancement, etc. By designing different micro-nano structure arrays, the corresponding transmission spectra are different after the light passes through different groups of micro-nano structure arrays.

[0172] Based on the content of the above embodiment, in this embodiment, one or more groups of the micro-nano structure arrays included in the micro-nano unit are empty structures.

[0173] The following is an example combined with Figure 9 the example shown. In this embodiment, as Figure 9 shown, the optical filter layer 1 includes a plurality of repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit is composed of multiple groups of micro-nano structure arrays, and the corresponding structures of the multiple groups of micro-nano structure arrays are different from each other. The micro-nano structure arrays are periodic structures. Different from the above embodiment, for any micro-nano unit, one or more groups of empty structures are included, and the empty structures are used to directly transmit incident light. It can be understood that when one or more groups of empty structures are included in the multiple groups of micro-nano structure arrays, a richer spectrum modulation effect can be formed, so as to meet the spectrum modulation requirements in specific scenarios (or meet the specific connection weight requirements between the input layer and the linear layer in specific scenarios).

[0174] As Figure 9As shown, each micro-nano unit includes a group of micro-nano structure arrays and three groups of empty structures. Micro-nano unit 11 contains one aperiodic structure array 111, micro-nano unit 22 contains one aperiodic structure array 221, micro-nano unit 33 contains one aperiodic structure array 331, micro-nano unit 44 contains one aperiodic structure array 441, micro-nano unit 55 contains one aperiodic structure array 551, and micro-nano unit 66 contains one aperiodic structure array 661. The micro-nano structure array is used to perform different modulations on the incident light. It should be noted that here only an example of including a group of micro-nano structure arrays and three groups of empty structures is given, which does not play a restrictive role. In actual applications, micro-nano units including a group of micro-nano structure arrays and five groups of empty structures or micro-nano structure arrays with other numbers of groups can also be set according to needs. In this embodiment, the micro-nano structure array can be made of modulation holes in the shapes of circles, crosses, regular polygons, and rectangles (not limited to this).

[0175] It should be noted that among the multiple groups of micro-nano structure arrays included in the micro-nano unit, none of them may include empty structures, that is, the multiple groups of micro-nano structure arrays can be aperiodic structure arrays or periodic structure arrays.

[0176] Based on the content of the above embodiment, in this embodiment, the micro-nano unit has polarization-independent characteristics.

[0177] In this embodiment, due to the polarization-independent characteristics of the micro-nano unit, the optical filter layer is insensitive to the polarization of the incident light, thus realizing an optical artificial neural network enhanced machine vision chip that is insensitive to both the incident angle and polarization. The optical artificial neural network enhanced machine vision chip provided by the embodiment of the present invention is insensitive to the incident angle and polarization characteristics of the incident light, that is, the measurement result will not be affected by the incident angle and polarization characteristics of the incident light, so as to ensure the stability of the spectral measurement performance, and further ensure the stability of the machine vision intelligent processing. It should be noted that the micro-nano unit can also have polarization-dependent characteristics.

[0178] Based on the content of the above embodiment, in this embodiment, the micro-nano unit has four-fold rotational symmetry.

[0179] In this embodiment, it should be noted that four-fold rotational symmetry belongs to a specific case of polarization-independent characteristics. By designing the micro-nano unit into a structure with four-fold rotational symmetry, the requirements of polarization-independent characteristics can be met.

[0180] The following combines Figure 7 The example shown is used for illustration. In this embodiment, as Figure 7As shown, the optical filter layer 1 includes multiple repeated micro-nano units, such as 11, 22, 33, 44, 55, 66. Each micro-nano unit is composed of multiple groups of micro-nano structure arrays, and the structures corresponding to the multiple groups of micro-nano structure arrays are different from each other. The micro-nano structure array is a periodic structure. Different from the above embodiments, the structures corresponding to each group of micro-nano structure arrays can be structures with four-fold rotational symmetry such as circles, crosses, regular polygons, rectangles, etc., that is, after the structure is rotated 90°, 180°, and 270°, it coincides with the original structure, so that the structure has the characteristic of polarization independence, and the same intelligent recognition effect can be obtained when different polarized lights are incident.

[0181] Based on the content of the above embodiments, in this embodiment, the optical filter layer is composed of one or more filter layers;

[0182] The filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon polaritons (SPP) micro-nano structures, tunable Fabry-Perot cavities (Fabry-perot Cavity, FP cavity).

[0183] The semiconductor materials include one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and / or, the nanostructures include one or more of two-dimensional nanodot materials, two-dimensional nanorod materials, and two-dimensional nanowire materials.

[0184] Among them, the combination of photonic crystals, metasurfaces and random structures can be compatible with the CMOS process and have a good modulation effect. Other materials can also be filled in the micropores of the micro-nano modulation structure for surface smoothing; quantum dots and perovskites can make the volume of a single modulation structure the smallest by utilizing the spectral modulation characteristics of the materials themselves; SPP has a small volume and can achieve polarization-related optical modulation; liquid crystals can be dynamically regulated by voltage to improve the spatial resolution; tunable Fabry-Perot cavities can be dynamically regulated to improve the spatial resolution.

[0185] Based on the content of the above embodiments, in this embodiment, the thickness of the optical filter layer is 0.1λ - 10λ, where λ represents the central wavelength of the incident light.

[0186] In this embodiment, it should be noted that if the thickness of the optical filter layer is much smaller than the central wavelength of the incident light, it cannot play an effective spectrum modulation role; if the thickness of the optical filter layer is much larger than the central wavelength of the incident light, it is difficult to fabricate in terms of technology and will introduce relatively large optical losses. Therefore, in this embodiment, in order to reduce optical losses, be easy to fabricate, and ensure an effective spectrum modulation effect, the overall size (area) of each micro-nano unit in the optical filter layer 1 is usually λ 2 ~10 5 λ 2 , and the thickness is usually 0.1λ to 10λ (λ represents the central wavelength of the incident light of the target object). As Figure 5 shown, the overall size of each micro-nano unit is 0.5μm 2 ~40000μm 2 , and the dielectric material in the optical filter layer 1 is polysilicon with a thickness of 50nm to 2μm.

[0187] Based on the content of the above embodiment, in this embodiment, the image sensor is any one or more of the following:

[0188] CMOS image sensor (Contact Image Sensor, CIS), charge coupled device (Charge Coupled Device, CCD), single photon avalanche diode (Single Photon Avalanche Diode, SPAD) array, and focal plane photodetector array.

[0189] In this embodiment, it should be noted that by using a wafer-level CMOS image sensor CIS and achieving monolithic integration at the wafer level, the distance between the image sensor and the optical filter layer can be minimized to the greatest extent, which is beneficial to reducing the size of the unit, reducing the device volume and packaging cost. SPAD can be used for weak light detection, and CCD can be used for strong light detection.

[0190] In this embodiment, the optical filter layer and the image sensor can be fabricated by complementary metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS) integrated process, which is beneficial to reducing the device failure rate, improving the device yield and reducing the cost. For example, a layer or multiple layers of dielectric materials can be directly grown on the image sensor, followed by etching. Before removing the sacrificial layer used for etching, metal materials are deposited, and finally the sacrificial layer is removed to prepare the optical filter layer.

[0191] Based on the content of the above embodiment, in this embodiment, the types of the artificial neural network include: feedforward neural network.

[0192] In this embodiment, a feedforward neural network (FNN), also known as a deep feedforward network (DFN) or a multi-layer perceptron (MLP), is the simplest neural network, with neurons arranged in layers. Each neuron is only connected to the neurons in the previous layer, receives the output of the previous layer, and outputs it to the next layer, without feedback between layers. The feedforward neural network has a simple structure, is easy to implement on hardware, has a wide range of applications, can approximate any continuous function and square-integrable function with arbitrary precision, and can accurately implement any finite training sample set. The feedforward network is a static non-linear mapping. Through the composite mapping of simple non-linear processing units, complex non-linear processing capabilities can be obtained.

[0193] Based on the content of the above embodiment, in this embodiment, a light-transmitting medium layer is provided between the optical filter layer and the image sensor.

[0194] In this embodiment, it should be noted that setting a light-transmitting medium layer between the optical filter layer and the image sensor can effectively separate the optical filter layer and the image sensor layer, avoiding mutual interference between the two.

[0195] Based on the content of the above embodiment, in this embodiment, the image sensor is a front-illuminated type, including: a metal wire layer and a light detection layer arranged from top to bottom, and the optical filter layer is integrated on the side of the metal wire layer away from the light detection layer; or,

[0196] The image sensor is a back-illuminated type, including: a light detection layer and a metal wire layer arranged from top to bottom, and the optical filter layer is integrated on the side of the light detection layer away from the metal wire layer.

[0197] In this embodiment, for the front-illuminated image sensor, the silicon detection layer is below the metal wire layer, and the optical filter layer 1 is directly integrated onto the metal wire layer.

[0198] In this embodiment, the difference between the back-illuminated image sensor and the front-illuminated image sensor is that the silicon detection layer is above the metal wire layer, and the optical filter layer 1 is directly integrated onto the silicon detection layer.

[0199] It should be noted that for the back-illuminated image sensor, the silicon detection layer is above the metal wire layer, which can reduce the influence of the metal wire layer on the incident light, thereby improving the quantum efficiency of the device.

[0200] As can be seen from the above content, in this embodiment, the optical filter layer is used as the input layer of the artificial neural network, the image sensor is used as the linear layer of the artificial neural network, and the filtering effect of the optical filter layer on the incident light entering the optical filter layer is used as the connection weight from the input layer to the linear layer. The optical filter layer and the image sensor in the optical artificial neural network enhanced machine vision chip provided by this embodiment implement the relevant functions of the input layer and the linear layer in the artificial neural network through hardware, so that subsequent complex signal processing and algorithm processing corresponding to the input layer and the linear layer are no longer required when using this enhanced machine vision chip for machine vision intelligent processing, which can greatly reduce the power consumption and delay during the processing of the artificial neural network. In addition, since this embodiment simultaneously utilizes the image information, spectral information, incident light angle information, and incident light phase information at different points in the space of the target object, the machine vision intelligent processing of the target object can be more accurately realized.

[0201] Based on the same inventive concept, another embodiment of the present invention provides a machine vision device, including: the optical artificial neural network enhanced machine vision chip as described in the above embodiment.

[0202] Since the machine vision device provided in this embodiment includes the optical artificial neural network enhanced machine vision chip described in the above embodiment, therefore, the machine vision device provided in this embodiment has all the beneficial effects of the optical artificial neural network enhanced machine vision chip described in the above embodiment. Since the above embodiment has described this in detail, this embodiment will not be repeated here.

[0203] Based on the same inventive concept, another embodiment of the present invention provides a preparation method of the optical artificial neural network enhanced machine vision chip as described in the above embodiment, as Figure 13 shown, specifically including the following steps:

[0204] Step 1310: Prepare an optical filter layer including an optical modulation structure on the surface of the photosensitive area of the image sensor;

[0205] Step 1320: Generate a processor with the functions of performing fully connected processing and non-linear activation processing on signals;

[0206] Step 1330: Connect the image sensor and the processor;

[0207] Wherein, the optical filter layer is used to perform different spectral modulations on the incident light entering different position points of the optical modulation structure through the optical modulation structure, so as to obtain the incident light carrying information corresponding to different position points on the surface of the photosensitive area; the incident light carrying information includes at least one of light intensity distribution information, spectral information, incident light angle information, and incident light phase information;

[0208] The image sensor is used to convert the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer into electrical signals corresponding to different position points, and send the electrical signals corresponding to different position points to the processor; the processor is used to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result; the electrical signal is an image signal modulated by the optical filter layer, and the incident light includes reflected light, transmitted light, and / or radiation light of a target object in a machine vision scene.

[0209] In this embodiment, it further includes: the training process of the optical artificial neural network enhanced machine vision chip, specifically including:

[0210] Using the input training samples and output training samples corresponding to the machine vision intelligent processing task, training the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and different non-linear activation parameters to obtain an optical modulation structure, image sensor, and processor that meet the training convergence conditions, and using the optical modulation structure, image sensor, and processor that meet the training convergence conditions as the trained optical modulation structure, image sensor, and processor.

[0211] Preparing an optical filter layer including an optical modulation structure on the surface of the photosensitive area of the image sensor, including:

[0212] Growing one or more layers of preset materials on the surface of the photosensitive area of the image sensor;

[0213] Performing dry etching on the one or more layers of preset materials with an optical modulation structure pattern to obtain an optical filter layer including an optical modulation structure;

[0214] Or performing imprint transfer on the one or more layers of preset materials to obtain an optical filter layer including an optical modulation structure;

[0215] Or obtaining an optical filter layer including an optical modulation structure by applying external dynamic regulation to the one or more layers of preset materials;

[0216] Or performing zone printing on the one or more layers of preset materials to obtain an optical filter layer including an optical modulation structure;

[0217] Or performing zone material growth on the one or more layers of preset materials to obtain an optical filter layer including an optical modulation structure;

[0218] Or performing quantum dot transfer on the one or more layers of preset materials to obtain an optical filter layer including an optical modulation structure.

[0219] When the optical artificial neural network enhanced machine vision chip is used for the machine vision intelligent processing task of a target object, the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and non-linear activation parameters is trained by using the input training samples and output training samples corresponding to the machine vision intelligent processing task, so as to obtain an optical modulation structure, an image sensor, and a processor that meet the training convergence conditions.

[0220] In this embodiment, it should be noted that, as Figure 1 shown, the optical filter layer 1 can be prepared by directly growing one or more layers of dielectric materials on the image sensor 2, then performing etching, depositing metal materials before removing the sacrificial layer used for etching, and finally removing the sacrificial layer. By designing the size parameters of the optical modulation structure, each unit can have different modulation effects on lights with different wavelengths within the target range, and this modulation effect is insensitive to the incident angle and polarization. Each unit in the optical filter layer 1 corresponds to one or more pixels on the image sensor 2. 1 is directly prepared on 2.

[0221] In this embodiment, it should be noted that assuming the image sensor 2 is a back-illuminated structure, the optical filter layer 1 can be directly etched on the silicon detector layer of the back-illuminated image sensor, and then metal is deposited for preparation.

[0222] In addition, it should be noted that the optical modulation structure on the optical filter layer can be obtained by dry etching the optical modulation structure pattern on one or more layers of preset materials. Dry etching is to directly remove the unnecessary parts of one or more layers of preset materials on the surface of the photosensitive area of the image sensor, so as to obtain an optical filter layer containing the optical modulation structure; or by imprint transfer of one or more layers of preset materials. Imprint transfer is to prepare the required structure by etching on other substrates, and then transfer the structure to the photosensitive area of the image sensor through materials such as PDMS, so as to obtain an optical filter layer containing the optical modulation structure; or by applying external dynamic regulation to one or more layers of preset materials. External dynamic regulation is to use active materials, and then apply electrodes to regulate the optical modulation characteristics of the corresponding area by changing the voltage, so as to obtain an optical filter layer containing the optical modulation structure; or by zone printing of one or more layers of preset materials. Zone printing is to use the printing technology in zones to obtain an optical filter layer containing the optical modulation structure; or by zone material growth of one or more layers of preset materials to obtain an optical filter layer containing the optical modulation structure; or by quantum dot transfer of one or more layers of preset materials to obtain an optical filter layer containing the optical modulation structure.

[0223] In addition, it should be noted that since the preparation method provided in this embodiment is the preparation method of the optical artificial neural network enhanced machine vision chip in the above embodiment, for the detailed content in terms of some principles and structures, etc., reference can be made to the introduction in the above embodiment, and this embodiment will not repeat it here.

[0224] Based on this, the optical artificial neural network enhanced machine vision chip provided by the embodiment of the present invention realizes a brand-new intelligent chip capable of realizing the functions of an artificial neural network. In this intelligent chip, the optical filter layer serves as the input layer of the artificial neural network, and the image sensor serves as the linear layer of the artificial neural network. At the same time, the filtering effect of the optical filter layer on the incident light entering the optical filter layer corresponds to the connection weight from the input layer to the linear layer. That is, the optical filter layer and the image sensor in this intelligent chip realize the related functions of the input layer and the linear layer in the artificial neural network. That is, the embodiment of the present invention separates the input layer and the linear layer in the artificial neural network implemented by software in the prior art and realizes these two-layer structures in the artificial neural network in a hardware manner. As a result, when using this intelligent chip for artificial neural network intelligent processing subsequently, there is no need to perform complex signal processing and algorithm processing corresponding to the input layer and the linear layer, and only the processor in the intelligent chip needs to perform related processing of full connection and non-linear activation with the electrical signal. In this way, the power consumption and delay during artificial neural network processing can be significantly reduced. Moreover, the embodiment of the present invention can also utilize the image information, spectral information, angle of the incident light, and phase information of the target object simultaneously, that is, the incident light at different points in space of the target object carries information. It can be seen that since the incident light at different points in space of the target object carries information covering the image, composition, shape, three-dimensional depth, structure, etc. of the target object, when performing recognition processing based on the information carried by the incident light at different points in space of the target object, multi-dimensional information such as the image, composition, shape, three-dimensional depth, and structure of the target object can be covered, thereby improving the accuracy of fingerprint recognition. It can be seen that the optical artificial neural network fingerprint recognition chip provided by the embodiment of the present invention can not only achieve the effects of low power consumption and low delay, but also improve the accuracy of target object recognition, and thus can be preferably applied in the field of machine vision intelligent processing.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optical artificial neural network enhanced machine vision chip, characterized in that, For machine vision intelligent processing tasks, including: an optical filter layer, an image sensor, and a processor; the optical filter layer corresponds to the input layer of the artificial neural network and the connection weights from the input layer to the linear layer, and the image sensor corresponds to the linear layer of the artificial neural network; the processor corresponds to the non-linear layer and the output layer of the artificial neural network; The optical filter layer is disposed on the surface of the photosensitive area of the image sensor. The optical filter layer includes an optical modulation structure. The optical filter layer is configured to perform different spectral modulations on the incident light entering different position points of the optical modulation structure respectively through the optical modulation structure, so as to obtain the information carried by the incident light corresponding to different position points on the surface of the photosensitive area; the incident light includes the reflected light, transmitted light, and / or radiation light of the target object in the machine vision scenario; The image sensor is configured to convert the information carried by the incident light corresponding to different position points modulated by the optical filter layer into electrical signals corresponding to different position points, and send the electrical signals corresponding to different position points to the processor; the electrical signal is an image signal modulated by the optical filter layer; The processor is configured to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result; Wherein, the machine vision intelligent processing tasks include the recognition, and / or measurement, and / or control of the target object in the machine vision scenario; the machine vision intelligent processing result includes the recognition result, and / or measurement result, and / or control result of the target object in the machine vision scenario.

2. The optical artificial neural network enhanced machine vision chip according to claim 1, characterized in that, The information carried by the incident light includes at least one of light intensity distribution information, spectral information, the angle information of the incident light, and the phase information of the incident light.

3. The optical artificial neural network enhanced machine vision chip according to claim 1, wherein The optical artificial neural network enhanced machine vision chip includes a trained optical modulation structure, an image sensor, and a processor; The trained optical modulation structure, image sensor, and processor refer to the optical modulation structure, image sensor, and processor that meet the training convergence conditions obtained by training the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and different non-linear activation parameters by using the input training samples and output training samples corresponding to the machine vision intelligent processing tasks; The input training samples include the incident light reflected, transmitted, and / or radiated by the target object with different machine vision scenarios; the output training samples include the recognition result, and / or measurement result, and / or control result of the target object.

4. The optical artificial neural network enhanced machine vision chip according to claim 3, wherein, When training the optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and non-linear activation parameters, the different optical modulation structures are designed and implemented by means of computer optical simulation design.

5. The optical artificial neural network enhanced machine vision chip according to claim 3, characterized in that, The machine vision scenarios include autonomous driving, robotic surgical navigation, express package sorting, robotic measurement, and beer production line detection.

6. The optical artificial neural network enhanced machine vision chip according to claim 5, characterized in that, When the machine vision scenario is autonomous driving, the input training samples include incident light reflected, transmitted, and / or radiated by roads with different road conditions and front obstacles; the output training samples include road condition recognition results and front obstacle recognition results. When the machine vision scenario is robotic surgical navigation, the input training samples include incident light reflected, transmitted, and / or radiated by surgical operation objects and surgical assistance tools; the output training samples include relative position recognition results of the corresponding surgical operation objects and surgical assistance tools. When the machine vision scenario is express package sorting, the input training samples include incident light reflected, transmitted, and / or radiated by different express packages; the output training samples include recognition results of the volume, weight, and recipient information of the express packages. When the machine vision scenario is robotic measurement, the input training samples include incident light reflected, transmitted, and / or radiated by different measurement objects; the output training samples include recognition results of the position, shape, and size of the measurement objects. When the machine vision scenario is beer production line detection, the input training samples include incident light reflected, transmitted, and / or radiated by different production lines; the output training samples include the working states of the corresponding beer production lines.

7. The optical artificial neural network enhanced machine vision chip according to any one of claims 1 to 6, characterized in that, The optical modulation structure in the optical filter layer includes regular structures and / or irregular structures; and / or, the optical modulation structure in the optical filter layer includes discrete structures and / or continuous structures.

8. The optical artificial neural network enhanced machine vision chip according to any one of claims 1 to 6, characterized in that, The optical modulation structure in the optical filter layer includes a unit array composed of multiple micro-nano units, and each micro-nano unit corresponds to one or more pixel points on the image sensor; the structures of the respective micro-nano units are the same or different.

9. The optical artificial neural network enhanced machine vision chip according to claim 8, wherein The micro-nano unit includes regular structures and / or irregular structures; and / or, the micro-nano unit includes discrete structures and / or continuous structures.

10. The optical artificial neural network enhanced machine vision chip according to claim 8, wherein The micro-nano unit includes multiple groups of micro-nano structure arrays, and the structures of the respective groups of micro-nano structure arrays are the same or different.

11. The optical artificial neural network enhanced machine vision chip according to claim 10, wherein Each group of micro-nano structure arrays has the function of broadband filtering or narrowband filtering.

12. The optical artificial neural network enhanced machine vision chip according to claim 10, wherein Each group of micro-nano structure arrays is a periodic structure array or a non-periodic structure array.

13. The optical artificial neural network enhanced machine vision chip according to claim 10, characterized in that, One or more groups of the multiple groups of micro-nano structure arrays included in the micro-nano unit are empty structures.

14. The optical artificial neural network enhanced machine vision chip according to claim 10, wherein, The micro-nano unit has polarization-independent characteristics.

15. The optical artificial neural network enhanced machine vision chip according to claim 14, characterized in that, The micro-nano unit has four-fold rotational symmetry.

16. The optical artificial neural network enhanced machine vision chip according to claim 1, characterized in that The optical filter layer is composed of one or more filter layers. The filter layer is prepared from one or more of semiconductor materials, metal materials, liquid crystals, quantum dot materials, perovskite materials; and / or, the filter layer is a filter layer prepared from one or more of photonic crystals, metasurfaces, random structures, nanostructures, metal surface plasmon polariton SPP nanostructures, tunable Fabry-Perot resonators.

17. The optical artificial neural network enhanced machine vision chip according to claim 16, characterized in that, The semiconductor materials include one or more of silicon, silicon oxide, silicon nitride, titanium oxide, composite materials mixed in a preset ratio, and direct bandgap compound semiconductor materials; and / or, the nanostructures include one or more of two-dimensional nanodot materials, two-dimensional nanocolumn materials, and two-dimensional nanowire materials.

18. The optical artificial neural network enhanced machine vision chip according to claim 1, characterized in that, The thickness of the optical filter layer is 0.1λ to 10λ, where λ represents the central wavelength of the incident light.

19. A machine vision device, characterized in that, Comprising: A control mechanism and an optical artificial neural network enhanced machine vision chip as described in any one of claims 1 to 18; Wherein, the control mechanism is connected to the optical artificial neural network enhanced machine vision chip, and the control mechanism is used to perform corresponding control according to the machine vision intelligent processing result of the artificial neural network enhanced machine vision chip.

20. A method for preparing an optical artificial neural network enhanced machine vision chip according to any one of claims 1 to 18, characterized in that, Comprising: Preparing an optical filter layer containing an optical modulation structure on the surface of the photosensitive area of the image sensor; Generating a processor with the function of performing fully connected processing and non-linear activation processing on signals; Connecting the image sensor and the processor; Wherein, the optical filter layer is used to perform different spectral modulations on the incident light entering different position points of the optical modulation structure through the optical modulation structure, so as to obtain the information carried by the incident light corresponding to different position points on the surface of the photosensitive area; the information carried by the incident light includes at least one of light intensity distribution information, spectral information, the angle information of the incident light, and the phase information of the incident light; The image sensor is used to convert the information carried by the incident light corresponding to different position points after being modulated by the optical filter layer into electrical signals corresponding to different position points, and send the electrical signals corresponding to different position points to the processor; the processor is used to perform fully connected processing and non-linear activation processing on the electrical signals corresponding to different position points to obtain a machine vision intelligent processing result; the electrical signal is an image signal modulated by the optical filter layer, and the incident light includes reflected light, transmitted light, and / or radiation light of a target object in a machine vision scenario.

21. The method for preparing the optical artificial neural network enhanced machine vision chip according to claim 20, wherein Preparing an optical filter layer containing an optical modulation structure on the surface of the photosensitive area of the image sensor, including: Growing one or more layers of a preset material on the surface of the photosensitive area of the image sensor; Etching a pattern of the optical modulation structure on the one or more layers of the preset material to obtain an optical filter layer containing the optical modulation structure; Or performing imprint transfer on the one or more layers of the preset material to obtain an optical filter layer containing the optical modulation structure; Or obtaining an optical filter layer containing the optical modulation structure by applying external dynamic modulation to the one or more layers of the preset material; Or performing zone printing on the one or more layers of the preset material to obtain an optical filter layer containing the optical modulation structure; Or performing zone growth on the one or more layers of the preset material to obtain an optical filter layer containing the optical modulation structure; Or performing quantum dot transfer on the one or more layers of the preset material to obtain an optical filter layer containing the optical modulation structure.

22. The method for preparing the optical artificial neural network enhanced machine vision chip according to claim 20, wherein, Further comprising: The training process of the optical artificial neural network enhanced machine vision chip specifically includes: Using the input training samples and output training samples corresponding to the machine vision intelligent processing task, train an optical artificial neural network enhanced machine vision chip including different optical modulation structures, image sensors, and processors with different fully connected parameters and different non-linear activation parameters to obtain an optical modulation structure, an image sensor, and a processor that meet the training convergence conditions, and use the optical modulation structure, image sensor, and processor that meet the training convergence conditions as the trained optical modulation structure, image sensor, and processor.

Images