High-speed multispectral parallel bionic visual imaging method and system and medium

Abstract

The invention provides a high-speed multispectral parallel bionic visual imaging method and system and a medium, and the method comprises the steps: simulating a mantis shrimp compound eye structure based on a microlens array, and receiving incident light in multiple directions; incident light with different central wavelengths is selectively transmitted in different channels, filtered light signals are received based on a photosensitive array, and a multi-channel image is generated; performing time sequence sampling based on the phase encoder to obtain a multi-frame sampling result on a time domain, and capturing motion information of the target based on the multi-frame sampling result; performing multi-channel image fusion based on a neural mimicry neural network, performing motion vector estimation based on operation information of a target, and outputting a fused image frame and a motion vector diagram; by simulating a mantis shrimp compound eye structure, parallel sensing of multi-channel spectrums is achieved, multi-dimensional sensing data are encoded and fused, and the method is widely applied to the fields of automatic driving, unmanned aerial vehicle navigation, complex environment sensing and the like and has the advantages of high-speed response, weak light adaptation and high-precision recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual imaging, in particular to a high-speed multi-spectral parallel bionic visual imaging method, system and medium. BACKGROUND

[0002] At present, in the field of unmanned driving and high dynamic target recognition, traditional integrated imaging equipment often adopts RGB single-band imaging or fusion type infrared + visible light dual-band technology. Such technology has the following limitations:

[0003] Spectral limitation: RGB images can only identify objects in the visible light segment, and the recognition rate is low when facing complex lighting or camouflage targets.

[0004] Response speed problem: Most image sensors use rolling shutters and single-frame exposure, which are difficult to capture the transient changes of high-speed moving targets.

[0005] Simple structure: Traditional cameras lack spatial optical control and field of view adaptive ability, and cannot simulate complex biological vision. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a high-speed multi-spectral parallel bionic visual imaging method, system and medium, which simulates the structure of mantis shrimp compound eyes, realizes parallel perception of multi-channel spectrum, encodes and fuses multi-dimensional perception data, and is widely used in the fields of autonomous driving, unmanned aerial vehicle navigation, complex environment perception, etc. It has the advantages of high-speed response, weak light adaptation and high-precision identification.

[0007] The embodiments of the present application also provide a high-speed multi-spectral parallel bionic visual imaging method, which comprises:

[0008] Simulate the structure of mantis shrimp compound eyes based on a microlens array to receive incident light from multiple directions;

[0009] Based on a multi-spectral filter array, selectively transmit incident light of different center wavelengths by channel to obtain filtered optical signals;

[0010] Based on a photosensitive array, receive the filtered optical signals, convert the filtered optical signals into multi-channel electrical signals, and generate multi-channel images;

[0011] Based on a phase encoder, perform time sequence sampling to obtain multi-frame sampling results in the time domain, and capture motion information of a target based on the multi-frame sampling results;

[0012] Based on a neural quasi-state neural network, perform multi-channel image fusion, estimate a motion vector based on the motion information of the target, and output a fused image frame and a motion vector map.

[0013] Optionally, in the high-speed multispectral parallel biomimetic visual imaging method, the micro-lens array is arranged at the front end of the incident light, and a biomimetic hexagonal compact arrangement is adopted. The micro-lens array includes a plurality of unit lenses, each unit lens has a diameter of 40-60 μm and a focal length of 50-150 μm.

[0014] The micro-lens array is arranged at the front end of the incident light, and a biomimetic hexagonal compact arrangement is adopted. The micro-lens array includes a plurality of unit lenses, each unit lens has a diameter of 40-60 μm and a focal length of 50-150 μm.

[0015] The micro-lens array is arranged at the front end of the incident light, and a biomimetic hexagonal compact arrangement is adopted. The micro-lens array includes a plurality of unit lenses, each unit lens has a diameter of 40-60 μm and a focal length of 50-150 μm.

[0016] The design formula is as follows:

[0017] The micro-lens focal length calculation formula is:

[0018]

[0019] Wherein: f: micro-lens focal length; d: lens diameter; n: material refractive index;

[0020] The field of view expansion angle is:

[0021]

[0022] Wherein R is the radius of the spherical crown.

[0023] Optionally, in the high-speed multispectral parallel biomimetic visual imaging method, the micro-lens array is arranged at the front end of the incident light, and a biomimetic hexagonal compact arrangement is adopted. The micro-lens array includes a plurality of unit lenses, each unit lens has a diameter of 40-60 μm and a focal length of 50-150 μm.

[0024] A plurality of surface array filters are arranged in a two-dimensional grid, and each filter corresponds to a pixel or a pixel group of the detector;

[0025] Based on the spectral coverage uniformity principle, the center wavelength is distributed to obtain a visible light band and a near-infrared band, the visible light band is 400-700 nm, and the near-infrared band is 700-1100 nm;

[0026] Based on the visible light band, the incident light is screened to obtain a first screening result, and based on the near-infrared band, the incident light is screened to obtain a second screening result;

[0027] Based on the first screening result and the second screening structure, the filtered light signal is obtained.

[0028] Optionally, in the high-speed multispectral parallel biomimetic visual imaging method, the micro-lens array is arranged at the front end of the incident light, and a biomimetic hexagonal compact arrangement is adopted. The micro-lens array includes a plurality of unit lenses, each unit lens has a diameter of 40-60 μm and a focal length of 50-150 μm.

[0029] The pixels based on the light sensing array are aligned with the multi-spectral filter, and the parallel reception of different waveband light signals is implemented;

[0030] After the photodiode or phototransistor absorbs photons of a specific waveband, photo-generated carriers are generated, and a photocurrent or charge accumulation proportional to the light intensity is formed;

[0031] The photocurrents of multiple channels are simultaneously converted into measurable voltage signals and amplified to obtain multi-channel electrical signals;

[0032] The multi-channel electrical signals are analog-to-digital converted to obtain digital signals, and multi-channel images are formed based on the digital signals.

[0033] Optionally, in the high-speed multi-spectral parallel biomimetic visual imaging method described in the embodiments of the present application, time-domain multi-frame sampling results are obtained based on a phase encoder for time sequence sampling, and the motion information of the target is captured based on the multi-frame sampling results, specifically including:

[0034] A micro time difference is introduced based on the phase encoder to form multi-frame sampling in the time domain, pseudo-synchronous sampling is achieved through time offset micro-mirror control of different channels, a micro time difference sampling structure of adjacent channels is formed, and the formula is as follows:

[0035] t i =t0+Δt i

[0036] t i represents the sampling time point of the i-th channel.

[0037] t0 represents the reference starting sampling time of all channels, i.e., the sampling time of the first channel;

[0038] The delay time Δt i is in the order of nanoseconds, which is used to calculate the local optical flow field, and the calculation formula is as follows:

[0039]

[0040] represents the motion vector at the image coordinates (x, y), i.e., the instantaneous motion direction and speed of the target at that point;

[0041] δ represents a small time interval, which is in seconds, milliseconds or nanoseconds;

[0042] I(x,y,t) represents the pixel intensity value or brightness value at the image coordinates (x, y) at time t.

[0043] The final motion direction is obtained in combination with the direction gradient, and the motion information is generated;

[0044] Wherein, the direction gradient calculation formula is as follows:

[0045]

[0046] Indicates an image gradient vector, pointing to the direction of the fastest brightness change;

[0047] Indicates a unit gradient direction, used to indicate the edge orientation;

[0048] The final motion direction calculation formula is as follows:

[0049]

[0050] Indicates the final motion direction vector at the image coordinates (x, y).

[0051] Optionally, in the high-speed multispectral parallel biomimetic visual imaging method described in the embodiments of the present application, multi-channel image fusion is performed based on a neurodynamical neural network, motion vector estimation is performed based on the running information of the target, and a fused image frame and a motion vector map are output, and specifically includes:

[0052] Based on a shared encoder, feature extraction is performed on each channel image to obtain encoded features z i , and the shared encoder formula is as follows:

[0053] E i (I i )=σ(W e ·I i +b e )E i (I i ) represents the encoded features of the i th channel image (i.e., the output after feature extraction);

[0054] σ represents an activation function (such as Sigmoid, ReLU, etc.), which is used to introduce a nonlinear transformation;

[0055] W e is a weight matrix (Weight Matrix) of the shared encoder, which is used for linear transformation of the input image I i ;

[0056] I i represents the input image of the i th channel (or the i th waveband data of the multispectral image);

[0057] b e is a bias (Bias) of the shared encoder, which is used to adjust the output after linear transformation.

[0058] The encoding features of each channel are spliced in a high-dimensional space based on a neural quasi-state neural network, and the calculation formula is as follows:

[0059] Z = Concat(z1, z2, …, z 16 ) Concat(z1, z2, …, z 16 ) represents the feature vectors extracted by the shared encoder for images of 16 different spectral channels, and concat is a "concatenation" operation, which means connecting multiple feature vectors in sequence into a higher-dimensional vector.

[0060] Z represents the high-dimensional feature space vector obtained after splicing, which contains the fusion information of all channels.

[0061] The space-time encoding vector is generated using the GRU structure to obtain the fused image frame.

[0062] The motion vector estimation is calculated by the following formula:

[0063]

[0064] represents the partial derivative of image intensity with respect to time t, which is used to describe the rate of change of brightness at this position with respect to time (i.e. the change of brightness caused by motion).

[0065] where I(x, y, t) is the image intensity, is the gradient.

[0066] In a second aspect, the embodiments of the present application provide a high-speed multispectral parallel biomimetic visual imaging system, which comprises a memory and a processor, the memory comprising a program of a high-speed multispectral parallel biomimetic visual imaging method, and the program of the high-speed multispectral parallel biomimetic visual imaging method is executed by the processor to implement the following steps:

[0067] Based on the microlens array simulating the mantis shrimp compound eye structure, multiple directions of incident light are received;

[0068] Based on the multispectral filter array, incident light of different center wavelengths is selected for channel-selective transmission to obtain filtered light signals;

[0069] Based on the photosensitive array, the filtered light signals are received and converted into multi-channel electrical signals to generate multi-channel images;

[0070] Based on the phase encoder, time sequence sampling is performed to obtain multiple frames of sampling results in the time domain, and the motion information of the target is captured based on the multiple frames of sampling results.

[0071] The multi-channel image fusion is performed based on a neural quasi-state neural network, the motion vector estimation is performed based on running information of the target, and a fused image frame and a motion vector diagram are output.

[0072] Optionally, in the high-speed multi-spectrum parallel bionic visual imaging system, a micro-lens array is used to simulate a mantis shrimp compound eye structure to receive incident light in multiple directions, and specifically includes the following steps.

[0073] The micro-lens array is arranged at the incident front end, and a bionic hexagonal compact arrangement is used, the micro-lens array includes a plurality of unit lenses, the diameter of each unit lens is 40-60 μm, and the focal length is 50-150 μm.

[0074] The micro-lens array is arranged in a three-dimensional curved support, and is uniformly distributed on the spherical crown surface.

[0075] The design formula is as follows.

[0076] The micro-lens focal length calculation formula is as follows.

[0077]

[0078] Wherein, f is the micro-lens focal length, d is the lens diameter, and n is the material refractive index.

[0079] The visual field expansion angle is as follows.

[0080]

[0081] Wherein, R is the spherical crown radius.

[0082] Optionally, in the high-speed multi-spectrum parallel bionic visual imaging system, a multi-spectrum filter array is used to perform channel-selective transmission on incident light of different center wavelengths to obtain filtered light signals, and specifically includes the following steps.

[0083] A plurality of surface array filters are arranged in a two-dimensional grid, and each filter corresponds to a pixel or a pixel group of a detector.

[0084] Based on the spectral coverage uniformity principle, the center wavelengths are distributed to obtain a visible light band and a near-infrared band, the visible light band is 400-700 nm, and the near-infrared band is 700-1100 nm.

[0085] Based on the visible light band, the incident light is screened to obtain a first screening result, and based on the near-infrared band, the incident light is screened to obtain a second screening result.

[0086] Based on the first screening result and the second screening structure, the filtered light signal is obtained.

[0087] In a third aspect, the embodiments of the present application further provide a computer readable storage medium, wherein a high-speed multi-spectrum parallel biomimetic visual imaging method program is stored in the computer readable storage medium, and the high-speed multi-spectrum parallel biomimetic visual imaging method program is executed by a processor to implement the steps of the high-speed multi-spectrum parallel biomimetic visual imaging method according to any one of the above aspects.

[0088] As can be seen from the above, the high-speed multi-spectrum parallel biomimetic visual imaging method, system and medium provided by the embodiments of the present application simulate the mantis shrimp compound eye structure based on a microlens array to receive incident light in multiple directions; perform channel-selective transmission on incident light of different center wavelengths to obtain filtered light signals; receive the filtered light signals based on a photosensitive array, convert the filtered light signals into multi-channel electrical signals, and generate multi-channel images; perform time sequence sampling based on a phase encoder to obtain multiple frames of sampling results in the time domain, and capture motion information of a target based on the multiple frames of sampling results; perform multi-channel image fusion based on a neural quasi-state neural network, perform motion vector estimation based on the motion information of the target, and output a fused image frame and a motion vector map; by simulating the mantis shrimp compound eye structure, parallel perception of multi-channel spectrum is realized, motion information is synchronously collected, multi-dimensional perception data is encoded and fused, and the method is widely applied in the fields of automatic driving, unmanned aerial vehicle navigation, complex environment perception and the like, and has the advantages of high-speed response, weak light adaptation and high-precision identification. BRIEF DESCRIPTION OF DRAWINGS

[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and other related drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0090] Figure 1 The flowchart of the high-speed multi-spectrum parallel biomimetic visual imaging method provided by the embodiments of the present application;

[0091] Figure 2 The motion vector estimation flowchart of the high-speed multi-spectrum parallel biomimetic visual imaging method provided by the embodiments of the present application;

[0092] Figure 3 The overall structure schematic diagram of the high-speed multi-spectrum parallel biomimetic visual imaging system provided by the embodiments of the present application;

[0093] Figure 4 The single microlens and light filtering path diagram of the high-speed multi-spectrum parallel biomimetic visual imaging system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0094] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0095] It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0096] Please refer to Figure 1 , Figure 1 is a flowchart of a high-speed multi-spectral parallel bionic visual imaging method in some embodiments of the present application. The high-speed multi-spectral parallel bionic visual imaging method is used in a terminal device, and the high-speed multi-spectral parallel bionic visual imaging method comprises the following steps:

[0097] S101, simulating a mantis shrimp compound eye structure based on a microlens array to receive incident light in multiple directions;

[0098] S102, performing channel-selective transmission of incident light of different center wavelengths based on a multi-spectral filter array to obtain filtered light signals;

[0099] S103, receiving the filtered light signals based on a photosensitive array, converting the filtered light signals into multi-channel electrical signals, and generating multi-channel images;

[0100] S104, performing time sequence sampling based on a phase encoder to obtain multi-frame sampling results in the time domain, and capturing motion information of a target based on the multi-frame sampling results;

[0101] S105, performing multi-channel image fusion based on a neural plasticity neural network, performing motion vector estimation based on the running information of the target, and outputting a fused image frame and a motion vector map.

[0102] It should be noted that the microlens array is used to collect images of a target scene from multiple directions; the light spectrum is selectively transmitted by the filter; the photosensitive array collects light intensity of each channel; the phase encoder realizes multi-time sampling and extracts target motion information; the neuromorphic neural network performs high-dimensional image fusion and motion vector estimation; and a fused image frame and motion prediction are output.

[0103] The phase encoder is a MEMS micro-mirror structure, which can adjust the incident angle and sampling time delay to realize micro-time displacement sampling.

[0104] The application realizes high-frame-rate multi-channel synchronous imaging (>1000fps); completely simulates the parallel visual nervous system of mantis shrimp; is suitable for dynamic obstacle recognition scenes such as high-speed flying unmanned aerial vehicles and autonomous driving vehicles; improves recognition confidence, is particularly suitable for low-illumination, complex background or dynamic camouflage recognition, significantly reduces the dependence on AI algorithm power of the back end, and realizes front-end recognition.

[0105] According to the embodiment of the application, the microlens array simulates the compound eye structure of mantis shrimp to receive incident light from multiple directions, specifically including:

[0106] The microlens array is arranged in front of the incident end, and a bionic hexagonal compact arrangement is adopted; the microlens array includes a plurality of unit lenses, each unit lens has a diameter of 40-60 μm and a focal length of 50-150 μm;

[0107] The microlens array is arranged in a three-dimensional curved support, and is uniformly distributed on the surface of a spherical crown,

[0108] The design formula is as follows:

[0109] The microlens focal length calculation formula is:

[0110]

[0111] Wherein: f: microlens focal length; d: lens diameter; n: material refractive index;

[0112] The field of view expansion angle is:

[0113]

[0114] Wherein R is the radius of the spherical crown.

[0115] According to the embodiment of the application, the multi-spectral filter array selectively transmits incident light of different center wavelengths by channel to obtain filtered light signals, specifically including:

[0116] A plurality of surface array filters are arranged in a two-dimensional grid, and each filter corresponds to a pixel or a pixel group of a detector;

[0117] The center wavelength is distributed based on the principle of uniform spectrum coverage, and a visible light band and a near-infrared band are obtained, the visible light band is 400-700 nm, and the near-infrared band is 700-1100 nm;

[0118] The incident light is screened based on the visible light band to obtain a first screening result, and the incident light is screened based on the near-infrared band to obtain a second screening result;

[0119] Based on the first screening result and the second screening structure, a filtered optical signal is obtained.

[0120] It should be noted that the filter includes 16 different center wavelength filters, covering a band of 400-900 nm with a bandwidth of ±10 nm.

[0121] Spectral filtering selection model

[0122] Each microlens corresponds to a channel, and the light filtering characteristic function is represented as:

[0123]

[0124] Wherein:

[0125] λ i : the center wavelength of the i th channel;

[0126] Δλ: the half value of the bandwidth.

[0127] The total imaging information is expressed as:

[0128]

[0129] Where R i is the response intensity of the i th channel.

[0130] According to the embodiment of the application, the filtered optical signal is converted into a multi-channel electrical signal based on the photosensitive array receiving the filtered optical signal, and a multi-channel image is generated, specifically including:

[0131] The pixels of the photosensitive array are aligned with the multi-spectral filter, and the parallel reception of different waveband optical signals is performed;

[0132] After the photodiode or phototransistor absorbs photons of a specific waveband, photo-generated carriers are generated, forming a photoelectric current or charge accumulation proportional to the light intensity;

[0133] Based on the multi-channel, the photoelectric current is converted into a measurable voltage signal and amplified to obtain a multi-channel electrical signal;

[0134] The multi-channel electrical signal is analog-to-digital converted to obtain a digital signal, and a multi-channel image is formed based on the digital signal.

[0135] According to an embodiment of the present invention, temporal sampling is performed based on a phase encoder to obtain multi-frame sampling results in the time domain, and motion information of the target is captured based on the multi-frame sampling results, specifically including:

[0136] Based on the phase encoder, a micro-time difference is introduced to form multi-frame sampling in the time domain. Different channels achieve pseudo-synchronous sampling through time-off micromirrors, forming a micro-time difference sampling structure for adjacent channels, as shown in the following formula:

[0137] t i =t0+Δt i

[0138] Delay time Δt i The calculation is performed at the nanosecond level to calculate the local optical flow field, and the formula is as follows:

[0139]

[0140] By combining the directional gradient, the final motion direction is obtained, and motion information is generated;

[0141] The formula for calculating the directional gradient is as follows:

[0142]

[0143] The formula for calculating the final direction of motion is as follows:

[0144]

[0145] like Figure 2 As shown, according to an embodiment of the present invention, multi-channel image fusion is performed based on a neuromorphic neural network, motion vector estimation is performed based on the target's running information, and fused image frames and motion vector maps are output, specifically including:

[0146] Feature extraction is performed on each channel image based on a shared encoder to obtain the encoded features z. i The formula for the shared encoder is as follows:

[0147] E i (I i )=σ(W e ·I i +b e )

[0148] The high-dimensional space is constructed by concatenating the encoded features of each channel based on the neuromorphic neural network, and the calculation formula is as follows;

[0149] Z = Concat(z1, z2, ..., z 16 )

[0150] The spatial-time encoding vector is generated using a GRU structure, and a fused image frame is obtained.

[0151] The motion vector estimation is calculated by the following formula:

[0152]

[0153] where I(x,y,t) is the image intensity, is the gradient.

[0154] It should be noted that the multispectral image fusion method

[0155] For each channel image I i (x,y), pixel-level superposition is performed to form a high-dimensional information vector:

[0156]

[0157] After fusion, principal component analysis (PCA) or sparse expression method is used to reduce the dimension to obtain a perception vector:

[0158]

[0159] where W is a dimension reduction matrix obtained by eigenvalue decomposition of a sample covariance matrix.

[0160] According to the embodiments of the present application, the actual application scenarios are as follows:

[0161] 1. UAV obstacle avoidance:

[0162] In a high dynamic flight environment, the device is installed on the nose of the UAV, and through the super-wide angle + high frame rate multispectral compound eye structure, the front obstacles are comprehensively identified in color + material + motion direction.

[0163] The system can achieve: <2ms response time; 95% obstacle recognition accuracy; weak light recognition ability as low as 0.01 lux; automatic prediction of target trajectory and guidance of path adjustment.

[0164] 2. Autonomous driving system:

[0165] Installed inside the front windshield of an autonomous vehicle, the system can perform:

[0166] Vehicle / pedestrian recognition under strong light during the day and weak light at night;

[0167] Determine whether it is a "disguised" static object (such as a speed bump disguised by painting);

[0168] Multispectral analysis of target surface material reflectivity, estimation of physical information such as friction coefficient.

[0169] For example Figures 3-4In the second aspect, the embodiment of the present application provides a high-speed multi-spectrum parallel bionic visual imaging system, which comprises a memory and a processor, the memory comprising a program of a high-speed multi-spectrum parallel bionic visual imaging method, and the program of the high-speed multi-spectrum parallel bionic visual imaging method is executed by the processor to realize the following steps:

[0170] simulate the mantis shrimp compound eye structure based on a microlens array to receive incident light from multiple directions;

[0171] selectively transmit incident light of different center wavelengths based on a multi-spectrum filter array to obtain filtered light signals;

[0172] receive the filtered light signals based on a photosensitive array, convert the filtered light signals into multi-channel electrical signals, and generate multi-channel images;

[0173] perform timing sampling based on a phase encoder to obtain multi-frame sampling results in the time domain, and capture motion information of a target based on the multi-frame sampling results;

[0174] perform multi-channel image fusion based on a neuromorphic neural network, estimate a motion vector based on the motion information of the target, and output a fused image frame and a motion vector map.

[0175] It should be noted that the system comprises the following modules:

[0176] a microlens array module, configured to simulate the mantis shrimp compound eye structure and receive incident light from multiple directions through hexagonally compactly arranged microlens units;

[0177] a multi-spectrum filter array, corresponding to narrowband filters arranged behind the microlens units respectively, and configured to selectively transmit light of different center wavelengths;

[0178] a photosensitive array, configured to receive filtered light signals and convert the filtered light signals into multi-channel electrical signals;

[0179] a phase encoder, configured to introduce a micro time difference to form multi-frame sampling in the time domain to capture motion information;

[0180] a neuromorphic processor, configured to encode, fuse and output a fused image frame and a motion vector map.

[0181] According to the embodiment of the present application, the mantis shrimp compound eye structure is simulated based on a microlens array to receive incident light from multiple directions, and specifically comprises:

[0182] The microlens array is arranged at the front end of incidence, a bionic hexagonal compact arrangement is adopted, the microlens array comprises a plurality of unit lenses, the diameter of each unit lens is 40-60 μm, and the focal length is 50-150 μm;

[0183] The microlens array is arranged by using a three-dimensional curved support to uniformly distribute on the spherical crown surface,

[0184] The design formula is as follows:

[0185] The microlens focal length calculation formula is:

[0186]

[0187] Wherein: f: microlens focal length; d: lens diameter; n: material refractive index;

[0188] The visual field expansion angle is:

[0189]

[0190] Where R is the spherical crown radius.

[0191] According to the embodiment of the application, the incident light of different center wavelengths is selectively transmitted by a multi-spectral filter array to obtain a filtered light signal, and the embodiment specifically comprises the following steps:

[0192] A plurality of surface array filters are arranged in a two-dimensional grid, and each filter corresponds to a pixel or a pixel group of a detector;

[0193] The center wavelengths are distributed based on the spectral coverage uniformity principle to obtain a visible light band and a near-infrared band, the visible light band is 400-700nm, and the near-infrared band is 700-1100nm;

[0194] The incident light is screened based on the visible light band to obtain a first screening result, and the incident light is screened based on the near-infrared band to obtain a second screening result;

[0195] Based on the first screening result and the second screening structure, a filtered light signal is obtained.

[0196] It should be noted that the filter includes 16 different center wavelength filters, covers a band of 400-900nm, and has a bandwidth of ±10nm.

[0197] Spectral filtering selection model

[0198] Each microlens corresponds to a channel, and the filter characteristic function thereof is expressed as:

[0199]

[0200] Wherein:

[0201] λ i : the center wavelength of the i th channel;

[0202] Δλ: half value of bandwidth.

[0203] The total imaging information is expressed as:

[0204]

[0205] wherein R i is the response intensity of the i-th channel.

[0206] According to the embodiment of the present application, the filtered light signal is converted into a multi-channel electrical signal based on the photosensitive array receiving the filtered light signal, and a multi-channel image is generated, specifically including:

[0207] The pixel of the photosensitive array is aligned with the multi-spectral filter, and the parallel reception of light signals of different wavebands is performed;

[0208] After the photodiode or phototransistor absorbs photons of a specific waveband, photo-generated carriers are generated, forming a photoelectric current or charge accumulation proportional to the light intensity;

[0209] The photoelectric current is converted into a measurable voltage signal based on the multi-channel, and amplification processing is performed to obtain a multi-channel electrical signal;

[0210] The multi-channel electrical signal is analog-to-digital converted to obtain a digital signal, and a multi-channel image is formed based on the digital signal.

[0211] According to the embodiment of the present application, the phase encoder is used for time sequence sampling to obtain a plurality of frames of sampling results in the time domain, and the motion information of the target is captured based on the plurality of frames of sampling results, specifically including:

[0212] The phase encoder introduces a micro time difference to form a plurality of frames of sampling in the time domain, and different channels are pseudo-synchronously sampled through time offset micro mirror control to form a micro time difference sampling structure of adjacent channels, and the formula is as follows:

[0213] t i = t0+ Δt i

[0214] The delay time Δt i is in the order of nanoseconds, which is used to calculate the local optical flow field, and the calculation formula is as follows:

[0215]

[0216] The final motion direction is obtained in combination with the direction gradient to generate the motion information;

[0217] The direction gradient calculation formula is as follows:

[0218]

[0219] The final motion direction calculation formula is as follows:

[0220]

[0221] As Figure 2 shown, according to an embodiment of the application, multi-channel image fusion is performed based on a neural quasi-state neural network, motion vector estimation is performed based on target running information, and a fused image frame and a motion vector map are output, specifically including:

[0222] Feature extraction is performed on each channel image based on a shared encoder to obtain encoded features z i , and the shared encoder formula is as follows:

[0223] E i (I i )=σ(W e ·I i +b e )

[0224] The encoded features of each channel are spliced in a high-dimensional space based on a neural quasi-state neural network, and the calculation formula is as follows:

[0225] Z=Concat(z1,z2,…,z 16 )

[0226] A space-time encoding vector is generated using a GRU structure to obtain a fused image frame.

[0227] Motion vector estimation is calculated by the following formula:

[0228]

[0229] Where I(x,y,t) is the image intensity, is the gradient.

[0230] It should be noted that the multispectral image fusion method

[0231] For each channel image I i (x,y), pixel-level superposition is performed to form a high-dimensional information vector:

[0232]

[0233] After fusion, principal component analysis (PCA) or sparse expression method is used to reduce the dimension to obtain a perception vector:

[0234]

[0235] Where W is a dimension reduction matrix obtained by eigenvalue decomposition of a sample covariance matrix.

[0236] According to an embodiment of the application, the actual application scenarios of the application are as follows:

[0237] 1. UAV obstacle avoidance:

[0238] In a high dynamic flight environment, the device is installed on the nose of the UAV, and through the multi-spectral compound eye structure of ultra-wide angle + high frame rate, the color + material + motion direction of the front obstacle is comprehensively identified.

[0239] The system can achieve: <2ms response time; 95% obstacle recognition accuracy; low to 0.01 lux weak light recognition ability; automatic prediction of target trajectory and guidance of path adjustment.

[0240] 2. Automatic driving system:

[0241] Installed inside the front windshield of the automatic driving vehicle, the system can perform:

[0242] Vehicle / pedestrian recognition under strong light during the day and weak light at night;

[0243] Determine whether it is a "disguised" static object (such as a speed bump disguised by painting);

[0244] Multi-spectral analysis of target surface material reflectivity, estimation of friction coefficient and other physical information.

[0245] The third aspect of the application provides a computer readable storage medium, the readable storage medium includes a high-speed multi-spectral parallel biomimetic visual imaging method program, and the high-speed multi-spectral parallel biomimetic visual imaging method program is executed by a processor to realize the steps of the high-speed multi-spectral parallel biomimetic visual imaging method of any one of the above.

[0246] The application discloses a kind of high-speed multi-spectral parallel biomimetic visual imaging method, system and medium, by simulating mantis shrimp compound eye structure based on microlens array, receive multiple directions incident light;Different center wavelength incident light is selected by channel selective transmission, obtain filtered light signal;Based on photosensitive array receives the light signal after filtering, the light signal after filtering is converted into multi-channel electrical signal, generates multi-channel image;Based on phase encoder carries out time series sampling, obtains multiple frames of sampling results in time domain, based on multiple frames of sampling results capture target motion information;Based on neural pseudo-state neural network carries out multi-channel image fusion, based on the running information of target carries out motion vector estimation, output fusion image frame and motion vector diagram;By simulating mantis shrimp compound eye structure, realize the parallel perception of multi-channel spectrum, synchronously collect motion information, encode and fusion are carried out to multidimensional perception data, widely used in automatic driving, unmanned aerial vehicle navigation, complex environment perception and other fields, with high-speed response, weak light adaptation and high-precision identification.

[0247] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The above described device embodiments are merely exemplary. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0248] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place, or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0249] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional units.

[0250] Those of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a readable storage medium, and when the program is executed, the steps of the method embodiments are executed; and the foregoing storage medium includes mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic discs or optical discs, and various media that can store program codes.

[0251] Alternatively, the integrated units of the present application, if implemented in the form of software functional modules and sold or used as independent products, can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROMs, RAMs, magnetic discs or optical discs, and various media that can store program codes.

Claims

1. A high-speed multispectral parallel biomimetic vision imaging method, characterized in that, The application relates to a multi-channel image acquisition system based on a mantis shrimp compound eye structure. The application comprises the following steps: A microlens array is used to simulate a mantis shrimp compound eye structure to receive incident light from multiple directions; A multi-spectral filter array is used to selectively transmit incident light of different central wavelengths in different channels to obtain filtered light signals; A photosensitive array is used to receive the filtered light signals and convert the filtered light signals into multi-channel electrical signals to generate multi-channel images; A phase encoder is used for time sequence sampling to obtain multi-frame sampling results in the time domain, and the motion information of a target is captured based on the multi-frame sampling results; 2. The method of claim 1, wherein, A neural state neural network is used for multi-channel image fusion, and a motion vector is estimated based on the motion information of the target to output a fused image frame and a motion vector map. The application comprises the following steps: A microlens array is used to simulate a mantis shrimp compound eye structure to receive incident light from multiple directions, specifically comprising the following steps: The microlens array is arranged at the front end of the incident light, and a bionic hexagonal compact arrangement is adopted, the microlens array comprises a plurality of unit lenses, the diameter of each unit lens is 40-60 mu m, and the focal length is 50-150 mu m; The microlens array is arranged on the spherical crown surface by using a three-dimensional curved support, The design formula is as follows: The microlens focal length calculation formula is as follows: Wherein: f: microlens focal length; d: lens diameter; n: material refractive index; 3. The method of claim 2, wherein, The visual field expansion angle is as follows: Wherein R is the spherical crown radius. The application comprises the following steps: A plurality of surface array filters are arranged in a two-dimensional grid, and each filter corresponds to a pixel or a pixel group of a detector; Based on the spectral coverage uniformity principle, the central wavelengths are distributed to obtain a visible light band and a near-infrared band, the visible light band is 400-700 nm, and the near-infrared band is 700-1100 nm; 4. The method of claim 3, wherein, Based on the visible light band, the incident light is screened to obtain a first screening result, and based on the near-infrared band, the incident light is screened to obtain a second screening result; Based on the first screening result and the second screening structure, the filtered light signals are obtained. The application comprises the following steps: The pixels of the photosensitive array are aligned with the multi-spectral filters to receive different band light signals in parallel; After a photodiode or a phototransistor absorbs photons of a specific band, photo-generated carriers are generated to form a photocurrent or a charge accumulation amount proportional to the light intensity; 5. The method of claim 4, wherein, The multi-channel electrical signals are converted into measurable voltage signals and amplified to obtain multi-channel electrical signals; The multi-channel electrical signals are analog-to-digital converted to obtain digital signals, and the multi-channel images are formed based on the digital signals. The application comprises the following steps: A phase encoder is used to introduce a micro time difference to form multi-frame sampling in the time domain, different channels are controlled by time offset mirrors to realize pseudo-synchronous sampling, a micro time difference sampling structure of adjacent channels is formed, and the formula is as follows: t i = t0+ At i t i denotes the sampling time point of the i-th channel; t0 represents the reference starting sampling time of all channels, that is, the sampling time of the first channel; Delay time Δt i is in the order of nanoseconds, and the calculation formula is as follows: represents the motion vector at the image coordinates (x, y), i.e. the instantaneous direction and speed of the target at that point; delta represents a small time interval, in seconds, milliseconds or nanoseconds; I(x, y, t) represents the pixel intensity value or brightness value at the image coordinates (x, y) at time t; The final motion direction is obtained by combining the direction gradient, and the motion information is generated; The direction gradient calculation formula is as follows: denotes the image gradient vector, pointing in the direction of the fastest change in brightness; unit gradient direction, for indicating edge orientation; The final motion direction calculation formula is as follows: denotes the final motion direction vector at image coordinates (x, y).

6. The method of claim 5, wherein, Based on the neural quasi-state neural network, the multi-channel image fusion is carried out, and the motion vector estimation is carried out based on the running information of the target, and the fused image frame and the motion vector map are output, which specifically includes: The shared encoder is used to extract features of each channel image to obtain encoded features z i The formula of the shared encoder is as follows: E i (I i )=σ(W e ·I i +b e ) E i (I i ) represents the encoding feature of the i-th channel image; sigma represents an activation function, which is used to introduce a nonlinear transformation; W e is a weight matrix of the shared encoder for linearly transforming the input image I i ; I i represents the i-th waveband data of the input image or multispectral image of the i-th channel; b e is a bias term for the shared encoder to adjust the output after linear transformation; Based on the neural quasi-state neural network, the encoding features of each channel are spliced in a high-dimensional space, and the calculation formula is as follows: Z = Concat(z1, z2,..., z 16 ) Concat(z1,z2,…,z 16 ) denotes the feature vectors extracted by the shared encoder for the images of the 16 different spectral channels, and indicates that the multiple feature vectors are sequentially concatenated into a higher-dimensional vector; Z represents the high-dimensional feature space vector obtained after splicing, which contains the fusion information of all channels; The space-time encoding vector is generated using the GRU structure to obtain the fused image frame; The motion vector estimation is calculated by the following formula: wherein denotes the partial derivative of the image intensity with respect to time t, which describes the rate of change of the brightness at this position with respect to time, i.e. the brightness change caused by motion; I(x,y,t) is the image intensity, is the gradient.

7. A high-speed multispectral parallel biomimetic vision imaging system, characterized in that, The system includes a memory and a processor, the memory includes a high-speed multispectral parallel biomimetic visual imaging method program, and the high-speed multispectral parallel biomimetic visual imaging method program is executed by the processor to realize the following steps: Based on the microlens array, the mantis shrimp compound eye structure is simulated to receive incident light in multiple directions; Based on the multispectral filter array, the incident light of different center wavelengths is selectively transmitted in different channels to obtain filtered light signals; Based on the photosensitive array, the filtered light signals are received and converted into multi-channel electrical signals to generate multi-channel images; Based on the phase encoder, time sequence sampling is carried out to obtain multiple frames of sampling results in the time domain, and the motion information of the target is captured based on the multiple frames of sampling results; Based on the neural quasi-state neural network, the multi-channel image fusion is carried out, and the motion vector estimation is carried out based on the running information of the target, and the fused image frame and the motion vector map are output.

8. The high-speed multispectral parallel biomimetic vision imaging system of claim 7, wherein, Based on the microlens array, the mantis shrimp compound eye structure is simulated to receive incident light in multiple directions, specifically including: The microlens array is set as the incident front end, and a biomimetic hexagonal compact arrangement is adopted, the microlens array includes a plurality of unit lenses, the diameter of each unit lens is 40-60 mu m, and the focal length is 50-150 mu m; The microlens array is arranged by a three-dimensional curved support, and is uniformly distributed on the spherical crown surface, The design formula is as follows: The microlens focal length calculation formula is as follows: Wherein: f: microlens focal length; d: lens diameter; n: material refractive index; The visual field expansion angle is: Where R is the spherical crown radius.

9. The high-speed multispectral parallel biomimetic vision imaging system of claim 8, wherein, Based on the multispectral filter array, the incident light of different center wavelengths is selectively transmitted in different channels to obtain filtered light signals, specifically including: A plurality of surface array filters are arranged in a two-dimensional grid, and each filter corresponds to a pixel or a pixel group of the detector; Based on the spectral coverage uniformity principle, the center wavelengths are allocated to obtain a visible light band and a near-infrared band, the visible light band is 400-700 nm, and the near-infrared band is 700-1100 nm; The first screening result is obtained by screening incident light based on a visible light band, and the second screening result is obtained by screening incident light based on a near-infrared band. The filtered light signal is obtained based on the first screening result and the second screening structure.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a high-speed multispectral parallel biomimetic visual imaging method program, and the high-speed multispectral parallel biomimetic visual imaging method program is executed by the processor to realize the steps of the high-speed multispectral parallel biomimetic visual imaging method in any one of claims 1 to 6.

Images