Bionic visual positioning system and method based on multispectral perception and storage medium

Abstract

The invention discloses a bionic visual positioning system and method based on multispectral perception and a storage medium, and the system comprises a front-end multidimensional optical perception module, a mesopic vision preprocessing module and a rear-end fusion perception and target positioning module, the front-end multi-dimensional optical sensing module is used for performing optical imaging, spectral decomposition and polarization imaging; the mesopic vision preprocessing module is used for carrying out image preprocessing and optical flow estimation; and the rear-end fusion perception and target positioning module is used for feature fusion and target identification and positioning. According to the bionic visual positioning mechanism, through a bionic compound eye structure + multispectral + polarization + optical flow fusion perception strategy, the perception bottleneck of a traditional visual system in a high-dynamic and complex illumination environment is effectively solved, and the bionic visual positioning mechanism has the advantages of high robustness, low delay, wide field of view, multiple modes and the like; the method is especially suitable for application scenes such as automatic driving, unmanned aerial vehicles, robots and the like with extremely high requirements on real-time performance and reliability.

Description

Technical Field

[0001] This invention relates to the field of visual processing technology, and more specifically, to a biomimetic visual positioning system, method, and storage medium based on multispectral sensing. Background Technology

[0002] Accurate target localization and tracking have always been challenging tasks in traditional computer vision systems and high dynamic target recognition systems, especially in high-speed dynamic environments, such as drone obstacle avoidance and autonomous driving collision avoidance.

[0003] Current mainstream solutions are mostly based on CMOS imaging arrays and convolutional neural networks (CNNs) for image feature extraction and moving target identification. However, these systems generally suffer from high processing latency, susceptibility to complex lighting conditions, and insufficient robustness. These shortcomings are particularly pronounced in extreme scenarios, such as high-speed travel through forests, densely built-up urban areas, and under strong sunlight reflection, severely limiting the scalability of existing methods in practical engineering deployments.

[0004] Currently, traditional vision processing systems typically employ a single optical path and a two-dimensional sensor array, such as ordinary RGB cameras or ToF sensors. While these systems offer a certain level of image clarity and color reproduction capability, they often face the following major challenges in dynamic environments:

[0005] (1) Large information acquisition delay: Single-channel image processing is often accompanied by inter-frame delay, which makes the system unable to respond to high-speed targets in real time;

[0006] (2) Sensitive to high light interference: The signal-to-noise ratio of CMOS drops sharply when it encounters specular reflection or high-contrast background;

[0007] (3) Unable to acquire motion and spectral features simultaneously: The lack of a cross-modal redundancy verification mechanism in target discrimination can easily lead to tracking failure;

[0008] (4) Limited field of view: Traditional imaging systems often have a field of view of less than 60°, which cannot cover the full-field perception requirements;

[0009] These technological shortcomings are becoming increasingly apparent in fields such as autonomous driving, aerial robots, and micro unmanned systems, and the problems urgently need to be solved. Summary of the Invention

[0010] The purpose of this invention is to provide a biomimetic visual positioning system, method, and storage medium based on multispectral sensing, to solve the technical problems of limited visual perception, poor anti-interference ability, slow response, and high latency in the prior art.

[0011] The first aspect of this invention provides a biomimetic visual positioning system based on multispectral sensing, comprising:

[0012] The system consists of a front-end multi-dimensional optical perception module, an intermediate visual preprocessing module, and a back-end fusion perception and target localization module.

[0013] The front-end multidimensional optical sensing module is used for optical imaging, spectral decomposition, and polarization imaging.

[0014] The intermediate visual preprocessing module is used for image preprocessing and optical flow estimation;

[0015] The backend fusion perception and target localization module is used for feature fusion and target recognition and localization.

[0016] In this solution, the front-end multidimensional optical sensing module includes a MEMS microlens array, a multispectral filter, and an electrically controlled polarizer group. The MEMS microlens array is used to simultaneously acquire images from multiple perspectives, the multispectral spectrometer is used to synchronously acquire spectral information from different bands, and the electrically controlled polarizer group is used to acquire images from different polarization angles.

[0017] In this scheme, the MEMS microlens array is arranged in an M×N configuration, with a total number of sensing units N. t = M×N, where the field of view of each microlens is set to θ0, and the direction of the central axis of the microlens is mapped by the array coordinates, as calculated below:

[0018]

[0019] in, For the positive forward direction, θ i,j ,φ i,j These represent the pitch and azimuth offset angles of the (i,j)th small eye unit, respectively, and R(θ,φ) is the three-dimensional rotation matrix. After field fusion, the array forms a full-coverage field of view Θ. total The calculation formula is as follows:

[0020] Θ total =max(θ) i,j )-min(θ i,j )+∈;

[0021] Where ∈ represents the overlapping redundant viewpoint, max(θ) i,j ) represents the maximum pitch angle of the (i,j)th small eye unit, min(θ) i,j ) represents the minimum pitch angle of the (i,j)th small eye unit.

[0022] In this scheme, a corresponding multispectral filter bank is set behind each microneedle, using a stacked Fabry-Perot structure or a grating-type beam splitter to divide the incident light into k channels according to wavelength. The signal expression for each channel is as follows:

[0023]

[0024] in, Here is the expression for the signal per channel, where k is the number of channels and λ is the value of the signal. k For channel wavelength, This represents the lower limit of the wavelength for k channels. L is the lower limit of the wavelength for k channels. scene For scene radiance, (x i,j ,y i,j ) represents the image plane coordinates, T k (λ) represents the transmittance of the filter in the k-th band, and QE(λ) represents the quantum efficiency of the photosensitive chip.

[0025] In this design, a polarizer group is placed in front of each small-eye optical path, and the polarization angle θ of each group is adjusted by a rotating electronically controlled liquid crystal. p The polarization state is periodically sampled as θ p Four images at 0°, 45°, 90°, and 135°:

[0026]

[0027] Among them, I ⊥ I represents the vertical component. || Representing the parallel components, the degree of polarization and polarization angle are reconstructed using the Stokes vector, and the calculation formula is as follows:

[0028]

[0029] Where DoLP is the degree of polarization and AoP is the polarization angle.

[0030] In this scheme, feature fusion is performed based on joint modeling of multi-channel spectroscopy and motion perception. Specifically, between frames t and t+Δt, the optical flow vector on each small eye channel k is calculated, and the least squares estimation of all channels is performed and merged to construct a joint optimization objective to obtain a unified full-band optical flow field V(x,y). The full-band optical flow field is combined into the spectral map as the final high-dimensional feature input for target recognition, classification and 3D reconstruction.

[0031] A second aspect of the present invention provides a biomimetic visual positioning method based on multispectral sensing, applicable to any of the biomimetic visual positioning systems based on multispectral sensing described in the present invention, wherein the method includes the following steps:

[0032] Optical imaging, spectral decomposition, and polarization imaging are performed based on a pre-defined front-end multi-dimensional optical sensing module.

[0033] Image preprocessing and optical flow estimation are performed based on a preset intermediate visual preprocessing module;

[0034] A pre-defined backend fusion perception and target localization module is used for feature fusion and target recognition and localization.

[0035] In this solution, optical imaging, spectral decomposition, and polarization imaging are performed based on a pre-set front-end multi-dimensional optical sensing module, specifically including:

[0036] The front-end multidimensional optical sensing module includes a MEMS microlens array, a multispectral filter, and an electrically controlled polarizer group.

[0037] The aforementioned MEMS microlens array is used to simultaneously acquire images from multiple perspectives.

[0038] The multispectral spectrometer is used to simultaneously acquire spectral information in different bands;

[0039] Images with different polarization angles are obtained based on the electronically controlled polarizer group.

[0040] In this solution, a pre-defined backend fusion perception and target localization module is used for feature fusion and target recognition and localization, specifically including:

[0041] Construct a joint feature vector and input it into a pre-defined model for target recognition and classification;

[0042] The system performs spatial localization and motion tracking on the identified targets and outputs the identification results with spatiotemporal labeling information.

[0043] A third aspect of the present invention provides a computer-readable storage medium comprising a machine program for a bionic visual positioning method based on multispectral perception, wherein when executed by a processor, the program implements the steps of the bionic visual positioning method based on multispectral perception as described in any of the preceding claims.

[0044] This invention discloses a biomimetic visual positioning system, method, and storage medium based on multispectral sensing. The biomimetic visual positioning mechanism effectively solves the perception bottleneck of traditional visual systems in high-dynamic and complex lighting environments through a fusion sensing strategy of biomimetic compound eye structure + multispectral + polarization + optical flow. It possesses advantages such as high robustness, low latency, wide field of view, and multimodal capabilities, and has a wide range of applications. Specific beneficial effects are as follows:

[0045] 1. Compared with traditional single-camera systems, this invention adopts a biomimetic distributed optical path design, with each small eye having an independent field of view, forming a spatial perception redundancy architecture. This hardware-level parallel processing greatly enhances dynamic response capability and fault tolerance, which is the foundation for realizing high-speed target perception.

[0046] 2. Traditional vision systems require time-series switching or multi-camera registration for multispectral acquisition, which results in inconsistencies between frames. In this invention, each eye integrates a band separation structure, which ensures that spectral data at any time and any spatial point are acquired synchronously, guaranteeing spatiotemporal consistency and real-time performance.

[0047] 3. This invention introduces a bio-inspired polarization differential imaging mechanism, which significantly enhances the system's ability to recognize highly interfering backgrounds such as specular reflection, water, and glass. By reconstructing polarization information to deduce the mask region, it provides an anti-interference mechanism for optical preprocessing for the visual perception system.

[0048] 4. This invention unifies the spatial spectral features and temporal optical flow features of an image into a single encoding vector, effectively preserving the target material properties and motion trend information, providing richer input for subsequent deep learning detectors, and improving the system's ability to distinguish between false targets and real obstacles;

[0049] 5. This invention has a compact structure and low power consumption, making it particularly suitable for installation and use on mobile platforms such as drones, vehicle-mounted systems, and micro-robots. It has the triple advantages of high speed, high robustness, and high energy efficiency, breaking through the performance bottleneck of existing visual perception systems in complex environments. Attached Figure Description

[0050] Figure 1 A schematic diagram of the composition of a biomimetic visual positioning system based on multispectral sensing according to the present invention is shown.

[0051] Figure 2 A schematic diagram of a biomimetic compound eye MEMS lens array structure of a biomimetic visual positioning system based on multispectral sensing according to the present invention is shown.

[0052] Figure 3 A schematic diagram of the signal transmission path of a single compound eye unit in a bionic visual positioning system based on multispectral sensing according to the present invention is shown.

[0053] Figure 4 A schematic diagram of multi-channel synchronous sensing and image fusion of a biomimetic visual positioning system based on multispectral sensing according to the present invention is shown.

[0054] Figure 5 The diagram illustrates the steps of a biomimetic visual positioning method based on multispectral sensing according to the present invention. Detailed Implementation

[0055] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0056] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0057] In nature, insects, especially flying insects such as bees, dragonflies, and fruit flies, exhibit exceptional three-dimensional spatial localization and moving object tracking capabilities, accurately perceiving obstacles and dynamic threats even with low-resolution imaging. This ability stems from their compound eye structure and neural parallel perception mechanism. Their compound eyes consist of multiple ommatidia, each receiving information from different parts of the visual field, while simultaneously processing motion vectors and spectral features to achieve stable perception of fast-moving targets. This inspiration led us to propose a multispectral parallel visual localization mechanism based on the biomimetic compound eye principle. Its core lies in embedding biological multisensory characteristics into miniature optical devices, supplemented by structured spectral polarization sensing, to enhance the system's response speed and robustness to highly dynamic targets.

[0058] To address this, the present invention designs a biomimetic vision system with multispectral sensing capabilities. The core components are a MEMS microlens array and a polarization spectral fusion sensing unit. By introducing a biomimetic parallel sensing architecture, it can simultaneously acquire multi-channel spectral information and motion vector estimation within a single shooting cycle, thereby significantly improving the ability to identify obstacles, threat sources, and targets in dynamic scenes. In application, it is particularly suitable for scenarios requiring high robustness and high response speed, such as high-speed obstacle avoidance of UAVs in complex terrain and transient obstacle recognition in autonomous driving.

[0059] Figure 1 A schematic diagram of the composition of a biomimetic visual positioning system based on multispectral sensing according to this application is shown.

[0060] like Figure 1 As shown, this application discloses a biomimetic visual positioning system based on multispectral sensing, comprising:

[0061] The system consists of a front-end multi-dimensional optical perception module, an intermediate visual preprocessing module, and a back-end fusion perception and target localization module.

[0062] The front-end multidimensional optical sensing module is used for optical imaging, spectral decomposition, and polarization imaging.

[0063] The intermediate visual preprocessing module is used for image preprocessing and optical flow estimation;

[0064] The backend fusion perception and target localization module is used for feature fusion and target recognition and localization.

[0065] It should be noted that, in this embodiment, the front-end multi-dimensional optical sensing module consists of a MEMS microlens array, a spectral beam splitter, and a polarizer, realizing spatial large field of view, multi-angle, and multi-channel imaging. The intermediate visual preprocessing module is used to perform operations such as image registration, spectral reconstruction, polarization difference, and optical flow estimation to complete the structuring of sensing data. The back-end fusion sensing and target localization module is used to perform dynamic target recognition, tracking, and localization estimation using motion-spectral joint features. The modules work together to achieve robust tracking capability for highly dynamic targets under complex lighting conditions. Specifically, optical imaging involves simultaneously acquiring multi-view images through a MEMS microlens array, while spectral decomposition corresponds to a multispectral filter after each small eye to simultaneously acquire multi-band images. Correspondingly, polarization imaging involves acquiring images at different polarization angles by rotating the polarizer and calculating DoLP and AoP. The image preprocessing process includes distortion correction, image registration, spectral fusion, and polarization difference, while optical flow estimation is based on multi-channel images to calculate pixel-level motion vectors and feature fusion: constructing a high-dimensional joint feature vector F(x,y).

[0066] According to an embodiment of the present invention, the front-end multidimensional optical sensing module includes a MEMS microlens array, a multispectral filter, and an electrically controlled polarizer group. The MEMS microlens array is used to simultaneously acquire images from multiple perspectives, the multispectral spectrometer is used to synchronously acquire spectral information of different bands, and the electrically controlled polarizer group is used to acquire images with different polarization angles.

[0067] It should be noted that, in this embodiment, as Figure 2 As shown, this is a schematic diagram of a biomimetic compound eye MEMS lens array structure. Multiple spherical microlenses are arranged in a honeycomb pattern, and the viewing angle of each microlens is slightly offset to form an overall spherical field of view. Behind each unit is a micro photosensitive chip with local spectral / polarization sensing capabilities. The array is arranged in a hemispherical pattern, and the overall field of view can cover 160° to 180°.

[0068] According to an embodiment of the present invention, the MEMS microlens array is arranged in an M×N configuration, with a total number of sensing units N. t = M×N, where the field of view of each microlens is set to θ0, and the direction of the central axis of the microlens is mapped by the array coordinates, as calculated below:

[0069]

[0070] in, For the positive forward direction, θ i,j ,φ i,j These represent the pitch and azimuth offset angles of the (i,j)th small eye unit, respectively, and R(θ,φ) is the three-dimensional rotation matrix. After field fusion, the array forms a full-coverage field of view Θ. totalThe calculation formula is as follows:

[0071] Θ total =max(θ) i,j )-min(θ i,i )+∈;

[0072] Where ∈ represents the overlapping redundant viewpoint, max(θ) i,j ) represents the maximum pitch angle of the (i,j)th small eye unit, min(θ) i,j ) represents the minimum pitch angle of the (i,j)th small eye unit.

[0073] It should be noted that, in this embodiment, the visual structure of the insect compound eye consists of multiple units called "ommatidia," each with its own photoreceptor cell and optical axis orientation. Based on the principles of optical imaging, each ommatidia can be considered a narrow-field pinhole imaging system, its field of view covering a certain angle θ. i It also exhibits directional selectivity. The overall visual perspective of the compound eye is:

[0074]

[0075] Where N is the number of eyelets. Different eyelets sample the same target from different angles, forming multi-view information in space. This redundancy greatly improves the accuracy of spatial object positioning and dynamic perception. Assuming the target object's three-dimensional coordinates in space are (x, y, z), its relative position projection onto the i-th eyelet is:

[0076] (x′ i ,y′ i )=f i (x,y,z,θ i ,φ i );

[0077] Among them, f i Represents the imaging function, θ i and φ i Given the pitch and azimuth angles of the small eye, the spatial back-projection coordinates can be obtained using the least squares estimation method through registration of multiple small eye images.

[0078]

[0079] This spatial point inversion can be used for precise positioning of three-dimensional targets and is one of the core algorithm foundations of this invention.

[0080] Specifically, to achieve spatial parallel perception mimicking the compound eyes of insects, this invention employs MEMS manufacturing processes to construct a microlens array, where each lens is an independent ommatidia unit. Assuming the array is arranged in an M×N pattern, the total number of sensing units is:

[0081] N t =M×N;

[0082] The field of view of each microlens is set to θ0, and its central axis direction is mapped by the array coordinates as follows:

[0083]

[0084] in, For the positive forward direction, θ i,j ,φ i,j These represent the pitch and azimuth offset angles of the (i,j)th small eye unit, respectively, and R(θ,φ) is the three-dimensional rotation matrix. After field fusion, the array forms a full-coverage field of view Θ. total The calculation formula is as follows:

[0085] Θ total =max(θ) i,j )-min(θ i,i )+∈;

[0086] Where ∈ represents the overlapping redundant viewpoint, used for subsequent image stitching and geometric correction, max(θ) i,i ) represents the maximum pitch angle of the (i,j)th small eye unit, min(θ) i,j ) represents the minimum pitch angle of the (i,j)th small eye unit.

[0087] According to an embodiment of the present invention, a corresponding multispectral filter bank is set behind each microneedle, using a stacked Fabry-Perot structure or a grating-type beam splitter to divide the incident light into k channels according to wavelength, and the signal expression of each channel is as follows:

[0088]

[0089] in, Here is the expression for the signal per channel, where k is the number of channels and λ is the value of the signal. k For channel wavelength, This represents the lower limit of the wavelength for k channels. L is the lower limit of the wavelength for k channels. scene For scene radiance, (x i,i ,y i,j ) represents the image plane coordinates, T k (λ) represents the transmittance of the filter in the k-th band, and QE(λ) represents the quantum efficiency of the photosensitive chip.

[0090] It should be noted that, in this embodiment, as Figure 3The diagram shows the signal transmission path of a single compound eye unit. External light is refracted by microlenses and enters the beam-splitting module. Multi-band filters are stacked in a Fabry-Perot configuration. A polarizer is placed in front of the photosensitive array and its angle is adjusted by an electronically controlled liquid crystal. Multiple signals (different bands / polarizations) are fed into the photosensitive array and then read out in parallel. All small-eye signals are sent to the central control module for fusion processing. It should be noted that multispectral imaging technology can acquire images of the same scene on different wavelength channels. Commonly used bands include infrared (IR), visible light (RGB), and near-ultraviolet (NUV). We define the wavelength channels as a set Λ = {λ1, λ2, ..., λ...}. k The spectral response of the image in the j-th channel is:

[0091]

[0092] Where R(x,y,λ) is the reflectivity function of the scene, and S(λ) is the spectral response function of the sensor, this multispectral structure is embedded into the compound eye system, that is, the spectral dispersion mechanism in each ommatidia is constructed, forming:

[0093]

[0094] This structure ensures that images are sampled simultaneously across multiple bands, improving the ability to identify invisible properties such as the composition and texture of target materials, and is especially effective against background interference under complex lighting conditions.

[0095] Specifically, a miniature multispectral filter bank is set behind each micro-eye, using a stacked Fabry-Perot structure or a grating-type beam splitter to divide the incident light into k channels according to wavelength. The signal expression for each channel is as follows:

[0096]

[0097] in, Here is the expression for the signal per channel, where k is the number of channels and λ is the value of the signal. k For channel wavelength, This represents the lower limit of the wavelength for k channels. L is the lower limit of the wavelength for k channels. scene For scene radiance, (x i,j ,y i,j ) represents the image plane coordinates, T k (λ) represents the transmittance of the filter in the k-th band, and QE(λ) represents the quantum efficiency of the photosensitive chip.

[0098] After multi-channel synchronous sampling, a multi-dimensional image cube I is obtained. cube (x,y,k), that is:

[0099]

[0100] The image cube forms the basis of the spectral dimension, which is used for material differentiation and weak texture target detection.

[0101] According to an embodiment of the present invention, a polarizer group is provided in front of each small-eye optical path, and the polarization angle θ of each group is adjusted by a rotating electro-controlled liquid crystal. p The polarization state is periodically sampled as θ p Four images at 0°, 45°, 90°, and 135°:

[0102]

[0103] Among them, I ⊥ I represents the vertical component. || Representing the parallel components, the degree of polarization and polarization angle are reconstructed using the Stokes vector, and the calculation formula is as follows:

[0104]

[0105] Where DoLP is the degree of polarization and AoP is the polarization angle.

[0106] It should be noted that, in this embodiment, reflected light in nature often carries polarization characteristics, with specular reflection being particularly significant. Insects can distinguish light rays with different polarization directions through the microstructures within their photoreceptor cells, thereby reducing interference from strong reflections. This invention simulates this mechanism by adding a polarizer to each eyelid, with the polarization direction set to θ. p The collected light intensity is recorded as:

[0107]

[0108] Among them, I ⊥ with I || The polarization components, orthogonal and parallel to the polarizer, are respectively used to remove the effect of specular reflection through differential calculation: I clean (x,y)=I p (x,y,0°)-I p (x,y,90°), where this operation greatly improves the visibility of targets in scenarios such as water, glass, and snow, and provides the system with a stable and reliable ability to identify targets in high-reflectivity areas.

[0109] Specifically, to simulate the response structure of insect compound eyes to polarized light, a group of polarizers is placed in front of each microphthalm optical path, and the polarization angle θ of each group is adjusted by a rotating electro-controlled liquid crystal. p The polarization state is periodically sampled as θ p Four images at 0°, 45°, 90°, and 135°:

[0110]

[0111] Reconstructing the degree of polarization (DoLP) and angle of polarization (AoP) using Stokes vectors:

[0112]

[0113] Therefore, based on this, the specular reflection region is suppressed and discriminated, and a polarization spectrum constraint mask M(x,y) is constructed to shield the interference region in the subsequent identification process.

[0114] According to an embodiment of the present invention, feature fusion is performed based on joint modeling of multi-channel spectroscopy and motion sensing. In this process, between frames t and t+Δt, the optical flow vector on each small-eye channel k is calculated, and the least squares estimation of all channels is performed and merged to construct a joint optimization objective to obtain a unified full-band optical flow field V(x,y). The full-band optical flow field is combined into the spectral map as the final high-dimensional feature input for target recognition, classification and three-dimensional reconstruction.

[0115] It should be noted that, in this embodiment, as Figure 4 The diagram illustrates multi-channel synchronous sensing and image fusion. It specifically explains how signals from multiple sensing channels (spatial angle × spectral channel × polarization channel) are structured and stitched together in the intermediate processing module. This includes geometric correction (such as distortion correction and scale unification) of all microlens imaging areas; registration of multi-band images to output a spectral cube in a unified spatial coordinate system; differential reconstruction of polarization images to generate DoLP and AoP parameter maps; and finally, integration of all data to form a joint feature map F(x,y).

[0116] To extract motion vector information, we performed optical flow calculations on the differences in image size within the small-eye field of view of consecutive frames, using the Horn-Schunck optical flow model, whose constraint equations are as follows:

[0117] I x u+I y v+I t =0;

[0118] Where u,v are the motion vector components of the pixel in the x,y directions, and I x ,I y ,I t The cost function is constructed by adding a smoothing term to the gradients of the image in each direction:

[0119]

[0120] By solving this optimization problem, the pixel-level motion vector field V(x,y)=(u(x,y),v(x,y)) within the image region can be obtained. This information, combined with the multi-channel image, constitutes a joint "motion-spectrum" feature vector.

[0121] F(x,y)=[I1(x,y),…,I k (x,y),u(x,y),v(x,y)] T ;

[0122] The use of this vector as input for target detection and trajectory prediction is another important innovative aspect of this invention.

[0123] Specifically, to achieve joint modeling of multi-channel spectroscopy and motion sensing, this system is designed with the following sensing process: Between frames t and t+Δt, the optical flow vector on each small-eye channel k is calculated:

[0124]

[0125] Then, least-squares estimation is performed on all channels and the results are combined to construct a joint optimization objective:

[0126]

[0127] Where Ω(u,v) represents the smoothing regularization term, and λ is the weighting coefficient.

[0128] Finally, a unified full-band optical flow field V(x,y) is obtained, which is then further combined into the spectral image:

[0129]

[0130] This vector serves as the final high-dimensional feature input, used for target recognition, classification, and 3D reconstruction.

[0131] Figure 5 The diagram illustrates the steps of a biomimetic visual positioning method based on multispectral sensing according to this application.

[0132] like Figure 5 As shown, this application discloses a biomimetic visual positioning method based on multispectral sensing, applicable to any of the biomimetic visual positioning systems based on multispectral sensing described in this application, wherein the method includes the following steps:

[0133] S502 performs optical imaging, spectral decomposition, and polarization imaging based on a preset front-end multidimensional optical sensing module.

[0134] S504 performs image preprocessing and optical flow estimation based on a preset intermediate visual preprocessing module;

[0135] S506, based on a preset backend fusion perception and target localization module, is used for feature fusion and target recognition and localization.

[0136] It should be noted that, in this embodiment, according to the present invention, optical imaging, spectral decomposition, and polarization imaging are performed based on a preset front-end multidimensional optical sensing module, specifically including:

[0137] The front-end multidimensional optical sensing module includes a MEMS microlens array, a multispectral filter, and an electrically controlled polarizer group.

[0138] The aforementioned MEMS microlens array is used to simultaneously acquire images from multiple perspectives.

[0139] The multispectral spectrometer is used to simultaneously acquire spectral information in different bands;

[0140] Images with different polarization angles are obtained based on the electronically controlled polarizer group.

[0141] It should be noted that, in this embodiment, according to the present invention, the preset backend fusion perception and target localization module is used for feature fusion and target recognition and localization, specifically including:

[0142] Construct a joint feature vector and input it into a pre-defined model for target recognition and classification;

[0143] The system performs spatial localization and motion tracking on the identified targets and outputs the identification results with spatiotemporal labeling information.

[0144] It should be noted that, in this embodiment, the specific implementation of the bionic visual positioning method based on multispectral perception disclosed in this application is consistent with the content of the above-mentioned bionic visual positioning system based on multispectral perception, and therefore will not be repeated in this embodiment.

[0145] A third aspect of the present invention provides a computer-readable storage medium comprising a program for a bionic visual positioning method based on multispectral perception, wherein when the program is executed by a processor, it implements the steps of the bionic visual positioning method based on multispectral perception as described in any of the preceding claims.

[0146] This invention discloses a biomimetic visual positioning system, method, and storage medium based on multispectral perception. The biomimetic visual positioning mechanism effectively solves the perception bottleneck of traditional vision systems in highly dynamic and complex lighting environments through a fusion perception strategy of biomimetic compound eye structure + multispectral + polarization + optical flow. It possesses advantages such as high robustness, low latency, wide field of view, and multimodal capabilities, and has a wide range of applications. Particularly in UAV obstacle avoidance, this invention can capture multi-angle images and motion vectors in real time during flight, achieving rapid navigation and obstacle avoidance without complex map priors. In autonomous driving, when faced with situations such as road surface water or window reflections, traditional vision systems are prone to misjudging targets or distortion. This system, through a polarization light filtering mechanism, can accurately extract effective information. In the future, it can be further integrated with microprocessor chips to realize real-time target detection and inference at the front end, constructing a complete vision SoC system.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0148] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0150] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A multispectral perception based biomimetic vision positioning system, characterized in that, The method comprises the following steps: The front-end multi-dimensional optical perception module, the intermediate visual preprocessing module, and the rear-end fusion perception and target positioning module are used for optical imaging, spectral decomposition, and polarization imaging. The intermediate visual preprocessing module is used for image preprocessing and optical flow estimation. The rear-end fusion perception and target positioning module is used for feature fusion, target identification, and positioning. The front-end multi-dimensional optical perception module comprises a MEMS microlens array, a multi-spectral filter, and an electrically controlled polarizer group.

2. The multispectral perception based biomimetic visual positioning system according to claim 1, characterized in that, Each ommatidium is provided with a corresponding multi-spectral filter group, which adopts a stacked Fabry-Perot structure or a grating type spectral element to divide the incident light into k channels according to the wavelength.

3. The multispectral perception based biomimetic visual positioning system according to claim 2, wherein, The MEMS microlens array is arranged in MxN, and the total number of sensing units N t = MxN, the field of view of each microlens is set to θ0, the microlens center axis direction is mapped by array coordinates, and the calculation formula is as follows: wherein, is the forward direction, θ i,j ,φ i,j are the pitch and yaw angles of the (i,j)th eyelet unit, R(θ,φ) is the 3D rotation matrix, and the array forms a full-coverage viewing angle Θ total The calculation formula is as follows: Θ total = max(θ i,j )- min(θ i,j )+ ε; where ∈ is the overlap redundancy view, max(θ i,j ) is the maximum tilt angle of the (i, j) ommatidium, and min(θ i,j ) is the minimum tilt angle of the (i, j) ommatidium.

4. The multispectral perception based biomimetic visual positioning system according to claim 3, characterized in that, The expression of each channel signal is as follows: wherein, is the expression for each channel signal, k is the channel, λ k is the channel wavelength, is the lower limit of wavelength for k channels, is the lower limit of wavelength for k channels, L scene is the scene radiance, (x i,j ,y i,j ) is the image plane coordinates, T k (λ) is the transmittance of the kth band filter, QE(λ) is the quantum efficiency of the photosensitive chip.

5. The multispectral perception based biomimetic visual positioning system according to claim 4, characterized in that, A polarizer set is arranged in front of each optic path of the microlens, each set is adjusted by rotating the electrically controlled liquid crystal to adjust the polarization angle θ p Four images are periodically collected with polarization states of θ p = 0°, 45°, 90°, 135°. where I ⊥ represents the vertical component, I || represents the parallel component, the degree of polarization and the polarization angle are calculated by reconstructing the Stokes vector, and the calculation formula is as follows: The expression of each channel signal is as follows:

6. The multispectral perception based biomimetic visual positioning system according to claim 5, characterized in that, The method comprises the following steps:

7. A method for biomimetic visual positioning based on multispectral perception, characterized in that The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps:

8. The method of claim 7, wherein, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps:

9. The method of claim 8, wherein, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps:

10. A computer-readable storage medium, characterized in that, The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps: The method comprises the following steps

Images