A multidimensional information processing method and sensing system based on curved surface bionic compound eye

Abstract

This invention relates to a spectral polarization imaging and image processing method and system, specifically a multi-dimensional information processing method and sensing system based on a curved bionic compound eye. It addresses the technical problems of existing spectral polarization imaging technologies, such as complex structural design, high precision requirements in manufacturing processes, and the difficulty in simultaneously acquiring spectral and polarization information. The multi-dimensional information processing method based on the curved bionic compound eye involves: 1) segmenting the original compound eye image into several sub-eye images; 2) reconstructing the sub-eye images to obtain a large field-of-view image containing seven types of information; 3) reconstructing the large field-of-view image; and 4) fusing the spectral and polarization information images to obtain a spectral polarization information fused image. The system of this invention includes a curved compound eye lens unit, an optical relay image-transfer unit, and a large-area image sensor arranged sequentially along the incident light; multi-dimensional information sensing can be achieved in a single imaging operation.

Description

Technical Field

[0001] This invention relates to a spectral polarization imaging and image processing method and system, specifically to a multidimensional information processing method and sensing system based on a curved surface bionic compound eye. Background Technology

[0002] Spatial, spectral, and polarization information are three different dimensions of effective information in the target information acquisition process. The fusion of multi-dimensional information greatly improves the accuracy of scene analysis and scene recognition. With the increasing demand for miniaturization and integration of optical imaging equipment, more and more researchers are beginning to integrate spectral imaging and polarization imaging technologies to achieve spectral polarization imaging, which simultaneously acquires object-side scene light intensity, spatial, spectral, and polarization information. Furthermore, compound eyes, as a common visual system in nature, have characteristics such as a large field of view and high temporal resolution. The mantis shrimp's eye is considered the most complex color visual system in the animal kingdom. Its compound eyes can provide stereoscopic vision, spectral vision sensitive to 16 bands, and polarization vision capable of identifying and analyzing linear and circular polarization, providing important references for research on spectral polarization imaging technology.

[0003] Current research on spectral polarization imaging systems focuses primarily on information fusion and practical applications, with limited progress in acquiring multi-dimensional information. Imaging systems capable of simultaneously acquiring spectral and polarization information typically utilize multi-camera arrays. These arrays acquire information from different dimensions, and image processing techniques are used for registration to obtain large-field-of-view spectral and polarization images. However, these systems generally suffer from complex structures, large sizes, and difficulties in data storage and transmission, hindering convenient real-time imaging or detection tasks. Furthermore, image registration is subject to errors. Focal plane detectors are currently a hot topic in spectral polarization imaging technology research. The common approach involves integrating mosaic-style micro-polarizers / filters onto the image sensor surface. If the mosaic-style micro-polarizer and filter are separate, each pixel can only acquire information from a specific wavelength or polarization state, failing to simultaneously acquire spectral and polarization information. While stacking the mosaic-style micro-polarizer and filter allows simultaneous acquisition of spectral and polarization information at the same pixel, energy utilization is extremely low, and focal plane detectors require high precision in manufacturing, resulting in high production costs. In addition to mosaic-type microarrays, researchers have also developed a series detector with four spectral channels and three polarization channels. The seven information channels are arranged along the same optical axis. Color selectivity is achieved through polarization interference, and polarization is used to control the spectral distribution of incident light. It can simultaneously acquire spectral and polarization information within a single pixel. However, this detector is difficult to design and requires extremely high precision in its manufacturing process, making it difficult to achieve engineering applications.

[0004] In summary, existing spectral polarization imaging technologies, whether at the imaging system level or the detector level, suffer from problems such as complex structural design, high precision requirements for manufacturing processes, and difficulty in simultaneously acquiring information in both spectral and polarization dimensions. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of existing spectral polarization imaging technologies, such as complex structural design, high precision requirements for manufacturing processes, and difficulty in simultaneously acquiring information in both spectral and polarization dimensions. This invention provides a multi-dimensional information processing method and sensing system based on a curved surface bionic compound eye, which enables the simultaneous acquisition of multi-dimensional information on the object-side scene space, multi-band spectra, and multi-angle polarization states.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A multidimensional information processing method based on a curved surface biomimetic compound eye, characterized by the following steps:

[0008] 1) Segment the original compound eye image into several sub-eye images;

[0009] 2) Reconstruct several sub-eye images to obtain a large field-of-view image with seven types of information;

[0010] 2.1 Classify and group several sub-eye images to obtain 3 groups of spectral information sub-eye images, 3 groups of polarization information sub-eye images, and 1 group of other information sub-eye images; the other information is spectral information, polarization information, or panchromatic information.

[0011] 2.2 Reconstruct the three sets of spectral information sub-eye images, the three sets of polarization information sub-eye images, and the one set of other information sub-eye images one by one to obtain three sets of spectral information large field-of-view images, three sets of polarization information large field-of-view images, and one set of other information large field-of-view images.

[0012] 3) Reconstruct a large field-of-view image;

[0013] 3.1. The three sets of large field-of-view spectral information images are reconstructed in a secondary manner to obtain spectral information images;

[0014] 3.2. The three sets of large field-of-view polarization information images are reconstructed in a secondary manner to obtain polarization information images;

[0015] 4) Fuse the spectral information image and the polarization information image to obtain a spectral polarization information fused image.

[0016] Furthermore, step 3) also includes step 3.3:

[0017] A set of other information large field-of-view images and spectral information images or polarization information images are reconstructed three times to obtain spectral information images or polarization information images;

[0018] Step 1) specifically involves:

[0019] 1.1 Using the Hough circle detection algorithm, detect the contours of N circular sub-eye images in the original compound eye image, establish spatial coordinates and pixel coordinates, and obtain the center coordinates (Xn, Yn) and radius Rn of the nth circular sub-eye image; n = 1…N, N = 3k 2 +3k+1, where k is a positive integer;

[0020] 1.2. Based on the center coordinates (Xn, Yn) of the circular sub-eye image, locate it, and use the radius Rn of the circular sub-eye image as the cropping radius. Then, segment the original compound eye image by image cropping, and divide it into several independent circular sub-eye images. After that, number each circular sub-eye image.

[0021] Furthermore, step 2.1 specifically includes:

[0022] The information type of each circular sub-eye image in step 1.2 is identified by numbering, and then classified and grouped to obtain 3 groups of spectral information sub-eye images, 3 groups of polarization information sub-eye images, and 1 group of other information sub-eye images;

[0023] Step 2.2 specifically involves:

[0024] 2.2.1. Using the central sub-eye of the original compound eye image as a reference, correct the magnification of all other sub-eye images to be consistent with the central sub-eye image;

[0025] 2.2.2. Based on the mathematical model of the imaging principle for obtaining the original compound eye image, map 3 sets of spectral information sub-eye images, 3 sets of polarization information sub-eye images, and 1 set of other information sub-eye images to the primary image plane, and then map them to the object-side scene.

[0026] 2.2.3. Based on the relationship between spatial coordinates and pixel coordinates, the object-side scene is directly mapped to the secondary image plane through coordinate transformation, completing one reconstruction to obtain 3 sets of spectral information large field-of-view images, 3 sets of polarization information large field-of-view images, and 1 set of other information large field-of-view images.

[0027] Furthermore, step 3.1 specifically includes:

[0028] Based on the fact that a large-area image sensor can obtain spectral information of three spectral bands at the same pixel point in the three sets of spectral information large field-of-view images, a spectral information image is reconstructed; the spectral information image is an RGB color image of the object-side scene; the three spectral bands are red spectral information, green spectral information and blue spectral information;

[0029] Step 3.2 specifically involves:

[0030] 3.2.1 Based on the polarization information of the three polarization states that can be obtained by the large-area image sensor at the same pixel point in the three sets of polarization information large field-of-view images, the polarization information image is reconstructed; by the relationship between the light intensity value received by the large-area image sensor and the gray value of the output image, the light intensity value under different polarization states at each pixel point is obtained, and the three sets of polarization information large field-of-view images are processed separately to obtain the three sets of light intensity values.

[0031] 3.2.2. Calculate the Mueller matrix based on the three sets of light intensity values, and then obtain the Stokes vector of the object-side scene;

[0032] 3.2.3. The object-side scene polarization information images with three polarization angles are reconstructed from the Stokes vectors of the object-side scene. The polarization information images include polarization degree images and polarization angle images. The three polarization angles are 0°, 90° and 135°.

[0033] Step 3.3 specifically involves:

[0034] Select one set of other large-field images containing spectral information and reconstruct them three times with the RGB color image obtained in step 3.1; the spectral information includes infrared spectral information or ultraviolet spectral information;

[0035] Alternatively, select one set of other large-field images containing polarization information and reconstruct them three times with the polarization degree image and polarization angle image obtained in step 3.2.3; the polarization information includes linear polarization state or circular polarization state.

[0036] Further, step 4) specifically involves:

[0037] 4.1 Map the polarization degree of the polarization image to a parameter representing brightness in the color space;

[0038] 4.2. Subtract the spectral information of the three different bands in the spectral information image and map them to the parameters representing hue and saturation in the color space respectively.

[0039] 4.3. In the color space, the parameters representing brightness are fused with the parameters representing hue and saturation, and the fused image with spectral polarization information is output.

[0040] Meanwhile, the present invention also provides a multi-dimensional information perception system based on a curved surface bionic compound eye, used to realize the above-mentioned multi-dimensional information processing method based on a curved surface bionic compound eye, which is characterized by including a data acquisition module and a processing module.

[0041] The acquisition module includes a curved multi-lens lens unit, an optical relay image transfer unit, and a large-area image sensor arranged sequentially along the incident light.

[0042] The curved lens unit includes a spherical support housing, an information acquisition array and a sub-eye lens array arranged sequentially on the spherical support housing along the incident light; the information acquisition array is used to acquire multi-dimensional information of the object scene; the sub-eye lens array is arranged concentrically with the information acquisition array and is used to convert the multi-dimensional information of the object scene into a first-order curved image;

[0043] The sub-eye lens array includes multiple sub-eye lenses arranged in a hexagonal pattern; each sub-unit of the information acquisition array corresponds one-to-one with each sub-eye lens of the sub-eye lens array and is coaxially arranged.

[0044] The optical relay image conversion unit is used to convert a primary curved surface image into a secondary planar image and guide it to a large-area image sensor;

[0045] Large-area image sensors are used to form original compound eye images from secondary planar images;

[0046] The processing module employs the aforementioned multidimensional information processing method based on a curved surface bionic compound eye.

[0047] Furthermore, the spherical support housing is provided with mounting holes;

[0048] Multiple sub-lenses are set in the mounting holes and arranged in a hexagonal pattern from the center to the periphery;

[0049] The information acquisition array includes multiple filters and multiple polarizers; the filters and polarizers are respectively disposed in the mounting holes corresponding to the sub-eye lenses and are located on the incident light path of the corresponding sub-eye lenses; the wavelength band of any filter is different from that of other filters; the polarization state of any polarizer is different from that of other polarizers.

[0050] A filter or polarizer in the same mounting hole and a sub-eye lens form an imaging sub-eye; seven imaging sub-eyes form a cluster eye, and each cluster eye is equipped with three imaging sub-eyes of different wavelengths and three imaging sub-eyes of different polarization states. Adjacent imaging sub-eyes have a certain overlap rate. All seven imaging sub-eyes image the same target in the object-side scene, thereby obtaining at least three different polarization angle information and three wavelength spectral information.

[0051] Furthermore, the optical relay image conversion unit includes 6 to 13 lenses arranged sequentially along the incident light;

[0052] The sub-lens is formed by cementing two lenses together;

[0053] Field of view of a single imaging sub-eye It is greater than the optical axis angle ΔΦ between adjacent imaging sub-eyes.

[0054] Furthermore, the total focal length of the acquisition module is 2.7mm, the total field of view is 122°, the relative aperture is 1 / 3, the maximum aperture is less than 45mm, and the total optical length is 190mm.

[0055] Furthermore, the optical relay image conversion unit includes 10 lenses arranged sequentially along the incident light;

[0056] Field of view of the imaging sub-eye The angle between the optical axes of adjacent imaging sub-eyes is 24°, the angle ΔΦ between adjacent imaging sub-eyes is 7°, and the field of view overlap rate of adjacent imaging sub-eyes is 66.38%.

[0057] The wavelength range of the filter includes far-infrared, near-infrared, red, green, blue and ultraviolet light.

[0058] The polarizers are combinations of polarizers with polarization angles of 0°, 30°, 45°, 60°, 90° and 135°, or circular polarizers or linear polarizers and quarter-wave plates.

[0059] The large-area image sensor uses a CMOS image sensor.

[0060] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0061] 1. The present invention is based on a multi-dimensional information processing method for curved surface bionic compound eyes, which can realize many functions such as large-scale target recognition and monitoring, target recognition under complex weather conditions, and close-range three-dimensional imaging, and can meet the requirements of integration and miniaturization of current spectral polarization imaging systems.

[0062] 2. This invention is based on a multi-dimensional information processing method for curved surface bionic compound eyes. It can reconstruct color images of large field-of-view images through image reconstruction algorithms, calculate the Stokes vector parameters of the large field-of-view scene, and calculate polarization information parameters such as polarization degree and polarization angle of the image in the scene. It can also achieve the fusion of spectral information images and polarization information images under the large field of view, improve the contrast between the target and the background in the original compound eye image, and further highlight the target's detailed texture information and edge information. It can also realize practical applications such as UAV polarization autonomous navigation and close-range three-dimensional imaging.

[0063] 3. The present invention is based on a multi-dimensional information processing method for curved surface bionic compound eyes. By applying the fusion of the above-mentioned spectral information and polarization information in the color space, it can realize functions such as wide-area monitoring of targets of interest and image dehazing under complex weather conditions.

[0064] 4. The present invention is based on a multi-dimensional information perception system for curved surface bionic compound eyes. It can simultaneously acquire spatial information, multi-band spectral information and multi-angle polarization state information of a large field of view scene through a single imaging, thereby achieving the purpose of multi-dimensional information perception.

[0065] 5. The present invention is based on a multi-dimensional information sensing system of a curved bionic compound eye. The optical relay image transfer unit adopts a telecentric optical path design with 6 to 13 lenses, so that the outgoing light rays from the object surface in different fields of view are perpendicular to the image plane, which can minimize the difference in the influence of defocusing of the large-area image sensor on different field of view angles.

[0066] 6. In the multi-dimensional information sensing system based on the curved bionic compound eye of this invention, multiple filters, multiple polarizers, and sub-eye lenses are designed for mass production, making assembly and adjustment simple, and easy to replace and maintain. All lens surfaces in the optical relay image-transfer unit can adopt standard surface types, and the material is ordinary optical glass, which can reduce costs to a certain extent.

[0067] 7. This invention is based on a multi-dimensional information perception system using a curved surface bionic compound eye. It employs an information acquisition array to perceive multi-dimensional information of the object scene. Seven imaging sub-eyes form a cluster of eyes, and each imaging sub-eye acquires different information. Each cluster of eyes can obtain at least three spectral information and at least three polarization information. The spectral and polarization information can be freely selected according to the actual application needs, providing high flexibility. Attached Figure Description

[0068] Figure 1 This is a flowchart illustrating an embodiment of the multidimensional information processing method based on curved surface bionic compound eye of the present invention;

[0069] Figure 2 This is a schematic diagram of the multi-dimensional information perception system based on curved surface bionic compound eye of the present invention;

[0070] Figure 3 This is a schematic diagram of the curved compound eye lens unit in an embodiment of the multi-dimensional information perception system based on curved bionic compound eye of the present invention;

[0071] Figure 4 This is a schematic diagram of the structure of the optical relay image-transfer unit in an embodiment of the multi-dimensional information perception system based on curved surface bionic compound eye of the present invention;

[0072] Figure 5 This is a schematic diagram of the information acquisition array arrangement in an embodiment of the multi-dimensional information perception system based on curved surface bionic compound eye of the present invention;

[0073] Figure 6 This is a schematic diagram of the field of view of a single imaging sub-eye and the angle between the optical axes of adjacent imaging sub-eyes in an embodiment of the multi-dimensional information perception system based on curved bionic compound eyes of the present invention.

[0074] Figure 7 This is a schematic diagram of the overlapping visual fields between cluster eyes in an embodiment of the multidimensional information perception system based on curved bionic compound eyes of the present invention.

[0075] The attached figures are labeled as follows:

[0076] 1-Curved compound eye lens unit, 11-Information acquisition array, 111-Filter, 112-Polarizer, 12-Sub-eye lens array, 121-Sub-eye lens, 13-Spherical support housing, 131-First spherical curved surface, 132-Second spherical curved surface, 2-Optical relay image transfer unit, 3-Large-area image sensor. Detailed Implementation

[0077] like Figure 1 As shown, a multidimensional information processing method based on a curved surface bionic compound eye includes the following steps:

[0078] 1) Segment the original compound eye image into several sub-eye images;

[0079] 1.1 Using the Hough circle detection algorithm, detect the contours of N circular sub-eye images in the original compound eye image, establish spatial coordinates and pixel coordinates, and obtain the center coordinates (Xn, Yn) and radius Rn of the nth circular sub-eye image; n = 1…N, N = 3k 2 +3k+1, where k is a positive integer;

[0080] 1.2. Based on the center coordinates (Xn, Yn) of the circular sub-eye image, locate it, and use the radius Rn of the circular sub-eye image as the cropping radius. Then, segment the original compound eye image by image cropping, and divide it into several independent circular sub-eye images. After that, number each circular sub-eye image.

[0081] 2) Reconstruct several sub-eye images to obtain a large field-of-view image with seven types of information;

[0082] 2.1. Classify and group several sub-eye images to obtain 3 groups of spectral information sub-eye images, 3 groups of polarization information sub-eye images, and 1 group of other information sub-eye images; the other information is spectral information, polarization information, or panchromatic information; specifically:

[0083] The information type of each circular sub-eye image in step 1.2 is identified by numbering, and then classified and grouped to obtain 3 groups of spectral information sub-eye images, 3 groups of polarization information sub-eye images, and 1 group of other information sub-eye images;

[0084] 2.2. Reconstruct the three sets of spectral information sub-eye images, three sets of polarization information sub-eye images, and one set of other information sub-eye images one by one to obtain three sets of spectral information large field-of-view images, three sets of polarization information large field-of-view images, and one set of other information large field-of-view images; specifically:

[0085] 2.2.1. Using the central sub-eye of the original compound eye image as a reference, correct the magnification of all other sub-eye images to be consistent with the central sub-eye image;

[0086] 2.2.2. Based on the mathematical model of the imaging principle for obtaining the original compound eye image, map 3 sets of spectral information sub-eye images, 3 sets of polarization information sub-eye images, and 1 set of other information sub-eye images to the primary image plane, and then map them to the object-side scene.

[0087] 2.2.3. Based on the relationship between spatial coordinates and pixel coordinates, the object-side scene is directly mapped to the secondary image plane through coordinate transformation, completing one reconstruction to obtain 3 sets of spectral information large field-of-view images, 3 sets of polarization information large field-of-view images, and 1 set of other information large field-of-view images.

[0088] 3) Reconstructing a large field-of-view image

[0089] 3.1. The three sets of large-field-of-view spectral information images are reconstructed in a secondary manner to obtain spectral information images; specifically:

[0090] Based on the fact that a large-area image sensor can obtain spectral information of three spectral bands at the same pixel point in the three sets of spectral information large field-of-view images, a spectral information image is reconstructed; the spectral information image is an RGB color image of the object-side scene; the three spectral bands are red spectral information, green spectral information, and blue spectral information;

[0091] 3.2. The three sets of large-field-of-view polarization information images are reconstructed in a secondary manner to obtain polarization information images; specifically:

[0092] 3.2.1 Based on the polarization information of the three polarization states that can be obtained by the large-area image sensor at the same pixel point in the three sets of polarization information large field-of-view images, the polarization information image is reconstructed; by the relationship between the light intensity value received by the large-area image sensor and the gray value of the output image, the light intensity value under different polarization states at each pixel point is obtained, and the three sets of polarization information large field-of-view images are processed separately to obtain the three sets of light intensity values.

[0093] 3.2.2. Calculate the Mueller matrix based on the three sets of light intensity values, and then obtain the Stokes vector of the object-side scene;

[0094] 3.2.3. Reconstruct polarization information images with three polarization angles from the Stokes vectors of the object-side scene. The polarization information images include polarization degree images and polarization angle images. The three polarization angles are 0°, 90°, and 135°.

[0095] 3.3. Reconstruct the spectral information image or polarization information image by combining the other large field-of-view images with the spectral information image or polarization information image three times to obtain the spectral information image or polarization information image.

[0096] Select one set of other large-field images containing spectral information, and reconstruct them three times with the RGB color image obtained in step 3.1; the spectral information includes infrared spectral information or ultraviolet spectral information;

[0097] Alternatively, select one set of other large-field images containing polarization information and reconstruct them three times with the polarization degree image and polarization angle image obtained in step 3.2.3; the polarization information includes other linear polarization states or circular polarization states.

[0098] 4) Fuse the spectral information image and the polarization information image to obtain a fused spectral and polarization information image;

[0099] 4.1 Map the polarization degree of the polarization image to a parameter representing brightness in the color space;

[0100] 4.2. Subtract the spectral information of the three different bands in the spectral information image and map them to the parameters representing hue and saturation in the color space respectively.

[0101] 4.3. In the color space, the parameters representing brightness are fused with the parameters representing hue and saturation, and the fused image with spectral polarization information is output.

[0102] In this embodiment, after obtaining the original compound eye image, the original compound eye image is first divided into several sub-eye images by an image segmentation algorithm, and then grouped according to the information type of the sub-eye images. The grouped sub-eye images are reconstructed into seven large field-of-view images with different information by a reconstruction algorithm. These images include spectral information images of the following three bands λ1, λ2, and λ3, polarization state information images of three angles P1, P2, and P3, and the remaining O is other information images.

[0103] The image segmentation algorithm is implemented using a circle segmentation algorithm based on the original compound eye image structure. It mainly includes two steps: circle detection and circle segmentation. The circle detection uses the well-known Hough circle detection algorithm. A detection radius threshold is set according to the pixel size of several sub-eye images. The algorithm detects the outline of circular sub-eye images in the original compound eye image of the whiteboard and obtains the coordinates (Xn, Yn) of each circle center and related parameters such as radius Rn.

[0104] Based on the obtained center coordinates (Xn, Yn) and radius Rn of each sub-eye image, image cropping is used to locate and segment several circular sub-eye images. After numbering each sub-eye image, the type of information acquired by the sub-eye image can be identified according to the arrangement of information acquisition units. Based on the type of information acquired by the sub-eye images (i.e., spectral information λ1, λ2, λ3, polarization information P1, P2, P3, and other information O), they are divided into 3 categories and 7 groups of sub-eye images, preparing for subsequent image reconstruction and image fusion.

[0105] The first image reconstruction is a spatial information reconstruction, which involves reconstructing the spatial information of each group of sub-eye images processed by the image segmentation algorithm. First, using the central sub-eye of the original compound eye image as a reference, the magnification of all other sub-eye images is corrected to match that of the central sub-eye image to reduce image distortion during reconstruction. Since there is a certain degree of field-of-view overlap between the imaging sub-eyes, any point in the object-side scene within the designed field of view can be acquired by the seven different imaging sub-eyes. Therefore, the corresponding spatial, spectral, and polarization information of any point within the designed field of view can be obtained. The sub-eye images acquired by the multi-dimensional information perception system are located at the secondary image plane, i.e., the focal plane of the large-area image sensor. According to the mathematical model of the imaging principle of the multi-dimensional information perception system, each group of sub-eye images is mapped to the primary image plane, and then mapped to the object scene. Based on the relationship between spatial coordinates and pixel coordinates, the spatial information of the object scene is directly mapped to the secondary image plane through coordinate transformation to obtain the large field-of-view image reconstructed from this group of information. The same image reconstruction method can also obtain 3 types of 7 groups of large field-of-view images with different information. The spatial information of the object scene should be the same between the large field-of-view images except at the edges.

[0106] The second image reconstruction involves reconstructing spatial and spectral information or polarization information. Filters 111, with center wavelengths of 650nm (red), 550nm (green), and 450nm (blue), are selected to acquire the object-side scene spectral information. From these three sets of large-field-of-view spectral information images, the R, G, and B spectral information of the object-side scene at the same pixel can be obtained, thus reconstructing the large-field-of-view RGB spectral information image of the object-side scene. Linear polarizers 112 at 0°, 90°, and 135° are selected to acquire the object-side scene polarization information. From these three sets of large-field-of-view polarization information images, based on the relationship between the light intensity value received at each pixel by the large-area image sensor 3 and the grayscale value of the output image, the light intensity value under three different polarization states at each point can be obtained. Calculating the three sets of linearly polarized light intensity values ​​from the three sets of large-field-of-view polarization information images allows for the calculation of the Mueller matrix of the polarization imaging unit of the multi-dimensional information sensing system, thereby obtaining the Stokes vector of each object point within the designed field of view. The polarization information image of the object scene can be reconstructed by the Stokes vector at each object point, including the polarization degree image and the polarization angle image. Both the polarization degree image and the polarization angle image can further highlight the target details and texture information in the image, and can realize related applications in the field of target recognition under complex weather conditions.

[0107] Furthermore, the acquired polarization information contains polarization state information that is orthogonal at 0° and 90°. Through known polarization difference operations, the polarization vision system of the compound eye of mantis shrimp can be simulated to obtain a polarization difference information image with the largest polarization contrast in the object scene. This information image can be used to realize the function of underwater image enhancement.

[0108] Image fusion algorithms further fuse the acquired spatial, spectral, and polarization information of the object scene to highlight image details, textures, and edge information. Since the information acquisition array 11 in the multi-dimensional information perception system corresponds one-to-one with the sub-eye lens 121, the fusion of spectral and spatial information, and polarization and spatial information, has been achieved during the imaging process of the large-area image sensor 3. When fusing spectral and polarization information, the two types of information are mapped to the HSI, HSV, Lab, or other color spaces according to certain rules. This embodiment selects the Lab color space, mapping the relatively independent polarization degree information in the polarization information image to the brightness parameter L in the Lab color space. * The three spectral information values ​​in the spectral information image are subtracted from each other and then mapped to the other two parameters 'a' in the Lab color space. * and b * The spectral and polarization information is fused in the Lab color space, and then mapped back to the RGB color space to finally output a large field-of-view spectral and polarization information fused image.

[0109] After the fusion of spatial, spectral, and polarization information of the object scene, the image details, textures, and edge features are further highlighted, and the target recognition and image enhancement effects can be further improved. It has broad application prospects in fields such as target recognition and tracking in a large field of view under complex weather conditions and underwater image enhancement.

[0110] like Figure 2 As shown, a multidimensional information perception system based on curved bionic compound eyes, inspired by the compound eye visual systems of arthropods and insects in the biological world, includes an acquisition module and a processing module; the acquisition module includes a curved compound eye lens unit 1, an optical relay image transfer unit 2, and a large-area image sensor 3 arranged sequentially along the incident light.

[0111] like Figure 3 As shown, the curved lens unit 1 includes a spherical support housing 13, and an information acquisition array 11 and a sub-eye lens array 12 are sequentially arranged on the spherical support housing 13 along the incident light. The information acquisition array 11 is used to acquire multi-dimensional information of the object scene. The sub-eye lens array 12 is concentrically arranged with the information acquisition array 11 and is used to convert the multi-dimensional information of the object scene into a single curved image.

[0112] The information acquisition array 11 is located on the first spherical surface 131 of the spherical support housing 13; the sub-eye lens array 12 includes a plurality of sub-eye lenses 121 located on the second spherical surface 132 of the spherical support housing 13 and concentric with the first spherical surface 131. The sub-eye lenses 121 are used to convert multi-dimensional information of the object scene into a first-dimensional curved image.

[0113] The spherical support housing 13 is used to set the information acquisition array 11 and the sub-eye lens array 12; the spherical support housing 13 is provided with mounting holes; multiple sub-eye lenses 121 are set in the mounting holes and arranged in a hexagonal pattern from the center to the periphery; the information acquisition array 11 includes multiple filters 111 and multiple polarizers 112; the filters 111 and polarizers 112 are respectively set in the mounting holes in a one-to-one correspondence with the sub-eye lenses 121 and are fixed by pressure rings or spacers, and the filters 111 and polarizers 112 are located on the incident light path of the sub-eye lenses 121; the wavelength of any filter 111 is different from that of other filters 111; the polarization state of any polarizer 112 is different from that of other polarizers 112.

[0114] In this embodiment, the sub-eye lens 121 is a cemented lens composed of a lens with a radius of curvature of 14mm, a thickness of 2.0mm and made of H-BAK2 material, and a lens with a radius of curvature of 6mm, a thickness of 6mm and made of H-ZBAF4 material. The sub-eye lenses 121 are all the same in terms of size, material and other basic properties, which can be easily installed and replaced, reducing the difficulty of initial installation and adjustment and subsequent maintenance, and saving costs to a certain extent.

[0115] The filter 111 or polarizer 112 in the same mounting hole and the sub-eye lens 121 form an imaging sub-eye; seven imaging sub-eyes form a cluster eye, and each cluster eye is provided with three imaging sub-eyes of different wavelengths and three imaging sub-eyes of different polarization states. Adjacent imaging sub-eyes have a certain overlap rate. All seven imaging sub-eyes image the same target in the object-side scene, thereby obtaining at least three different polarization angle information and three wavelength spectral information.

[0116] Apart from the edge imaging sub-eyes, the remaining six imaging sub-eyes, centered on each of the seven sub-eyes, together form a hexagonal seven-channel cluster eye. The seven imaging sub-eyes within the cluster eye, according to the information acquisition array 11 design, can acquire at least three different spectral information images in different bands and three different polarization information images at different angles. The last remaining imaging sub-eye can acquire either spectral or polarization information depending on the actual application. Through optical design, a certain degree of field-of-view overlap exists between the seven imaging sub-eyes within the cluster eye. This allows all seven different imaging sub-eyes in a single cluster eye to acquire corresponding spectral and polarization information at any object point. In other words, at the same object point, a single imaging session can acquire object-side scene space, multi-band spectral information, and multi-angle polarization state information, achieving multi-dimensional information perception.

[0117] There is image overlap between the images of the imaging sub-eyes, but no image aliasing. Image overlap refers to a certain degree of repetition or similarity among the sub-eye images in the original compound eye image, which is caused by the overlap of the imaging sub-eye's field of view and is a beneficial aspect in this invention. Image aliasing refers to the overlapping and mixing of images between the imaging sub-eyes, making it impossible to effectively distinguish between the images of each imaging sub-eye, and this must be avoided in this invention. Any object point in the object-space scene can be detected simultaneously by all seven imaging sub-eyes, ensuring that the information acquisition array 11 can acquire the corresponding multi-dimensional information of the target at the same object point in the object-space scene.

[0118] The optical relay image conversion unit 2 is used to convert a primary curved surface image into a secondary planar image and guide it to the large-area image sensor 3 for easy reception by the planar large-area image sensor 3; such as Figure 4 As shown, the optical relay image-transfer unit 2 consists of 10 lenses. The lens materials of the first lens 21 to the tenth lens 210 are all ordinary optical glass such as H-F13 and H-ZK7A, and their combination is not fixed. The relevant optical parameters of each lens in the optical relay image-transfer unit 2 in this embodiment are shown in the following table:

[0119]

[0120] Furthermore, the optical relay image-transfer unit 22 can effectively control optical aberrations and ensure that the image planes of adjacent imaging sub-eyes do not overlap, and the acquired information is free from crosstalk. All 10 lenses have standard surface shapes, which further helps control costs.

[0121] The large-area image sensor 3 is located behind the optical relay image conversion unit 2. It is used to acquire multi-dimensional information of the object-side scene of the array 11 and form the original compound eye image from the secondary planar image. In this embodiment, the large-area CMOS image sensor is used to convert the multi-dimensional information primary curved image acquired by the imaging sub-eye into a low-distortion secondary planar image for the large-area image sensor 3 to receive. The large-area image sensor 3 selected in this embodiment is a commercial CMOS image sensor, which can discretize continuous optical images into digital images after sampling. It does not require self-design or manufacturing and is responsible for image acquisition, which facilitates image reconstruction or image fusion by subsequent image processing algorithms.

[0122] like Figure 5As shown, the information acquisition array 11 is arranged in a hexagonal pattern on the first spherical curved surface 131. Each small hexagon and the image sub-eyes it encloses constitute a cluster eye (as enclosed by the dashed hexagons). That is, each cluster eye contains seven imaging sub-eyes, which can acquire seven different channels of information. Taking the imaging sub-eye at the edge of a cluster eye as the center, another cluster eye can still be defined by the six sub-eyes around it. Therefore, except for the outermost imaging sub-eye of the curved compound eye lens unit, the remaining imaging sub-eyes can form an image eye with the surrounding hexagons to form a cluster eye. There is field of view overlap between cluster eyes. The design allows any point in the object-side scene within the field of view to be simultaneously observed by the seven imaging sub-eyes of a cluster eye. That is, the system can acquire the corresponding information of the seven channels of that point to achieve the purpose of multi-dimensional information perception. In this embodiment, the information acquisition contains at least three band spectral information λ1, λ2, λ3 and three angular polarization state information P1, P2, P3. The remaining information channel O can be freely selected according to the pre-implemented function. The image received by the large-area image sensor 3 is the original compound eye image. The information obtained corresponds one-to-one with the information acquisition array 11. Due to the overlapping of the fields of view, at any pixel point in the image, multi-dimensional information such as object scene space, multi-band spectrum, and multi-angle polarization state can be acquired simultaneously. This enables multi-dimensional information perception with a large field of view, which can be applied to many fields such as target recognition under complex weather conditions, underwater image enhancement, and biomimetic polarization navigation.

[0123] like Figure 6 As shown, according to the optical design of the curved bionic compound eye system, the information acquisition array 11 is located on the first spherical curved surface 131, and the sub-eye lens array 12 is located on the second spherical curved surface 132, with their optical axes concentric. The field of view of a single imaging sub-eye... The angle between the optical axes of adjacent sub-eyes is 24°, and the angle ΔΦ between adjacent sub-eyes is 7°. The field-of-view overlap rate of adjacent imaging sub-eyes is 66.38%. The angle between the optical axes of adjacent imaging sub-eyes is smaller than the field-of-view angle of a single imaging sub-eye, which ensures that there is a certain field-of-view overlap between sub-eyes when the system of the present invention images the object-side scene. The larger the field-of-view angle of a single imaging sub-eye and the smaller the angle between the optical axes of adjacent imaging sub-eyes, the greater the degree of field-of-view overlap. However, it cannot be increased indefinitely, otherwise the imaging sub-eyes cannot meet the rigid support of the structure. Therefore, it is necessary to maximize the field-of-view overlap as much as possible while ensuring the assembly requirements of the imaging sub-eyes.

[0124] like Figure 7 As shown, by satisfying the relationship between the field of view angle of a single imaging sub-eye and the angle between the optical axes of adjacent imaging sub-eyes, overlap between the field of view of the imaging sub-eyes can be achieved. Multidimensional information systems and sensing methods need to acquire the object-space scene information, spectral information, and polarization information within the designed field of view. Therefore, within a cluster of eyes, the seven imaging sub-eyes need to have a certain degree of field of view overlap, with a minimum overlap of [value missing]. Figure 7As shown in the hexagon in the diagram, any point in the object space should be detected simultaneously by seven imaging sub-eyes; otherwise, multi-dimensional information perception of the object space scene cannot be achieved. Based on this, mathematical calculations show that the overlap rate of the fields of view of the seven imaging sub-eyes among the cluster eyes must be at least greater than 20.76%. The greater the overlap rate, the higher the accuracy of the subsequent image reconstruction algorithm and the better the image reconstruction quality.

[0125] The system of this invention has a total focal length of 2.7mm, a designed field of view of 122°, a relative aperture of 1 / 3, and 169 imaging sub-eyes. Preferably, the spectral channel can select from the far-infrared band, near-infrared band, red light band, green light band, blue light band, and ultraviolet band; the polarization channel can select from linear polarizers 112 with polarization angles of 0°, 90°, and 135°. In other embodiments, the polarization channel can also select from a combination of linear polarizers 112 capable of producing left-handed or right-handed circularly polarized light and quarter-wave plates, or linear polarizers 112 with polarization angles of 30°, 45°, and 60°.

[0126] The spherical support housing 1313 can be machined by a five-axis CNC machine tool, using aerospace aluminum as the material. In other embodiments, it can also be made using other materials through 3D printing. The spherical support housing 1313 features hexagonal stepped through holes for mounting the information acquisition array 11 and the sub-eye lens array 12. A certain distance of approximately 1.5mm to 2.0mm is maintained between adjacent stepped through holes to ensure that there is no crosstalk between the imaging sub-eyes, that is, to ensure that the images of the imaging sub-eyes do not overlap, and to isolate stray light from the gaps between the imaging sub-eyes. The total focal length of the multi-dimensional information sensing system of this invention is 2.7mm, the total field of view is 122°, the relative aperture is 1 / 3, the maximum aperture is less than 45mm, and the total optical length is 190mm.

Claims

1. A multidimensional information processing method based on curved surface bionic compound eye, characterized in that, Includes the following steps: 1) Segment the original compound eye image into several sub-eye images; 2) Reconstruct several sub-eye images to obtain a large field-of-view image with seven types of information; 2.1 Classify and group several sub-eye images to obtain 3 groups of spectral information sub-eye images, 3 groups of polarization information sub-eye images, and 1 group of other information sub-eye images; the other information is spectral information, polarization information, or panchromatic information; 2.2 Reconstruct the three sets of spectral information sub-eye images, the three sets of polarization information sub-eye images, and the one set of other information sub-eye images one by one to obtain three sets of spectral information large field-of-view images, three sets of polarization information large field-of-view images, and one set of other information large field-of-view images. 3) Reconstruct a large field-of-view image; 3.

1. The three sets of large field-of-view spectral information images are reconstructed in a secondary manner to obtain spectral information images; 3.

2. The three sets of large field-of-view polarization information images are reconstructed in a secondary manner to obtain polarization information images; 4) Fuse the spectral information image and the polarization information image to obtain a fused spectral and polarization information image: 4.1 Map the degree of polarization in the polarization image to the parameters representing brightness in the color space; 4.

2. Subtract the spectral information of the three different bands in the spectral information image and map them to the parameters representing hue and saturation in the color space respectively. 4.

3. In the color space, the parameters representing brightness are fused with the parameters representing hue and saturation, and the fused image with spectral polarization information is output.

2. The multi-dimensional information processing method based on curved surface bionic compound eye according to claim 1, characterized in that, Step 3) also includes step 3.3: A set of other information large field-of-view images and spectral information images or polarization information images are reconstructed three times to obtain spectral information images or polarization information images; Step 1) specifically involves: 1.1 Using the Hough circle detection algorithm, detect the contours of N circular sub-eye images in the original compound eye image, establish spatial coordinates and pixel coordinates, and obtain the center coordinates (Xn, Yn) and radius Rn of the nth circular sub-eye image; n = 1…N, N = 3k 2 +3k+1, where k is a positive integer; 1.

2. Based on the center coordinates (Xn, Yn) of the circular sub-eye image, locate it, and use the radius Rn of the circular sub-eye image as the cropping radius. Then, segment the original compound eye image by image cropping, and divide it into several independent circular sub-eye images. After that, number each circular sub-eye image.

3. The multidimensional information processing method based on curved surface bionic compound eye according to claim 2, characterized in that, Step 2.1 specifically involves: The information type of each circular sub-eye image in step 1.2 is identified by numbering, and then classified and grouped to obtain 3 groups of spectral information sub-eye images, 3 groups of polarization information sub-eye images, and 1 group of other information sub-eye images; Step 2.2 specifically involves: 2.2.

1. Using the central sub-eye of the original compound eye image as a reference, correct the magnification of all other sub-eye images to be consistent with the central sub-eye image; 2.2.

2. Based on the mathematical model of the imaging principle for obtaining the original compound eye image, map 3 sets of spectral information sub-eye images, 3 sets of polarization information sub-eye images, and 1 set of other information sub-eye images to the primary image plane, and then map them to the object-side scene. 2.2.

3. Based on the relationship between spatial coordinates and pixel coordinates, the object-side scene is directly mapped to the secondary image plane through coordinate transformation, completing one reconstruction to obtain 3 sets of spectral information large field-of-view images, 3 sets of polarization information large field-of-view images, and 1 set of other information large field-of-view images.

4. The multidimensional information processing method based on curved surface bionic compound eye according to claim 3, characterized in that step 3.1 specifically comprises: Based on the spectral information of the three sets of large field-of-view images, the large-area array image sensor (3) can obtain spectral information of three spectral bands at the same pixel point, and reconstruct the spectral information image; the spectral information image is an RGB color image of the object scene; the three spectral bands are red spectral information, green spectral information and blue spectral information; Step 3.2 specifically involves: 3.2.1 Based on the polarization information of the three polarization states that can be obtained by the large-area image sensor (3) at the same pixel point in the three sets of polarization information large field-of-view images, the polarization information image is reconstructed; by the relationship between the light intensity value received by the large-area image sensor (3) and the gray value of the output image, the light intensity value under different polarization states at each pixel point is obtained, and the three sets of polarization information large field-of-view images are processed respectively to obtain the three sets of light intensity values; 3.2.

2. Calculate the Mueller matrix based on the three sets of light intensity values, and then obtain the Stokes vector of the object-side scene; 3.2.

3. Reconstruct polarization information images with three polarization angles from the Stokes vectors of the object-side scene. The polarization information images include polarization degree images and polarization angle images; the three polarization angles are 0°, 90°, and 135°. Step 3.3 specifically involves: Select one set of other large-field images containing spectral information, and reconstruct them three times with the RGB color image obtained in step 3.1; the spectral information includes infrared spectral information or ultraviolet spectral information; Alternatively, select one set of other large-field images containing polarization information and reconstruct them three times with the polarization degree image and polarization angle image obtained in step 3.2.3; the polarization information includes other linear polarization states or circular polarization states.

5. A multidimensional information perception system based on a curved surface bionic compound eye, used to implement the multidimensional information processing method based on a curved surface bionic compound eye as described in any one of claims 1-4, characterized in that: Includes a data acquisition module and a processing module; The acquisition module includes a curved multi-lens lens unit (1), an optical relay image conversion unit (2), and a large-area image sensor (3) arranged sequentially along the incident light. The curved lens unit (1) includes a spherical support shell (13), and an information acquisition array (11) and a sub-eye lens array (12) are sequentially arranged on the spherical support shell (13) along the incident light. The information acquisition array (11) is used to acquire multi-dimensional information of the object scene. The sub-eye lens array (12) is concentrically arranged with the information acquisition array (11) and is used to convert the multi-dimensional information of the object scene into a first-order curved image. The sub-eye lens array (12) includes multiple sub-eye lenses (121) arranged in a hexagonal pattern; each sub-unit of the information acquisition array (11) corresponds one-to-one with each sub-eye lens (121) of the sub-eye lens array (12) and is coaxially arranged. The optical relay image conversion unit (2) is used to convert a primary curved surface image into a secondary planar image and guide it to a large-area image sensor (3); The large-area image sensor (3) is used to form the original compound eye image from the secondary planar image; The processing module employs a multi-dimensional information processing method based on a curved surface bionic compound eye, as described in any one of claims 1-4.

6. A multi-dimensional information perception system based on a curved surface bionic compound eye according to claim 5, characterized in that: The spherical support housing (13) is provided with mounting holes; Multiple sub-eye lenses (121) are disposed in the mounting hole and arranged in a hexagonal pattern from the center to the periphery; The information acquisition array (11) includes multiple filters (111) and multiple polarizers (112); the filters (111) and polarizers (112) are respectively disposed in the mounting holes corresponding to the sub-eye lenses (121), and are respectively located on the incident light path of the corresponding sub-eye lenses (121); the wavelength band of any filter (111) is different from that of other filters (111); the polarization state of any polarizer (112) is different from that of other polarizers (112); A filter (111) or polarizer (112) in the same mounting hole and a sub-eye lens (121) form an imaging sub-eye; seven imaging sub-eyes form a cluster eye, and each cluster eye is provided with three imaging sub-eyes of different wavelengths and three imaging sub-eyes of different polarization states. Adjacent imaging sub-eyes have a certain overlap rate. All seven imaging sub-eyes image the same target in the object-side scene, thereby obtaining at least three different polarization angle information and three wavelength spectral information.

7. A multi-dimensional information perception system based on a curved surface bionic compound eye according to claim 6, characterized in that: The optical relay image conversion unit (2) includes 6 to 13 lenses arranged sequentially along the incident light; The sub-eye lens (121) is formed by cementing two lenses together; The field of view Δφ of a single imaging sub-eye is greater than the optical axis angle ΔΦ of the adjacent imaging sub-eyes.

8. A multi-dimensional information perception system based on a curved surface bionic compound eye according to claim 7, characterized in that: The acquisition module has a total focal length of 2.7mm, a total field of view of 122°, a relative aperture of 1 / 3, a maximum aperture of less than 45mm, and a total optical length of 190mm.

9. A multi-dimensional information perception system based on a curved surface bionic compound eye according to any one of claims 6-8, characterized in that: The optical relay image conversion unit (2) includes 10 lenses arranged sequentially along the incident light; The field of view angle Δφ of the imaging sub-eye is 24°, the optical axis angle ΔΦ between adjacent imaging sub-eyes is 7°, and the field of view overlap rate between adjacent imaging sub-eyes is 66.38%. The filter (111) has the following wavelengths: far-infrared, near-infrared, red, green, blue and ultraviolet. The polarizer (112) is a combination of a polarizer or a circular polarizer or a linear polarizer with polarization angles of 0°, 30°, 45°, 60°, 90° and 135° and a quarter-wave plate. The large-area image sensor (3) is a CMOS image sensor.

Images