Multi-view Collaborative Super-Resolution Reconstruction Method Based on Large Field-of-View Compound Eye Images

Abstract

The present invention relates to computer vision and digital image processing methods, and more specifically, to a multi-perspective collaborative super-resolution reconstruction method based on large-field-of-view compound eye images. To address the shortcomings of super-resolution reconstruction, which struggles to simultaneously maintain both image and temporal resolution, requires high-precision equipment, and requires complex post-processing procedures, the present invention proposes a multi-perspective collaborative super-resolution reconstruction method based on large-field-of-view compound eye images. This method sequentially obtains the original compound eye image and compound eye imaging model, performs pixel-space mapping and addressing, and reconstructs the super-resolution image to obtain a super-resolution image, achieving large-field-of-view, high-signal-to-noise ratio, and super-resolution imaging.

Description

Technical Field

[0001] The present invention relates to computer vision and digital image processing methods, and in particular to a multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images. Background Art

[0002] With the growing demand for wide-field, high-resolution staring imaging in drone-mounted optoelectronic equipment, as well as the need for precise capture of moving targets in complex environments, super-resolution imaging technology holds the promise of further improving the spatial resolution of the system. Without changing any hardware, signal processing methods can be used to achieve imaging resolutions "superior" to those of traditional detectors. Image super-resolution technology offers advantages such as simplicity, reliability, and low cost, and therefore has high application value in a wide range of fields, including remote sensing, satellite imaging, security, astronomy, and biomedicine. Currently, image reconstruction algorithms can be used to restore high-resolution images from single-frame images, while adaptive weighted super-resolution reconstruction methods can be used to improve the spatial resolution of multi-frame remote sensing images. Scanning a microlens array to obtain light field information of a scene, followed by digital adaptive optics, can achieve high-resolution reconstruction to correct for wavefront distortion introduced to the telescope by atmospheric turbulence. Super-resolution reconstruction research using single-aperture lenses, whether using multi-frame image fusion or Fourier stacking techniques, generally faces the reliance on high-precision movement or changes in illumination direction. While improving image resolution, these methods inevitably sacrifice temporal resolution, increasing the complexity and cost of related experiments, and have significant limitations in capturing dynamic scenes. Another prominent issue is that they are almost impossible to apply to rapidly changing scenes, as even slight changes in the scene will lead to a significant decrease in reconstruction quality. In addition, the high-precision equipment requirements and complex post-processing processes have greatly limited the application of these technologies in mobile devices and real-world scenarios. Therefore, how to improve image and temporal resolution while reducing reliance on high-precision equipment has become a key issue that needs to be urgently addressed in the field of super-resolution reconstruction. Summary of the Invention

[0003] The purpose of the present invention is to solve the shortcomings of the prior art that super-resolution reconstruction is difficult to ensure both image resolution and temporal resolution, and has high-precision equipment requirements and complex post-processing procedures, and to provide a multi-perspective collaborative super-resolution reconstruction method based on large-field-of-view compound eye images.

[0004] The inventive concept of the present invention is: in view of the particularity of the imaging mechanism of a large-field-of-view bionic curved compound eye, the influence of different angles between the optical axis of each small eye and the central optical axis of the curved compound eye lens on the accuracy of image super-resolution reconstruction is analyzed. The present invention proposes a multi-perspective collaborative super-resolution reconstruction method based on a large-field-of-view compound eye image, and combines the structural parameters of the large-field-of-view curved compound eye imaging system to realize a single-frame super-resolution reconstruction process without movement. Utilizing a compound eye imaging model with a sub-pixel spatial mapping relationship, super-resolution reconstruction is performed based on pixel-space mapping addressing, opening up a new path for super-resolution imaging and real-time image processing in dynamic scenes.

[0005] In order to achieve the above-mentioned invention objectives and inventive concepts, the present invention provides the following technical solutions:

[0006] A multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images is special in that it includes the following steps:

[0007] S1, obtain the original compound eye image and compound eye imaging model;

[0008] The original compound eye image is obtained by a large-field-of-view curved compound eye imaging system including a plurality of ommatidia. The original compound eye image includes a plurality of sub-images with a resolution of m×n corresponding to the plurality of ommatidia. The compound eye imaging model is a mathematical mapping model of the curved compound eye space.

[0009] S2, pixel-space mapping addressing;

[0010] According to the compound eye imaging model, pixel-space mapping addressing is performed point by point, and each image point in the original compound eye image is associated with the corresponding ommatidium. Then, the associated image point of each ommatidium in the original compound eye image is converted to the object space to obtain the world coordinates of the corresponding object point. An ommatidium corresponding to each object point and the ommatidium in its neighborhood are selected as the corresponding cluster eye of the object point. The sub-image of the corresponding cluster eye is the corresponding image unit of the object point, which respectively contains the image point corresponding to the object point. The pixel grayscale value of the image point of each object point in each sub-image of its corresponding image unit is indexed;

[0011] S3, reconstructed super-resolution image;

[0012] S3.1, construct a super-resolution blank image with a resolution of M×N, where M×N satisfies:

[0013] M×N≥2m×2n;

[0014] S3.2, using the pixel grayscale values of the image points of each object point in each sub-image of the corresponding image unit, weighted calculation is performed to obtain the pixel grayscale value of the corresponding image point of each object point in the super-resolution blank image;

[0015] S3.3, calculating the pixel grayscale values of the remaining pixels in the super-resolution blank image using an interpolation method;

[0016] S3.4, filling the pixel grayscale values calculated in steps S3.2 and S3.3 into the super-resolution blank image to obtain a super-resolution image.

[0017] Furthermore, in step S1, the compound eye imaging model is specifically:

[0018]

[0019] in, K is the number of the ommatidia and their corresponding sub-images in the large field of view curved compound eye imaging system, u 、 v Image points In the K The horizontal and vertical coordinates of each ommatidium correspond to the sub-image. f is the focal length of the eye, 、 Respectively K The direction and elevation angle of the optical axis of the small eye, Image point The direction vector in world coordinates, For the K The coordinate component of each ommatidium perpendicular to the sub-image plane, x w 、 y w 、 z w Object points The coordinate values in the x, y, and z directions of the world coordinate system.

[0020] Furthermore, step S3.2 is specifically as follows:

[0021] The pixel grayscale value of the corresponding image point in the super-resolution blank image is calculated by the following formula: ;

[0022]

[0023]

[0024] in, k is the number of the ommatidia in the cluster and its corresponding sub-image, is the weight coefficient matrix of the pixel grayscale value of the object point in each sub-image of the corresponding image unit, is the angle matrix between the optical axes of the small eyes in the cluster and the central optical axis of the large field of view curved compound eye imaging system, is the angle matrix between the position of the object point in each sub-image of the corresponding image unit and the optical axis of the eye, a, b are the factors affecting the weight coefficients of the corresponding positions of the object point in the world coordinate system and the image point in the sub-image, is the pixel gray value matrix of the object point in each sub-image of the corresponding image unit, is the pixel gray value of the corresponding pixel in the super-resolution blank image, i 、 j are the horizontal and vertical coordinates of the image points in the super-resolution blank image.

[0025] Furthermore, in step S3.2, a =45~55, b =45~55.

[0026] Furthermore, in step S1, the large field of view curved compound eye imaging system includes a curved compound eye lens, an optical relay image module and a large array planar image sensor;

[0027] The curved surface compound eye lens comprises a plurality of small eyes arranged along a spherical surface; an image plane is provided between the curved surface compound eye lens and the optical relay image module;

[0028] The optical relay image transfer module is used to transmit the image on the image plane to the large-array planar image sensor to form a planar compound eye image as the original compound eye image.

[0029] Furthermore, in step S2, the plurality of ommatidia are arranged in a hexagonal honeycomb structure, and the corresponding cluster eye of the object point includes the ommatidia corresponding to the object point and adjacent ommatidia respectively arranged at six vertex positions of the hexagonal honeycomb structure.

[0030] Furthermore, step S1 also includes performing sub-pixel camera calibration on the large field of view curved compound eye imaging system to correct the principal point position of the curved compound eye lens.

[0031] Beneficial effects of the present invention:

[0032] 1. The present invention uses the cluster eye as a functional unit for super-resolution reconstruction, maximizing the utilization of the multi-view overlapping information of the curved compound eye imaging system in the super-resolution reconstruction process, achieving large field of view, high signal-to-noise ratio, and super-resolution imaging, and outputting high-quality super-resolution images. The image information can also be used to achieve depth measurement and 3D imaging.

[0033] 2. The present invention obtains the mapping relationship between the object point and the cluster eye associated image point through the compound eye imaging model, realizes the high-precision registration of multiple sub-images with different viewing angles and object points, and then uses the angle relationship between the optical axes of the small eyes and the object point At the image point position in the corresponding sub-image, the weight coefficients corresponding to the pixel grayscale values of different image points in the original compound eye image are calculated, and the pixel grayscale values of the super-resolution blank image are calculated more accurately to obtain the super-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the optical path of a large-field-of-view curved compound eye imaging system according to an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of an original compound eye image in an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of the principles of step 3 and step 4 in the embodiment of the present invention.

[0037] Description of reference numerals:

[0038] 1-Curved compound eye lens, 2-Optical relay image module, 3-Large array flat image sensor. DETAILED DESCRIPTION

[0039] The present invention is based on a multi-view collaborative super-resolution reconstruction method of a large field of view compound eye image and uses a large field of view curved surface compound eye imaging system to obtain a large field of view compound eye image. The design of a large field of view curved surface compound eye imaging system mainly involves two problems: one is to imitate organisms in nature and design the compound eye structure; the other is to realize a compound eye structure with high field of view overlap through the pupil matching principle and compound eye constraint parameters. Based on this, the present invention uses an existing large field of view curved surface compound eye imaging system, the structure of which is as follows Figure 1 As shown, it includes a curved compound eye lens 1, an optical relay image module 2 and a large array planar image sensor 3.

[0040] The curved compound eye lens 1 comprises a number of ommatidia arranged along a spherical surface with identical optical parameters. An image plane is positioned between the curved compound eye lens 1 and the optical relay image transfer module 2, concentrically arranged with the spherical surface where the ommatidia reside. The optical relay image transfer module 2 comprises 13 sequentially arranged lenses, which transmit the spherical image plane to a large-array planar image sensor 3. Unlike conventional imaging systems, the large-array planar image sensor 3 utilizes only a single flat detector to collect images from all ommatidia arranged along the spherical surface. The imaging process of the bionic curved compound eye imaging system is as follows: after incident light passes through the curved compound eye lens 1, it is imaged on an image plane concentric with the spherical surface, known as the spherical compound eye image plane. This spherical compound eye image plane is then transformed into a planar compound eye image after passing through the optical relay image transfer module 2. This image plane is then received by the large-array planar image sensor 3, producing the original compound eye image.

[0041] In this embodiment, multiple ommatidia are arranged in a hexagonal honeycomb structure, with each ommatidia located at the six vertex corners and the center of the hexagonal honeycomb structure. According to the ommatidia arrangement pattern, a target point in object space can be captured by at least seven ommatidia from different perspectives at the same time. This imaging mechanism provides a natural advantage for multi-perspective collaborative super-resolution reconstruction. The larger the field of view of a single ommatidium, the smaller the angle between the optical axes of adjacent ommatidia, the higher the overlap rate of the fields of view of adjacent ommatidia, and the greater the number of ommatidia with overlapping fields of view. Each ommatidium and the ommatidia within its hexagonal neighborhood are defined as a cluster of ommatidia. In this embodiment, the field of view angle of a single ommatidium is 20°, the angle between the optical axes of adjacent ommatidia is 9°, the total field of view angle of the system is 182°, and the field of view overlap rate of adjacent ommatidia can reach 55%. Each ommatidium and its six adjacent ommatidia constitute a cluster eye, and the seven ommatidia in the same cluster eye have overlapping fields of view at the same time. The seven ommatidia with overlapping fields of view, that is, a cluster eye, are used as a functional unit for super-resolution reconstruction. The seven sub-images corresponding to the cluster eye are used as an image unit to ensure that the same object point has seven image points of different perspectives in the original compound eye image, that is, there are seven different pixel positions.

[0042] Since the large field of view curved surface compound eye imaging system can provide observation information from more angles, it solves the uncertainty problem in the super-resolution reconstruction process and provides a new idea for super-resolution reconstruction. Based on the imaging process of the above-mentioned large field of view curved surface compound eye imaging system, and taking advantage of the fact that multiple small eyes can image the same target point from different perspectives, the present invention proposes a multi-perspective collaborative super-resolution reconstruction method based on large field of view compound eye images. First, the image points in the large field of view compound eye image are converted to the object space of the large field of view curved surface compound eye imaging system. According to the compound eye imaging model, the spatial mapping method is used to perform pixel-space mapping addressing to find the image points of the image units corresponding to each object point in the object space. The pixel grayscale value of each image point in the super-resolution blank image is calculated by constructing a pixel weighting function and interpolation method to obtain a super-resolution image. Specifically, the following steps are included:

[0043] Step 1: Obtain the original compound eye image and perform sub-pixel camera calibration for the large field of view curved compound eye imaging system to obtain camera calibration parameters, and correct the principal point position of the curved compound eye lens 1 using the camera calibration parameters.

[0044] like Figure 2 The original compound eye image is shown. This image is extremely complex and consists of multiple sub-images corresponding to multiple ommatidia, with high field of view overlap but no image aliasing. These sub-images are obtained by several ommatidia, and their distribution corresponds one-to-one with the arrangement of the ommatidia in this embodiment. The resolution is m×n.

[0045] Sub-pixel camera calibration uses a high-resolution target for calibration. Camera calibration parameters include but are not limited to ommatidium focal length, distortion parameters, translation matrix, and rotation matrix. Pixel coordinates of the original compound eye image The mapping relationship between them is as follows:

[0046]

[0047] in, p is the pixel size of the large array planar image sensor, f is the focal length of the eye, K is the number of the ommatidia and their corresponding sub-images in the large field of view curved compound eye imaging system, 、 For the K The horizontal and vertical coordinates of the pixel at the center of the sub-image are the internal parameters of the ommatidium. For the K The elevation angle of the optical center of each ommatidium in the world coordinate system is the external parameter of the ommatidium.

[0048] Step 2: Based on the camera calibration parameters, a spatial mapping method is used to complete the addressing process from any object point in space to the associated ommatidium pixel, thereby establishing a compound eye imaging model. The compound eye imaging model is specifically a mathematical mapping model of the surface compound eye space established based on the compound eye imaging process, thereby obtaining the sub-pixel spatial mapping relationship of "object space-compound eye imaging system-image space", that is, the world coordinate is ( x w , y w , z w ) Object Point The original compound eye image K The coordinates of the ommatidium image are ( u , v ) The mapping relationship between them is as follows:

[0049]

[0050] in, 、 Respectively K The direction and elevation angle of the optical axis of the small eye, Image point The direction vector in world coordinates, For the K The coordinate component of each ommatidium perpendicular to the sub-image plane.

[0051] Step 3: According to the compound eye imaging model, pixel-space mapping addressing is performed point by point, so that each image point in the original compound eye image is associated with the corresponding ommatidium. Then, the associated image point of each ommatidium in the original compound eye image is converted to the object space to obtain the world coordinates of the corresponding object point. One ommatidium corresponding to each object point and its six adjacent ommatidium are selected as the corresponding cluster eye of the object point, and the seven sub-images of the corresponding cluster eye are used as the corresponding image units of the object point. The seven sub-images respectively contain the image points corresponding to the object point, and the seven pixel grayscale values of the corresponding image points of each object point in each sub-image of its corresponding image unit are indexed.

[0052] Step 4: Construct a super-resolution blank image, calculate the pixel grayscale value of each pixel and fill it into the super-resolution blank image to obtain a super-resolution image. This specifically includes the following steps:

[0053] 4.1. Constructing a super-resolution blank image. The resolution of the super-resolution blank image is the same as the resolution of the super-resolution image, denoted as M×N, and meets the following requirements:

[0054] M×N≥2m×2n.

[0055] 4.2. Using the 7 pixel grayscale values in the image unit corresponding to each object point, adaptively weighted by the pixel weighting function, the pixel grayscale value of the corresponding image point in the super-resolution blank image is calculated. ; The details are as follows:

[0056]

[0057]

[0058] in, k is the number of the ommatidia in the cluster and its corresponding sub-image, k ={1,2,...,7}, is the weight coefficient matrix of the pixel grayscale value of the object point in each sub-image corresponding to the image unit, is the angle matrix between the optical axes of the small eyes in the cluster and the central optical axis of the large field of view curved compound eye imaging system, is the angle matrix between the position of the object point in each sub-image of the corresponding image unit and the optical axis of the eye, a, b are the factors affecting the weight coefficients of the corresponding positions of the object point in the world coordinate system and the image point in the sub-image, a =45~55, b =45~55, in this embodiment, a = b =50. I kuv is the pixel gray value matrix of the object point in each sub-image of the corresponding image unit, is the pixel gray value of the corresponding pixel in the super-resolution blank image, i 、 j are the horizontal and vertical coordinates of the image points in the super-resolution blank image.

[0059] For large field of view curved compound eye imaging systems, the closer to the central optical axis, the higher the spatial resolution of the ommatidia. The smaller the value, the higher the spatial resolution of the corresponding image point; for a single ommatidium, the closer the image point is to the optical axis of the ommatidium, The smaller the value, the higher the spatial resolution of the corresponding image point.

[0060] 4.3. Use the interpolation method to calculate the pixel grayscale values between the image points of the super-resolution blank image corresponding to adjacent object points.

[0061] 4.4. Fill all pixel grayscale values calculated in steps 4.2 and 4.3 into the super-resolution blank image to obtain a super-resolution image.

[0062] like Figure 3 As shown, M×N=2m×2n, with image points and For example, after step 3, the corresponding object points can be obtained and The world coordinates of the object point are calculated in step 4.2. and Pixel grayscale value and Then, through step 4.3, the object point is calculated using the interpolation method and Pixel grayscale values between corresponding pixels , and fill them into the pixel coordinates (1, 0), (2, 0) and (3, 0) of the super-resolution blank image respectively.

[0063] Because the object point has a natural sub-pixel displacement between different ommatidia when imaging with a large field of view curved compound eye imaging system, and this perspective difference can bring richer observation information. and Multiple viewing angle differences, that is, the object point gets 7 different pixel positions after passing through a cluster eye. Multiple pixel grayscale values can be calculated for each object point in an image unit, and then the adjacent object points can be calculated by interpolation. and Multiple pixel grayscale values between corresponding image points to accommodate larger super-resolution images.

Claims

1. A multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images, characterized in that: The following steps are involved: S1, obtain the original compound eye image and compound eye imaging model; The original compound eye image is obtained by a large-field-of-view curved compound eye imaging system including multiple ommatidia. The original compound eye image includes multiple sub-images with a resolution of m×n corresponding to the multiple ommatidia. The compound eye imaging model is a mathematical mapping model of the curved compound eye space, specifically: Among them, K is the number of the ommatidia and its corresponding sub-image of the large field of view curved compound eye imaging system, u and v are the numbers of the image point P respectively. uv ′ is the horizontal and vertical coordinates of the sub-image corresponding to the Kth ommatidium, f is the focal length of the ommatidium, θ K are the direction angle and elevation angle of the optical axis of the K-th small eye, is the image point P uv ′ is the direction vector in the world coordinate system, is the coordinate component of the Kth ommatidium perpendicular to the sub-image plane, x w 、y w 、z w They are object points P uv Coordinate values in the x, y, and z directions of the world coordinate system; S2, pixel-space mapping addressing; According to the compound eye imaging model, pixel-space mapping addressing is performed point by point, and each image point in the original compound eye image is associated with the corresponding ommatidium. Then, the associated image point of each ommatidium in the original compound eye image is converted to the object space to obtain the world coordinates of the corresponding object point. An ommatidium corresponding to each object point and the ommatidium in its neighborhood are selected as the corresponding cluster eye of the object point. The sub-image of the corresponding cluster eye is the corresponding image unit of the object point, which respectively contains the image point corresponding to the object point. The pixel grayscale value of the image point of each object point in each sub-image of its corresponding image unit is indexed; S3, reconstructed super-resolution image; S3.1, construct a super-resolution blank image with a resolution of M×N, where M×N satisfies: M×N≥2m×2n; S3.2, using the pixel grayscale value of each object point in each sub-image of the corresponding image unit, the pixel grayscale value of each object point in the super-resolution blank image corresponding to the pixel grayscale value I is obtained by weighted calculation using the following formula: ij ; I ij =I kuv ·Ω k Where k is the number of the ommatidia in the cluster and its corresponding sub-image, Ω k is the weight coefficient matrix of the pixel grayscale value of the object point in each sub-image of the corresponding image unit, is the angle matrix between the optical axes of the small eyes in the cluster and the central optical axis of the large field of view curved compound eye imaging system, is the angle matrix between the position of the object point in each sub-image of the corresponding image unit and the optical axis of the eye, a and b are the factors affecting the weight coefficient of the corresponding position of the object point in the world coordinate system and the image point in the sub-image, respectively. kuv is the pixel gray value matrix of the object point in each sub-image of the corresponding image unit, I ij is the pixel grayscale value of the corresponding pixel in the super-resolution blank image, i and j are the horizontal and vertical coordinates of the pixel in the super-resolution blank image respectively; S3.3, calculating the pixel grayscale values of the remaining pixels in the super-resolution blank image using an interpolation method; S3.4, filling the pixel grayscale values calculated in steps S3.2 and S3.3 into the super-resolution blank image to obtain a super-resolution image.

2. The multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images according to claim 1, characterized in that: In step S3.2, a=45~55, b=45~55.

3. The multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images according to claim 1 or 2, characterized in that: In step S1, the large-field-of-view curved compound eye imaging system comprises a curved compound eye lens (1), an optical relay image module (2), and a large-array planar image sensor (3); The curved surface compound eye lens (1) comprises a plurality of small eyes arranged along a spherical surface; an image plane is provided between the curved surface compound eye lens (1) and the optical relay image transfer module (2); The optical relay image transfer module (2) is used to transmit the image on the image plane to the large-array planar image sensor (3) to form a planar compound eye image as an original compound eye image.

4. The multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images according to claim 3, characterized in that: In step S2, the plurality of ommatidia are arranged in a hexagonal honeycomb structure, and the cluster eye corresponding to the object point includes the ommatidia corresponding to the object point and adjacent ommatidia respectively arranged at the six vertex positions of the hexagonal honeycomb structure.

5. The multi-view collaborative super-resolution reconstruction method based on large-field-of-view compound eye images according to claim 4, characterized in that: Step S1 also includes performing sub-pixel camera calibration on the large-field-of-view curved compound eye imaging system to correct the principal point position of the curved compound eye lens (1).

Images