Large dynamic imaging device and method based on spectral polarization and storage medium

Abstract

The embodiment of the invention discloses a large dynamic imaging device and method based on spectral polarization and a storage medium, and the device comprises a beam shrinkage imaging optical system which is used for converging a target region and outputting target light in a parallel light form; the regional spectral polarization imaging microlens array is divided into an upper half region and a lower half region, the upper half region is used for performing multi-aperture imaging and polarization multispectral modulation on target light to obtain a spectral polarization image, and the lower half region is used for directly imaging the target light to obtain an original light image; the target surface of the CMOS detector corresponds to the upper half region and the lower half region of the regional spectral polarization imaging microlens array, and the CMOS detector is used for acquiring bright part detail information and dark part detail information of a target region according to the spectral polarization image and the original light image in an exposure period; and the super-resolution image processor is used for performing gradient mapping and feature extraction on the spectral polarization image to generate a super-resolution image. The large dynamic imaging device based on spectral polarization has a large dynamic range and an anti-backlight effect.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral polarization imaging, and in particular to a large dynamic imaging device, method and storage medium based on spectral polarization. Background Art

[0002] In backlit scenes, due to the limitation of dynamic range and the influence of strong light, the camera is prone to overexposure, resulting in the loss of target information. By using spectral and polarization information, the impact of strong light can be effectively reduced. By combining multispectral and polarization imaging methods, rich four-dimensional information of the target in the spectral, polarization, and spatial fields can be obtained, and various properties of the target can be quickly analyzed.

[0003] The existing snapshot spectral polarization imaging methods widely use optical elements such as filters and polarizers. The imaging process is often accompanied by multiple divisions of light energy, which helps to suppress the light intensity of the bright area of ​​the scene so that the bright information will not cause the camera to overexpose and lose detailed texture information. However, it also suppresses the light information intensity of the dark area of ​​the scene, making it impossible for the camera to detect and cause information loss. The common snapshot imaging methods such as dispersion spectral imaging and filter array spectral polarization imaging still have the problem of low spatial resolution. The spectral imaging method based on single pixel has a long reconstruction time and cannot meet the scenes with high real-time requirements. Summary of the invention

[0004] Based on this, it is necessary to propose a large dynamic imaging device, method and storage medium based on spectral polarization to address the above problems.

[0005] A large dynamic imaging device based on spectral polarization, the device comprises a beam reduction imaging optical system, a sub-regional spectral polarization imaging microlens array, a CMOS detector, and a super-resolution image processor:

[0006] The beam reduction imaging optical system is used to converge the target area and output the target light in the form of parallel light.

[0007] The regional spectral polarization imaging microlens array is divided into an upper half area and a lower half area, the upper half area is used to perform multi-aperture imaging and polarization multi-spectral modulation on the target light to obtain a spectral polarization image, and the lower half area is used to directly image the target light to obtain an original light image.

[0008] The CMOS detector, whose target surface corresponds to the upper and lower regions of the sub-region spectral polarization imaging microlens array, is used to obtain bright detail information and dark detail information of the target area according to the spectral polarization image and the original light image within one exposure cycle.

[0009] The super-resolution image processor is used to perform gradient mapping and feature extraction on the spectral polarization image to generate a super-resolution image.

[0010] The beam reduction imaging optical system specifically comprises: an imaging lens group and a beam reduction lens group, wherein the imaging lens group is used to perform convergent imaging on the target area and obtain convergent light; the beam reduction lens group is used to compress the convergent light and output the target light in the form of parallel light.

[0011] Among them, the upper half area specifically includes a regional microlens array and a regional spectral polarization array film. The regional microlens array is used to perform multi-aperture imaging of the target light to obtain a low-resolution image. The regional spectral polarization array film is closely attached to the upper half area and is used to perform polarization multi-spectral modulation on the low-resolution image to obtain a spectral polarization image.

[0012] The array size of the sub-regional microlens array is 2M×2N, wherein M is the number of micromirror units in the two-dimensional array in the horizontal direction, and N is the number of micromirror units in the two-dimensional array in the vertical direction.

[0013] Among them, the sub-regional microlens array includes a plurality of micromirror units, each of which is composed of adjacent 2×2 micromirrors and is used to obtain low-resolution images of M×N spectral bands. Behind each of the micromirror units corresponds a spectral polarization array sub-region of the sub-regional spectral polarization array film, and each of the spectral polarization array sub-region is divided into four quadrants, which are coated with four polarization films in preset linear polarization directions in sequence from the first quadrant to the fourth quadrant, so as to perform polarization multi-spectral modulation on the low-resolution image and obtain four spectral polarization images under the spectral band.

[0014] The lower half area is composed of a complete micromirror, which is used to directly image the target light and obtain the original light image.

[0015] The super-resolution image processor includes a super-resolution branch, a gradient branch and a super-resolution main branch.

[0016] The super-resolution branch is used to extract the super-resolution structural features of the spectral polarization image, connect the spectral polarization image to the super-resolution structural features, obtain the super-resolution structural feature image, and map the super-resolution structural features to the gradient branch as gradient branch prior information.

[0017] The gradient branch is used to extract gradient features based on the spectral polarization image and the super-resolution structural features to obtain an image gradient image.

[0018] The super-resolution main branch is used to obtain a super-resolution image based on the super-resolution structural feature image and the image gradient image.

[0019] A large dynamic imaging method based on spectral polarization, comprising:

[0020] The target area is converged and the target light is output in the form of parallel light.

[0021] The target light is subjected to multi-aperture imaging and polarization multi-spectral modulation to obtain a spectral polarization image, and the target light is directly imaged to obtain an original light image.

[0022] The bright part detail information and the dark part detail information of the target area are acquired according to the spectral polarization image and the original light image in one exposure cycle.

[0023] The spectral polarization image is subjected to gradient mapping and feature extraction to generate a super-resolution image.

[0024] The step of performing gradient mapping and feature extraction on the spectral polarization image to generate a super-resolution image specifically includes:

[0025] The gradient mapping image is obtained according to the difference between adjacent pixels in the spectral polarization image.

[0026] The super-resolution structural features of the spectral polarization image are extracted, and the spectral polarization image is connected to the super-resolution structural features to obtain a super-resolution feature image. Meanwhile, the super-resolution structural features are mapped to the gradient branch as gradient branch prior information.

[0027] The gradient features of the gradient mapping image are extracted according to the super-resolution structural features, and the gradient mapping image is connected to the gradient features to obtain a gradient feature image.

[0028] The super-resolution feature image and the gradient feature image are feature-connected to obtain a feature stitching image.

[0029] Extracting feature long-range dependencies in the feature stitched image, and generating a super-resolution image based on the feature long-range dependencies.

[0030] A storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method described above.

[0031] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0032] The embodiments of the present invention have the following beneficial effects:

[0033] The present invention uses a sub-region spectral polarization imaging microlens array to regulate light field information. The system has a simple structure and a large aperture. It can obtain the original light field information and spectral polarization information within one exposure cycle, and has strong real-time performance. Specifically, the upper half of the sub-region spectral polarization imaging microlens array performs polarization spectrum regulation, and performs multi-aperture imaging and polarization multi-spectral modulation on the target light, so that the system can effectively suppress the light energy in the highlight area of ​​the scene and obtain a spectral polarization image; the lower half retains the original light field information, and at the same time combines with a super-resolution algorithm to obtain dark and bright details of the target scene and a high-resolution image of the target scene. The large dynamic imaging device based on spectral polarization has a large dynamic range and anti-backlight effect, which improves the perception ability of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] in:

[0036] Figure 1 A schematic structural diagram of an embodiment of a large dynamic imaging device based on spectral polarization provided by the present invention;

[0037] Figure 2 A schematic structural diagram of an embodiment of a beam reduction imaging optical system provided by the present invention;

[0038] Figure 3 A schematic structural diagram of an embodiment of a large dynamic imaging device based on spectral polarization provided by the present invention;

[0039] Figure 4 A schematic diagram of the structure of an embodiment of a super-resolution image processor provided by the present invention;

[0040] Figure 5 A schematic diagram of a flow chart of an embodiment of a large dynamic imaging method based on spectral polarization provided by the present invention;

[0041] Figure 6 A schematic flow chart of another embodiment of a large dynamic imaging method based on spectral polarization provided by the present invention;

[0042] Figure 7 A schematic diagram of the structure of an embodiment of a nested residual dense block provided by the present invention;

[0043] Figure 8 A schematic diagram of the structure of an embodiment of a residual visual Mamba block provided by the present invention;

[0044] Fig. 9 A schematic structural diagram of an embodiment of a storage medium provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, Figure 1 A schematic diagram of the structure of an embodiment of a large dynamic imaging device based on spectral polarization provided by the present invention. A large dynamic imaging device based on spectral polarization 10, the device includes a beam reduction imaging optical system 1, a sub-regional spectral polarization imaging microlens array 2, a CMOS detector 3, and a super-resolution image processor 4:

[0047] The beam reduction imaging optical system 1 is used to converge the target area and output the target light in the form of parallel light.

[0048] For example, referring to Figure 2 , Figure 2 The present invention is a schematic structural diagram of an embodiment of a beam reduction imaging optical system provided by the present invention. The beam reduction imaging optical system 1 comprises an imaging lens group 11 and a beam reduction lens group 12. The imaging lens group 11 performs convergent imaging on a target scene within a large aperture field of view, adopts a large aperture to ensure that the system has a high luminous flux, and according to the principle of refraction and reflection of light, converges light from the target area onto an imaging plane to output convergent light; the beam reduction lens group 12 compresses the convergent light output by the imaging lens group 11, and outputs the target light in the form of parallel light to match the size of the subsequent regional spectral polarization imaging microlens array and its imaging requirements for incident parallel light.

[0049] It should be noted that both the imaging lens group and the beam reduction lens group use a combination of positive and negative lenses to reduce the aberration of the system. Preferably, the lens is made of flint glass to reduce the aberration.

[0050] The regional spectral polarization imaging microlens array 2 is divided into an upper region and a lower region. The upper region is used for multi-aperture imaging and polarization multi-spectral modulation of the target light to obtain a spectral polarization image, and the lower region is used for directly imaging the target light to obtain an original light image.

[0051] For example, referring to Figure 3 , Figure 3This is a schematic diagram of the structure of an embodiment of a large dynamic imaging device based on spectral polarization provided by the present invention, wherein the regional spectral polarization imaging microlens array 2 is divided into an upper half region and a lower half region. Specifically, the upper half region specifically includes a regional microlens array 21 and a regional spectral polarization array film 22. The regional microlens array 21 performs multi-aperture imaging on the target light output through the beam reduction lens group 12, and divides the incident target light into multiple sub-beams, each of which is focused and imaged by a microlens. Since the size of the microlens is very small, a low-resolution image is output. After the regional spectral polarization array film is closely attached to the upper half region, the low-resolution image output by the regional microlens array 21 is subjected to polarization multi-spectral modulation to obtain a spectral polarization image.

[0052] The array size of the sub-regional microlens array is 2M×2N, where M is the number of micromirrors in the horizontal direction of the two-dimensional array, and N is the number of micromirrors in the vertical direction of the two-dimensional array. The sub-regional microlens array includes a number of micro-transmission units, each of which is composed of adjacent 2×2 micromirrors. Each micro-transmission unit contains the same spectral band film, and low-resolution images of M×N spectral bands are obtained through the sub-regional microlens array. Each micro-transmission unit corresponds to a spectral polarization array sub-region of a sub-regional spectral polarization array film. Each spectral polarization array sub-region is divided into four quadrants. Polarization films of four preset linear polarization directions are sequentially coated from the first quadrant to the fourth quadrant, and polarization multi-spectral modulation is performed on the low-resolution image to obtain four spectral polarization images under the spectral band, and finally four-directional polarization images of M×N spectral bands can be obtained.

[0053] It should be noted that polarizing films with linear polarization directions of 0°, 45°, 90°, and 135° are coated in sequence from the first quadrant to the fourth quadrant.

[0054] The lower area is composed of complete micromirrors without polarization filtering, which directly performs high-resolution and high-light flux imaging on the target light to obtain the original light image.

[0055] The CMOS detector 3 is used to obtain the bright part detail information and the dark part detail information of the target area according to the spectral polarization image and the original light image within one exposure cycle.

[0056] Exemplarily, the CMOS detector 3 precisely corresponds to the regional spectral polarization imaging microlens array 2, the upper half of the target surface corresponds to the upper half of the regional spectral polarization imaging microlens array 2, which is divided into 2M×2N small target surfaces, and the bright detail information of the target area is obtained according to the spectral polarization image within one exposure cycle, and the lower half of the target surface corresponds to the lower half of the regional spectral polarization imaging microlens array 2, and the dark detail information of the target area is obtained according to the original light image within one exposure cycle.

[0057] The super-resolution image processor 4 is used to perform gradient mapping and feature extraction on the spectral polarization image to generate a super-resolution image.

[0058] Exemplarily, the super-resolution image processor 4 includes: a super-resolution branch, a gradient branch and a super-resolution main branch. The super-resolution branch is used to extract the super-resolution structural features of the spectral polarization image, obtain the super-resolution structural feature image, and the super-resolution structural features are mapped to the gradient branch as the gradient branch prior information; the gradient branch is used to extract the image gradient features based on the spectral polarization image with the super-resolution structural features as the prior, and obtain the image gradient image; the super-resolution main branch is used to obtain the super-resolution image according to the super-resolution structural feature image and the image gradient image.

[0059] Specifically, Figure 4 As stated, Figure 4 The super-resolution image processor 4 includes a super-resolution branch 41, a gradient branch 42 and a Mamba super-resolution main branch 43. First, the gradient mapping image is obtained according to the difference between adjacent pixels in the spectral polarization image. Spectral polarization image The image is input to the super-resolution branch 41 for super-resolution reconstruction and mapping the feature map to the gradient branch. Specifically, the super-resolution branch 41 includes three super-resolution feature extraction modules, which extract the spectral polarization image step by step. The structural features of Connect to the output of the third super-resolution feature extraction module. Further, the gradient map image The input gradient branch 42 is used to extract gradient features by three gradient feature extraction modules corresponding to the three super-resolution feature extraction modules, and connected by a self-attention mechanism layer to extract image detail texture information to guide the image reconstruction process of the super-resolution branch. Finally, in the Mamba super-resolution main branch 43, the super-resolution feature image F generated by the super-resolution branch 41 is ISR And the gradient feature image generated by the gradient branch Perform feature connection and input into Mamba super-resolution main branch to generate the final super-resolution image

[0060] From the above description, it can be seen that the present invention uses a regional spectral polarization imaging microlens array to regulate the light field information. The system has a simple structure and a large aperture. It can obtain the original light field information and spectral polarization information within one exposure cycle, and has strong real-time performance. Specifically, the upper half of the regional spectral polarization imaging microlens array performs polarization spectrum regulation, and performs multi-aperture imaging and polarization multi-spectral modulation on the target light, so that the system can effectively suppress the light energy in the highlight area of ​​the scene and obtain a spectral polarization image; the lower half retains the original light field information, and at the same time combines with a super-resolution algorithm to obtain dark and bright details of the target scene and a higher-resolution image of the target scene. The large dynamic imaging device based on spectral polarization has a large dynamic range and anti-backlight effect, which improves the perception ability of the device.

[0061] like Figure 5 As shown, Figure 5 A schematic diagram of a flow chart of an embodiment of a large dynamic imaging method based on spectral polarization provided by the present invention. A large dynamic imaging method based on spectral polarization, the method comprising:

[0062] S101: Converging the target area and outputting the target light in the form of parallel light.

[0063] Exemplarily, the imaging lens group 11 is used to perform convergent imaging on a target scene within a large aperture field of view, and output the converged light as parallel light through the beam reduction lens group 12, thereby ensuring that a clear and uniform image is formed on the regional spectral polarization imaging microlens array 2, thereby meeting the input light requirement for full field of view overlap between apertures of the regional spectral polarization imaging microlens array 2.

[0064] S102: Perform multi-aperture imaging and polarization multi-spectral modulation on the target light to obtain a spectral polarization image, and directly image the target light to obtain an original light image.

[0065] Exemplarily, the upper half of the sub-regional spectral polarization imaging microlens array 2 uses multi-aperture imaging technology to divide the target light into multiple sub-beams, each of which is imaged through a microlens to obtain a low-resolution image. After each micro-transmitter unit, the sub-regional spectral polarization array film 22 divides the incident light into different spectral bands, and applies a preset polarization film to each band for polarization splitting, and finally obtains a spectral polarization image containing spectral and polarization information. The lower half of the sub-regional spectral polarization imaging microlens array 2 directly images the target light without spectral or polarization analysis, and directly captures the original light image of the target area.

[0066] Specifically, the upper half area specifically includes a sub-regional microlens array 21 and a sub-regional spectral polarization array film 22. The sub-regional microlens array 21 is used to perform multi-aperture imaging of the target light to obtain a low-resolution image. The sub-regional spectral polarization array film 22 is closely attached to the upper half area and is used to perform polarization multi-spectral modulation on the low-resolution image to obtain a spectral polarization image. Among them, the array size of the sub-regional microlens array 21 is 2M×2N, wherein M is the number of micromirror units in the horizontal direction of the two-dimensional array, and N is the number of micromirror units in the vertical direction of the two-dimensional array. The sub-regional microlens array 21 includes a plurality of micromirror units, each of which is composed of adjacent 2×2 micromirrors, and obtains low-resolution images of M×N spectral bands. Each micromirror unit corresponds to a spectral polarization array sub-region of the sub-regional spectral polarization array film 22. Each spectral polarization array sub-region is divided into four quadrants, and four polarization films with preset linear polarization directions are sequentially coated from the first quadrant to the fourth quadrant, which is used to perform polarization multi-spectral modulation on the low-resolution image to obtain four spectral polarization images under the spectral band.

[0067] S103: Acquire bright part detail information and dark part detail information of the target area according to the spectral polarization image and the original light image within one exposure cycle.

[0068] For example, the target surface of the CMOS detector 3 corresponds to the upper and lower regions of the sub-region spectral polarization imaging microlens array 2, and simultaneously receives the spectral polarization image of the upper region and the original light image of the lower region. By combining the information of the two images, the bright details of the target region from the spectral polarization image and the dark details from the original light image can be obtained simultaneously in one exposure cycle.

[0069] S104: Perform gradient mapping on the spectral polarization image and the original light image to generate a super-resolution image.

[0070] Exemplarily, the super-resolution image processor 4 receives the spectral polarization image from the CMOS detector 3. The gradient mapping image thereof is obtained according to the difference between adjacent pixels in the spectral polarization image; the super-resolution structural features of the spectral polarization image are extracted by nesting the residual dense blocks, the spectral polarization image is connected to the super-resolution structural features by skip connection, the super-resolution feature image is obtained, and the super-resolution structural features are mapped to the gradient branch as the prior information of the gradient branch 42; the gradient features of the gradient mapping image are extracted according to the prior information of the gradient branch, the gradient mapping image is connected to the gradient features by skip connection, and the gradient feature image is obtained; the super-resolution feature image and the gradient feature image are feature-connected to obtain a feature splicing image; the feature long-range dependency relationship in the feature splicing image is extracted by the residual visual Mamba block, and the super-resolution image is generated based on the feature long-range dependency relationship.

[0071] It can be seen from the above description that the present invention obtains a spectral polarization image by performing multi-aperture imaging and polarization multi-spectral modulation on the target light. Multi-aperture imaging can capture image information from different perspectives, thereby providing richer spatial information and enhancing stereoscopic and depth perception. Polarization multi-spectral modulation can separate light of different wavelengths and polarization states, providing rich spectral and polarization information for subsequent image processing and analysis. Furthermore, by combining the information of the spectral polarization image and the original light image, the bright and dark detail information of the target area is simultaneously obtained within one exposure cycle. This is particularly important for imaging under low-light conditions, and taking into account the details of both bright and dark parts at the same time helps to improve the dynamic range of imaging and the anti-backlight effect. Furthermore, combined with gradient mapping and feature extraction technology, super-resolution images can be generated, and the detail expression of the image can be enhanced by improving the resolution of the image, providing users with clearer and more detailed image information.

[0072] like Figure 6 As shown, Figure 6 A schematic diagram of another embodiment of a large dynamic imaging method based on spectral polarization provided by the present invention. A large dynamic imaging method based on spectral polarization, the method comprising:

[0073] S201: Converging the target area and outputting the target light in the form of parallel light.

[0074] S202: Perform multi-aperture imaging and polarization multi-spectral modulation on the target light to obtain a spectral polarization image, and directly image the target light to obtain an original light image.

[0075] S203: Acquire bright part detail information and dark part detail information of the target area according to the spectral polarization image and the original light image within one exposure cycle.

[0076] It should be noted that steps S201-S203 are Figure 5 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0077] S204: Obtaining a gradient mapping image according to the difference between adjacent pixels in the spectral polarization image.

[0078] For example, according to the spectral polarization image Get its gradient map Among them, λ i p k Represents the spectral polarization image of the kth polarization direction of the i-th spectrum segment. Specifically, the gradient image I is obtained by the difference between adjacent pixels. G , as shown below:

[0079] I G={[I(x+Δx,y)-I(x-Δx,y)],[I(x,y+Δy)-I(x,y-Δy)]}

[0080] Among them, I(x+Δx,y), I(x-Δx,y), I(x,y+Δy), and I(x,y-Δy) represent the intensity values ​​of the neighboring pixels (x+Δx,y), (x-Δx,y), (x,y+Δy), and (x,y-Δy) centered at the pixel (x,y). Usually, Δx and Δy can be 1. Taking the 2-norm of the gradient map can obtain the gradient mapping of the spectral polarization image, that is, the gradient mapping image As shown below:

[0081]

[0082] S205: extracting super-resolution structural features of the spectral polarization image, connecting the spectral polarization image to the super-resolution structural features, obtaining a super-resolution feature image, and mapping the super-resolution structural features to the gradient branch as gradient branch prior information.

[0083] For example, referring to Figure 4 , the spectral polarization image The image is input to the super-resolution branch 41 for super-resolution reconstruction and mapping the feature map to the gradient branch 42. Specifically, the super-resolution branch 41 includes three super-resolution feature extraction modules, which extract the spectral polarization image step by step. The super-resolution structural features of the image are transformed into the input spectral polarization image through skip connections. Connected to the output of the third super-resolution feature extraction module. The super-resolution structural feature extraction process is shown in the following formula:

[0084]

[0085] in, is a spectral polarization image, SR1 is the structural feature extraction operation after the first super-resolution feature extraction module, SR2 is the structural feature extraction operation after the second super-resolution feature extraction module, SR3 is the structural feature extraction operation after the third super-resolution feature extraction module, and F SR It is the feature map after three super-resolution feature extraction modules.

[0086]

[0087] Among them, F ISR It is the super-resolution feature image finally output by the super-resolution branch.

[0088] It should be noted that the super-resolution feature extraction module can be composed of any advanced feature extraction network, and no specific limitation is made here.

[0089] Preferably, the present invention adopts nested residual dense blocks, such as Figure 7 As shown, Figure 7 A schematic diagram of the structure of an embodiment of a nested residual dense block provided by the present invention. Each nested residual dense block consists of three dense blocks, each dense block consists of five layers of 3×3 convolutional layers, and uses LReLU as a nonlinear activation function. At the same time, each dense block maps the extracted super-resolution structural features to the gradient branch as the prior information of the gradient branch to improve the performance of the gradient branch and reduce the number of parameters.

[0090] S206: extracting gradient features of the gradient mapping image according to the super-resolution structural features, connecting the gradient mapping image to the gradient features, and obtaining a gradient feature image.

[0091] For example, referring to Figure 4 , the gradient map image The gradient branch 42 is input, and the gradient feature extraction module performs gradient feature extraction, and is connected by the self-attention mechanism layer to extract image detail texture information to guide the image reconstruction process of the super-resolution branch 41.

[0092] Specifically, the gradient branch includes three gradient feature extraction modules, each of which can use any advanced feature extraction block. Preferably, the present invention uses a nested residual dense block and uses the prior information of the gradient branch 42 obtained by the super-resolution branch 41. The gradient is mapped to the image through a jump connection. Connected to the output of the third gradient feature extraction module, the specific process is shown in the following formula:

[0093]

[0094] Wherein, DR1 is the gradient feature extraction operation of the first gradient feature extraction module using the super-resolution structural features extracted by the first super-resolution feature extraction module as prior information, DR2 is the gradient feature extraction operation of the second gradient feature extraction module using the super-resolution structural features extracted by the second super-resolution feature extraction module as prior information, and DR3 is the gradient feature extraction operation of the third gradient feature extraction module using the super-resolution structural features extracted by the third super-resolution feature extraction module as prior information. SR is the gradient feature map after passing through three gradient feature extraction modules. It is the gradient feature map finally output by the gradient branch.

[0095] S207: Feature-connecting the super-resolution feature image and the gradient feature image to obtain a feature spliced ​​image.

[0096] S208: extracting long-range dependencies of features in the feature stitching image, and generating a super-resolution image based on the long-range dependencies of the features.

[0097] For example, referring to Figure 4 , the Mamba super-resolution main branch 43 consists of two residual visual Mamba blocks and a reconstruction block to achieve more efficient global feature acquisition and high-quality super-resolution reconstruction of the image. Figure 8 As shown, Figure 8 A schematic diagram of the structure of an embodiment of the residual visual Mamba block provided by the present invention. Each residual visual Mamba block performs residual combination of the residual visual Mamba block and the multi-layer perceptron (MPL), and adds layer normalization before the visual Mamba block and the multi-layer perceptron to effectively capture the long-term spatial dependencies and extract the feature long-range dependencies in the feature spliced ​​image. The reconstruction block reconstructs the high-resolution image using the deep features and feature long-range dependencies of the feature spliced ​​image. The specific process is shown in the following formula:

[0098]

[0099] Among them, VM1 represents the long-range dependency capture of features after the first residual visual Mamba block, VM2 represents the long-range dependency capture of features after the second residual visual Mamba block, Concat represents the connection operation of feature channels, and I F is the output of two residual visual Mamba blocks, is the final reconstructed super-resolution image.

[0100] like Fig. 9 As shown, Fig. 9 The storage medium 50 stores at least one computer program 51, which is executed by a processor 52 to implement the following. Figure 5 and Figure 6 In one embodiment, the storage medium 50 may be a storage chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server, etc.

[0101] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0103] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0104] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0105] For the convenience of description, the above device is described by being divided into various units according to their functions and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware. It should be understood by those skilled in the art that this specification embodiment can be provided as a method, system, or computer program product. Therefore, this specification embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0110] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0111] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0113] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0114] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0115] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A large dynamic imaging device based on spectral polarization, characterized in that: The device comprises a beam reduction imaging optical system, a sub-region spectral polarization imaging microlens array, a CMOS detector, and a super-resolution image processor: The beam reduction imaging optical system is used to converge the target area and output the target light in the form of parallel light; The sub-region spectral polarization imaging microlens array is divided into an upper half region and a lower half region, the upper half region is used to perform multi-aperture imaging and polarization multi-spectral modulation on the target light to obtain a spectral polarization image, and the lower half region is used to directly image the target light to obtain an original light image; The CMOS detector, whose target surface corresponds to the upper half area and the lower half area of ​​the sub-region spectral polarization imaging microlens array, is used to obtain the bright part detail information and the dark part detail information of the target area according to the spectral polarization image and the original light image within one exposure cycle; The super-resolution image processor is used to perform gradient mapping and feature extraction on the spectral polarization image to generate a super-resolution image.

2. The large dynamic imaging device based on spectral polarization according to claim 1, characterized in that: The beam reduction imaging optical system specifically includes: an imaging lens group and a beam reduction lens group, wherein the imaging lens group is used to perform convergent imaging on the target area to obtain convergent light; the beam reduction lens group is used to compress the convergent light and output the target light in the form of parallel light.

3. The large dynamic imaging device based on spectral polarization according to claim 2, characterized in that: The upper region specifically includes a regional microlens array and a regional spectral polarization array film. The regional microlens array is used to perform multi-aperture imaging on the target light to obtain a low-resolution image. The regional spectral polarization array film is closely attached to the upper region and is used to perform polarization multi-spectral modulation on the low-resolution image to obtain a spectral polarization image.

4. The large dynamic imaging device based on spectral polarization according to claim 3, characterized in that: The array size of the sub-regional microlens array is 2M×2N, wherein M is the number of micromirror units in the two-dimensional array in the horizontal direction, and N is the number of micromirror units in the two-dimensional array in the vertical direction.

5. The large dynamic imaging device based on spectral polarization according to claim 4, characterized in that: The sub-regional microlens array includes a plurality of micromirror units, each of which is composed of adjacent 2×2 micromirrors and is used to obtain low-resolution images of M×N spectral bands. Behind each of the micromirror units corresponds a spectral polarization array sub-region of the sub-regional spectral polarization array film, and each of the spectral polarization array sub-regions is divided into four quadrants, which are coated with four polarization films in preset linear polarization directions in sequence from the first quadrant to the fourth quadrant, so as to perform polarization multi-spectral modulation on the low-resolution image and obtain four spectral polarization images under the spectral band.

6. The large dynamic imaging device based on spectral polarization according to any one of claims 1 to 5, characterized in that: The lower half area is composed of a complete micromirror, which is used to directly image the target light and obtain an original light image.

7. The large dynamic imaging device based on spectral polarization according to claim 6, characterized in that: The super-resolution image processor includes: a super-resolution branch, a gradient branch and a super-resolution main branch. The super-resolution branch is used to extract the super-resolution structural features of the spectral polarization image, connect the spectral polarization image to the super-resolution structural features, obtain the super-resolution structural feature image, and map the super-resolution structural features to the gradient branch as gradient branch prior information; The gradient branch is used to extract gradient features based on the spectral polarization image and the super-resolution structural features to obtain an image gradient image; The super-resolution main branch is used to obtain a super-resolution image based on the super-resolution structural feature image and the image gradient image.

8. A large dynamic imaging method based on spectral polarization, characterized in that: include: Converging the target area and outputting the target light in the form of parallel light; Performing multi-aperture imaging and polarization multi-spectral modulation on the target light to obtain a spectral polarization image, and directly imaging the target light to obtain an original light image; Acquire bright part detail information and dark part detail information of the target area according to the spectral polarization image and the original light image within one exposure cycle; The spectral polarization image is subjected to gradient mapping and feature extraction to generate a super-resolution image.

9. The large dynamic imaging method based on spectral polarization according to claim 7, characterized in that: The step of performing gradient mapping and feature extraction on the spectral polarization image to generate a super-resolution image specifically includes: Acquire a gradient mapping image according to the difference between adjacent pixels in the spectral polarization image; Extracting super-resolution structural features of the spectral polarization image, connecting the spectral polarization image to the super-resolution structural features, obtaining a super-resolution feature image, and mapping the super-resolution structural features to the gradient branch as gradient branch prior information; Extracting the gradient features of the gradient mapping image according to the super-resolution structural features, connecting the gradient mapping image to the gradient features, and obtaining a gradient feature image; Performing feature connection on the super-resolution feature image and the gradient feature image to obtain a feature stitching image; Extracting feature long-range dependencies in the feature stitched image, and generating a super-resolution image based on the feature long-range dependencies.

10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 7 to 8.

Images