A multi-dimensional perception-oriented super-resolution imaging method and system

Abstract

The application provides a multi-dimensional perception-oriented super-resolution imaging method and system. The method comprises the following steps: acquiring an original low-resolution polarization mosaic image collected by a defocus plane polarization camera, and calculating a first Stokes parameter and a second Stokes parameter according to light intensity components of multiple polarization directions contained in the original low-resolution polarization mosaic image; inputting the original low-resolution polarization mosaic image, the first Stokes parameter and the second Stokes parameter into a pre-trained polarization reconstruction model to obtain a high-resolution polarization image subjected to super-resolution reconstruction; wherein the pre-trained polarization reconstruction model is constructed through a multi-scale large kernel attention mechanism; the multi-scale large kernel attention mechanism is arranged at an encoder layer of the pre-trained polarization reconstruction model and is used for global feature modeling. Based on this, the super-resolution reconstruction of the polarization image is realized, which is physically faithful, efficient in feature extraction and strong in robustness.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of polarization optical imaging and deep learning image processing, specifically to a super-resolution imaging method and system for multi-dimensional perception. Background Technology

[0002] Polarization imaging technology, by sensing the polarization state of light waves, can effectively distinguish materials, identify camouflage, and enhance image contrast. In the field of real-time polarization imaging, the Division of Focal Plane (DoFP) polarization camera has become the mainstream device due to its compact structure and ability to acquire polarization information instantaneously. However, because the DoFP camera integrates a micro polarizer array (MPA) on its sensor surface, each pixel can only acquire the light intensity in one polarization direction, thus segmenting the spatial sampling at the pixel level. This significantly reduces the spatial resolution of the image directly output by the camera and results in a severe "de-mosaic" effect during subsequent image reconstruction. To obtain a complete Stokes vector image with high spatial resolution, the development of efficient and high-precision polarization image reconstruction technology is crucial.

[0003] To address the aforementioned problems, deep learning-based methods have become a current research hotspot. These methods typically treat polarization image reconstruction as an end-to-end mapping problem. The basic principle is to utilize deep convolutional neural networks (CNNs) or generative adversarial networks (GANs) to directly recover high-resolution polarization images from low-resolution mosaic images. The network model generally includes modules for feature extraction, nonlinear mapping, and upsampling reconstruction. By training on large-scale datasets, it learns complex mapping relationships from low-quality inputs to high-quality outputs, aiming to achieve reconstruction results surpassing traditional interpolation methods.

[0004] However, existing deep learning-based super-resolution methods are mostly derived from general models designed for RGB images. When directly transferred to the field of polarization imaging, these methods have significant limitations. First, the loss functions of general networks (such as mean squared error) usually only focus on the reconstruction accuracy of pixel intensity, lacking consideration of the inherent physical constraints between the components of the Stokes vector. This leads to distortion of physical information in the reconstructed Degree of Linear Polarization (DoLP) or Angle of Polarization (AoP) images, for example, producing values ​​exceeding reasonable ranges. Second, there are strong physical correlations and complementarities between multiple channels of polarization images, but general network structures often treat them as ordinary multi-channel features, failing to effectively model these specific inter-channel coupling relationships, resulting in low learning efficiency for polarization features. Furthermore, models pre-trained on visible light images of natural scenes struggle to adapt to the unique texture distribution of polarization images, exhibiting poor generalization ability and robustness in complex reflection scenes. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention provides a super-resolution imaging method and system for multi-dimensional perception.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a super-resolution imaging method for multi-dimensional perception, comprising: The original low-resolution polarization mosaic image acquired by the split-plane polarization camera is obtained, and the first Stokes parameter and the second Stokes parameter are calculated based on the light intensity components of multiple polarization directions contained in the original low-resolution polarization mosaic image. The original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter are input into the pre-trained polarization reconstruction model to obtain a high-resolution polarization image reconstructed by super-resolution. The pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism. This mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model to perform global feature modeling. The multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image.

[0007] In a second aspect, the present invention provides a super-resolution imaging system for multi-dimensional perception, the super-resolution imaging system for multi-dimensional perception comprising: an acquisition unit and a polarization reconstruction unit; The acquisition unit is used to acquire the original low-resolution polarized mosaic image captured by the split-plane polarization camera, and calculate the first Stokes parameter and the second Stokes parameter based on the light intensity components of multiple polarization directions contained in the original low-resolution polarized mosaic image. The polarization reconstruction unit is used to input the original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter into the pre-trained polarization reconstruction model to obtain a high-resolution polarization image after super-resolution reconstruction. The pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism. This mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model to perform global feature modeling. The multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image.

[0008] This invention provides a super-resolution imaging method and system for multi-dimensional perception. The super-resolution imaging method for multi-dimensional perception includes: acquiring an original low-resolution polarized mosaic image captured by a focal plane polarization camera; calculating a first Stokes parameter and a second Stokes parameter based on the light intensity components of multiple polarization directions contained in the original low-resolution polarized mosaic image; inputting the original low-resolution polarized mosaic image, the first Stokes parameter, and the second Stokes parameter into a pre-trained polarization reconstruction model to obtain a high-resolution polarized image reconstructed by super-resolution; wherein the pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism; the multi-scale large kernel attention mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model for global feature modeling; the multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image. In this invention, firstly, the original low-resolution polarized mosaic image, along with the calculated first and second Stokes parameters, are input into a pre-trained polarization reconstruction model. This allows the pre-trained model to directly incorporate the physical constraints between Stokes vectors during training and inference, thus solving the problem of DoLP / AoP physical information distortion caused by the general loss function focusing only on pixel intensity reconstruction accuracy and ignoring polarization physical correlations. Secondly, a multi-scale large-kernel attention mechanism deployed in the encoder layer constructs an equivalent large receptive field through cascading depthwise separable convolutions and dilated convolutions of different scales. This adaptively fuses local details and global polarization distribution features, effectively modeling the strong physical coupling relationships between multiple channels of the polarized image. This solves the problems of general network structures being unable to efficiently learn inter-channel correlations and low polarization feature learning efficiency. Finally, this pre-trained polarization reconstruction model is specifically designed for polarization imaging tasks. Its multi-scale large-kernel attention mechanism fully adapts to the unique texture distribution and reflection characteristics of polarized images, avoiding the transfer bias of pre-trained models in natural scenes, thus solving the problem of poor generalization ability and robustness in complex reflection scenes. In summary, this invention achieves super-resolution reconstruction that physically preserves polarization images, efficiently extracts features, and is robust.

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a super-resolution imaging method for multi-dimensional perception provided in an embodiment of the present invention; Figure 2 An exemplary flowchart of a super-resolution imaging method for multidimensional perception is shown. Figure 3 A schematic diagram of the structure of the first encoder is shown as an example; Figure 4 This is a schematic diagram of a super-resolution imaging system for multi-dimensional perception provided in an embodiment of the present invention. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0012] To achieve physically faithful, efficient feature extraction, and robust super-resolution reconstruction of polarization images, this invention provides a super-resolution imaging method oriented towards multi-dimensional perception. Figure 1 This is a flowchart illustrating a super-resolution imaging method for multi-dimensional perception provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: S101. Acquire the original low-resolution polarized mosaic image captured by the split-focus plane polarization camera, and calculate the first Stokes parameter and the second Stokes parameter based on the light intensity components of multiple polarization directions contained in the original low-resolution polarized mosaic image.

[0013] Optionally, the first Stokes parameter and the second Stokes parameter are expressed as: ; in, Indicates the first Stokes parameter. Indicates the second Stokes parameter. , , as well as These represent the light intensity components in the four polarization directions of 0°, 45°, 90°, and 135° extracted from the original low-resolution polarized mosaic image.

[0014] S102. Input the original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter into the pre-trained polarization reconstruction model to obtain a high-resolution polarization image reconstructed by super-resolution.

[0015] The pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism. This mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model to perform global feature modeling. The multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image.

[0016] Figure 2 An exemplary overall flowchart of a super-resolution imaging method for multi-dimensional perception is shown, such as... Figure 2 As shown, S102 includes: For the first Stokes parameter Second Stokes parameter Perform in sequence Convolutional processing and SiLU activation function operation yield the first Stokes feature and the second Stokes feature, respectively. The first Stokes feature and the second Stokes feature are sequentially fused and compressed to obtain the polarization fusion feature Polimg1. Perform on the original low-resolution polarized mosaic image Convolution processing yields the initial low-resolution feature F. img ; The polarization fusion features and the initial low-resolution features are jointly input into a deep feature extraction network for processing to obtain demosaic features. The demosaic features are processed by depthwise separable convolution to obtain deep mosaic features; The deep mosaic features are refined to obtain the refined features. The refined features are combined with the initial low-resolution features using residual fusion to obtain the de-mosaic fused features. Calculate the Stokes parameters corresponding to the demosaicized fusion features to obtain the first demosaicized Stokes parameters. Second to Masaxophone parameters ; The first and second demasked parameters were processed sequentially. Convolutional processing and SiLU activation function operation yield the first and second de-Mathiestokes features, respectively. The first and second de-Masses-Stokes features are sequentially fused and compressed to obtain the de-masses polarization fusion feature Polimg2. The demosaic polarization fusion feature and some features from the deep feature extraction network are input into the super-resolution decoding module for decoding to obtain the super-resolution decoding features. The original low-resolution polarized mosaic image is subjected to bilinear interpolation to obtain the interpolated image; The super-resolution decoded features and the interpolated image are subjected to residual fusion processing to obtain a high-resolution polarization image.

[0017] Optionally, the first Stokes feature and the second Stokes feature are sequentially fused and compressed to obtain polarization fused features, including: Perform channel feature concatenation on the first Stokes feature and the second Stokes feature to obtain the Stokes channel concatenation feature; The Stokes channel splicing features are subjected to multi-layer convolution operations to perform feature dimensionality reduction, resulting in polarization fusion features.

[0018] Feature fusion and feature compression begin by concatenating the first and second Stokes features at the same spatial resolution along the channel dimension, resulting in an intermediate tensor with double the number of channels, known as the "Stokes channel concatenated feature". Then, based on several layers of convolution (in this embodiment, we use...),... First, perform cross-channel mixing during convolution and reduce the number of channels to the target dimension, then combine with... Convolution performs local spatial smoothing / refinement, and in the middle, normalization and nonlinear transformations such as SiLU / ReLU are applied to the spliced ​​features. Nonlinear transformations and dimensionality reduction / recalibration of the channel directions are then performed. Finally, a "polarization fusion feature" with a fixed number of channels is output, which realizes the fusion of the complementary strength relationship between the two Stokes sub-informations into a more compact representation and completes feature compression.

[0019] The implementation process of depthwise separable convolution can be broken down into the following steps: The first step is to perform depthwise convolution, that is, to use a spatial convolution kernel independently for each channel of the input (in this embodiment, we use...). The first step extracts spatial details only within a channel, without changing the number of channels; the second step performs pointwise convolution (in this embodiment, pointwise convolution is used). Convolution linearly reassembles the outputs of each channel and maps them to the target number of channels; for example, if the tensor shape of the demosaic feature is [H,W,C] (e.g., C=32), it first uses... Convolution (groups=C) still yields [H,W,C], then... Convolution maps C to C' (C'=64), thereby significantly reducing the number of parameters and computation while approximately maintaining the receptive field, and outputting "deep mosaic features".

[0020] The process of residual fusion can be described as follows: First, the refined features are aligned with the initial low-resolution features in terms of spatial dimensions. If the number of channels is inconsistent between the two, a third layer is needed. Convolution unifies the number of channels. Then, a pixel-wise addition operation is performed on the two feature tensors, directly summing the values ​​of the corresponding channels at each spatial location to obtain a new feature map. This new feature map contains both the high-frequency details and polarization correction information learned in the refined features and retains the basic intensity and low-frequency structure from the initial low-resolution features, thus forming the final residual fusion result, i.e., the demosaic fusion feature. Throughout this process, there are no additional non-linear activation or normalization steps; the information from the two paths is merged solely through addition, enabling the network to learn the residual signal more stably while avoiding gradient vanishing or exploding problems.

[0021] Optionally, the polarization fusion features and the initial low-resolution features are jointly input into a deep feature extraction network for processing to obtain demosaic features, including: The polarization fusion features and the initial low-resolution features are input together into the first encoder for encoding processing to obtain the first encoded features; The first encoded feature and the polarization fusion feature are input together into the second encoder for encoding processing to obtain the second encoded feature; The second coding feature is processed by constructing an equivalent large receptive field based on a multi-scale large kernel attention mechanism to obtain the equivalent coding feature. The equivalent coding features and polarization fusion features are input together into the first decoder for decoding to obtain the first decoded features; The first decoding feature and the polarization fusion feature are input together into the second decoder for decoding processing to obtain the de-mosaic feature.

[0022] Optionally, the first encoder, the second encoder, the first decoder, and the second decoder have the same structure; Figure 3 An exemplary structural schematic diagram of the first encoder is shown, such as... Figure 3 As shown, the polarization fusion features and the initial low-resolution features are jointly input into the first encoder for encoding processing to obtain the first encoded features, including: The polarization fusion features are input into the first branch to obtain the first branch features; the first branch is provided with a first normalization layer Norm1 and a first convolutional layer Conv1 connected in sequence; The initial low-resolution features are input into the second branch to obtain the second branch features. The second branch is configured with a multi-head attention layer MDTA, a second normalization layer Norm2, a second convolutional layer Conv2, a gated unit GELU, and an activation function layer GDFN connected in sequence. The second branch features are the output features of the initial low-resolution features processed sequentially by the multi-head attention layer, the second normalization layer, the second convolutional layer, and the gated unit. The second branch feature is concatenated with the first branch feature to obtain the branch concatenation feature. The branch splicing features are added pixel by pixel to the initial low-resolution features to obtain the superimposed features; The superimposed features are input into the activation function layer for processing to obtain the first encoded features.

[0023] Optionally, the depth mosaic features are refined to obtain refined features, including: The feature refinement unit performs feature refinement processing on the deep mosaic features to obtain refined features; wherein, the feature refinement unit is provided with a cascaded first feature refinement subunit, a second feature refinement subunit, and a third feature refinement subunit; The first feature refinement subunit, the second feature refinement subunit, and the third feature refinement subunit have the same structure and are all based on Convolution module and SiLU activation function module are constructed.

[0024] Optionally, some features from the de-mosaic polarization fusion feature and the deep feature extraction network are input into the super-resolution decoding module for decoding processing to obtain super-resolution decoded features, including: The equivalent coding features and the de-mosaic polarization fusion features are input into the third decoder for decoding to obtain the third decoding features. The third decoding feature and the de-mosaic polarization fusion feature are input into the fourth decoder for decoding processing to obtain the fourth decoding feature; The fourth decoded feature is convolutionally processed using a depthwise separable convolution module to obtain a depthwise separable convolution feature. The depthwise separable convolutional features are refined to obtain super-resolution decoded features; The third and fourth decoders have the same structure as the first decoder.

[0025] Optionally, the equivalent large receptive field construction process is based on Figure 2 The MLKA module is implemented. Specifically, the second encoded features are processed using a multi-scale large kernel attention mechanism to construct an equivalent large receptive field, resulting in equivalent encoded features, including: based on Convolution maps the second encoded features across channels to obtain the first mapped features; based on The first mapping feature is processed by depth convolution to obtain the first depth feature; Perform on the first depth feature Dilated convolution processing yields the second depth feature; based on Convolution performs non-linear mapping on the second depth features to obtain the second mapped features; The second mapping feature and the second coding feature are subjected to residual fusion processing to obtain the equivalent coding feature.

[0026] Correspondingly, the first mapping feature is represented as: ; It is the first mapping feature. These are the weight parameters of the first layer. These are the first-layer bias parameters. This represents the second coding feature.

[0027] The first depth feature is represented as: ; Indicates the first depth feature, This represents the weight parameters of the depthwise convolution, where *g represents group convolution.

[0028] Then, based on local features, the void ratio is introduced. of Deeply dilated convolution enables skip-spatial sampling to obtain a second depth feature. Represented as: ; in The weight parameters represent the depthwise dilated convolution.

[0029] In order to extract the second depth features Transformed into a weighted distribution, and then passed through... The convolutional layer performs a non-linear mapping to generate a second mapping feature. , is represented as: ; Among them, These are the weight parameters of the second layer. It is the second-layer bias parameter.

[0030] Finally, the generated second mapping features The second coding feature x is applied and fused through residual connections to obtain the equivalent coding feature: ; This represents the equivalent coding features.

[0031] The following embodiments will fully describe the execution process of the super-resolution imaging method for multi-dimensional perception. This method first acquires a raw low-resolution polarized mosaic image captured by a focal plane polarization camera, and then uses the light intensity components in the four polarization directions (0°, 45°, 90°, and 135°) contained in the image. , , as well as Calculate the first Stokes parameter Second Stokes parameter The original low-resolution polarization mosaic image, along with the first and second Stokes parameters, are then input into a pre-trained polarization reconstruction model to obtain a high-resolution polarization image reconstructed using super-resolution. The pre-trained polarization reconstruction model is constructed using a multi-scale large-kernel attention mechanism deployed in the model's encoder layer for global feature modeling. This mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions at different scales, adaptively fusing local detail features and global polarization distribution features of the image.

[0032] In the specific implementation process, the first Stokes parameter is first... Second Stokes parameter The first and second Stokes features are obtained by sequentially performing 1×1 convolution and SiLU activation functions. These two features are then subjected to feature fusion and compression: first, they are concatenated along the channel dimension at the same spatial resolution to obtain the Stokes channel concatenated features; then, they are processed through several layers of convolution (including...). Convolution performs cross-channel blending and dimensionality reduction, and Convolution performs local spatial smoothing, followed by normalization and SiLU activation for nonlinear transformation and channel orientation dimensionality reduction, ultimately outputting a polarization fusion feature P with a fixed number of channels. olimg1 Simultaneously, the original low-resolution polarized mosaic image was processed... Convolution processing yields the initial low-resolution feature F. img .

[0033] Next, the polarization fusion feature P olimg1 and initial low-resolution features F img The features are processed by a deep feature extraction network to obtain de-mosaic features. The deep feature extraction network consists of a two-stage encoder and a two-stage decoder, with each encoder and decoder having the same structure. Taking the first encoder as an example, it inputs the polarization fusion features into the first branch (passing sequentially through the first normalization layer Norm1 and the first convolutional layer Conv1) to obtain the first branch features. It then inputs the initial low-resolution features into the second branch (passing sequentially through the multi-head attention layer MDTA, the second normalization layer Norm2, the second convolutional layer Conv2, and the gated unit GELU) to obtain the second branch features. The second branch features are then concatenated with the first branch features, and the concatenated result is added pixel-by-pixel to the initial low-resolution features. Finally, it passes through the activation function layer GDFN to obtain the first encoded feature. Similarly, the second encoder takes the first encoded feature and the polarization fusion features as input and outputs the second encoded feature.

[0034] Subsequently, the second encoded features are processed using a multi-scale large kernel attention mechanism (MLKA module) to construct an equivalent large receptive field, resulting in equivalent encoded features. The specific process is as follows: First, using... Convolution performs inter-channel mapping on the second encoded features to obtain the first mapped features. Then then Depth convolution is applied to the first mapped features to obtain the first depth features. Next, the first depth feature is subjected to a hole rate of 3. Deeply dilated convolution processing enables skip-spatial sampling to obtain a second depth feature. ; then use Convolution performs a non-linear mapping on the second depth features to generate second mapped features. Finally, the second mapping feature A and the second coding feature X are multiplied element-wise and the residuals are fused to obtain the equivalent coding feature. .

[0035] The equivalent encoded features and polarization fusion features are then input into the first decoder for decoding to obtain the first decoded features. The first decoded features and polarization fusion features are then input into the second decoder for decoding to obtain the de-mosaic features. Next, the de-mosaic features are subjected to depthwise separable convolution: first, each channel is processed independently... Depthwise convolution (groups=C) extracts spatial details, then uses... Pointwise convolution maps the number of channels to the target dimension, yielding deep mosaic features. These features are then refined; each refinement unit consists of three cascaded subunits, each based on... The convolutional module and the SiLU activation function module are used to construct the refined features. The refined features are then residually fused with the initial low-resolution features Fimg: firstly through... The convolutional alignment channels are then summed pixel by pixel to obtain the de-mosaic fusion features.

[0036] Calculate the Stokes parameters corresponding to the demosaicing fusion features to obtain the first and second demosaicing Stokes parameters. Then, process these two parameters sequentially. Convolutional processing and SiLU activation function operations yield the first and second de-Masses-Stokes features, respectively. These two features are then subjected to feature fusion and compression (using the same methods as described above) to obtain the de-mosaic polarization fusion feature Polemg2. Finally, the equivalent encoded features and the de-mosaic polarization fusion feature P... olimg2 The input is decoded by the third decoder (which has the same structure as the first decoder) to obtain the third decoded feature; then the third decoded feature and P are... olimg2The input is processed by the fourth decoder to obtain the fourth decoded feature. Then, the fourth decoded feature is subjected to depthwise separable convolution (using the same method as described above) to obtain the depthwise separable convolutional feature. This feature is then refined (using the same refinement unit as described above) to obtain the super-resolution decoded feature. Simultaneously, bilinear interpolation is performed on the original low-resolution polarized mosaic image to obtain the interpolated image. Finally, the super-resolution decoded feature and the interpolated image are subjected to residual fusion (pixel-by-pixel addition) to obtain the final high-resolution polarized image.

[0037] This invention provides a super-resolution imaging method for multi-dimensional perception. First, the original low-resolution polarized mosaic image, along with the calculated first and second Stokes parameters, are input into a pre-trained polarization reconstruction model. This allows the pre-trained model to directly incorporate the physical constraints between Stokes vectors during training and inference, thus solving the problem of DoLP / AoP physical information distortion caused by the general loss function focusing only on pixel intensity reconstruction accuracy and ignoring polarization physical correlations. Then, a multi-scale large-kernel attention mechanism deployed in the encoder layer constructs an equivalent large receptive field through cascading depthwise separable convolutions and dilated convolutions of different scales. This adaptively fuses local details and global polarization distribution features, effectively modeling the strong physical coupling relationships between multiple channels of the polarization image. This solves the problems of general network structures being unable to efficiently learn inter-channel correlations and low polarization feature learning efficiency. Finally, this pre-trained polarization reconstruction model is specifically designed for polarization imaging tasks. Its multi-scale large-kernel attention mechanism fully adapts to the unique texture distribution and reflection characteristics of polarization images, avoiding the transfer bias of pre-trained models in natural scenes, thus solving the problem of poor generalization ability and robustness in complex reflection scenes. In summary, this invention achieves super-resolution reconstruction that physically preserves polarization images, efficiently extracts features, and is robust.

[0038] The method provided in this embodiment of the invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc., and this embodiment of the invention does not limit the application to such devices.

[0039] Based on the same inventive concept, embodiments of the present invention also provide a super-resolution imaging system for multi-dimensional perception. Figure 4 This is a schematic diagram of the structure of a super-resolution imaging system for multi-dimensional perception provided in an embodiment of the present invention, as shown below. Figure 4 As shown, it includes: an acquisition unit 401 and a polarization reconstruction unit 402; The acquisition unit 401 is used to acquire the original low-resolution polarization mosaic image captured by the split-plane polarization camera, and calculate the first Stokes parameter and the second Stokes parameter based on the light intensity components of multiple polarization directions contained in the original low-resolution polarization mosaic image. The polarization reconstruction unit 402 is used to input the original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter into the pre-trained polarization reconstruction model to obtain a high-resolution polarization image reconstructed by super-resolution. The pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism. This mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model to perform global feature modeling. The multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image.

[0040] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In this description, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0041] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered within the scope of protection of the present invention.

Claims

1. A super-resolution imaging method for multi-dimensional perception, characterized in that, include: The original low-resolution polarized mosaic image acquired by the split-plane polarization camera is obtained, and the first Stokes parameter and the second Stokes parameter are calculated based on the light intensity components of multiple polarization directions contained in the original low-resolution polarized mosaic image. The original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter are input into the pre-trained polarization reconstruction model to obtain a high-resolution polarization image reconstructed by super-resolution. The pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism. This multi-scale large kernel attention mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model for global feature modeling. The multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image.

2. The super-resolution imaging method for multi-dimensional perception according to claim 1, characterized in that, The first Stokes parameter and the second Stokes parameter are expressed as follows: ； in, Indicates the first Stokes parameter. Indicates the second Stokes parameter. , , as well as These represent the light intensity components in the four polarization directions of 0°, 45°, 90°, and 135° extracted from the original low-resolution polarized mosaic image.

3. The super-resolution imaging method for multi-dimensional perception according to claim 1, characterized in that, The step of inputting the original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter into a pre-trained polarization reconstruction model to obtain a high-resolution polarization image after super-resolution reconstruction includes: The first Stokes parameter and the second Stokes parameter are respectively processed sequentially. Convolutional processing and SiLU activation function operation yield the first Stokes feature and the second Stokes feature, respectively. The first Stokes feature and the second Stokes feature are sequentially fused and compressed to obtain polarization fused features; Perform the following steps on the original low-resolution polarized mosaic image: Convolution processing yields initial low-resolution features; The polarization fusion feature and the initial low-resolution feature are input together into the deep feature extraction network for processing to obtain the demosaic feature. The demosaic features are subjected to depthwise separable convolution to obtain deep mosaic features; The depth mosaic features are refined to obtain refined features; The refined features and the initial low-resolution features are subjected to residual fusion processing to obtain the de-mosaic fused features; Calculate the Stokes parameters corresponding to the demosaic fusion features to obtain the first demosaic Stokes parameter and the second demosaic Stokes parameter; The first demosaictoke parameter and the second demosaictoke parameter are respectively processed sequentially. Convolutional processing and SiLU activation function operation yield the first and second de-Mathiestokes features, respectively. The first de-Masses-Stokes feature and the second de-Masses-Stokes feature are sequentially fused and compressed to obtain the de-masses polarization fusion feature. The demosaic polarization fusion feature and some features from the deep feature extraction network are input into the super-resolution decoding module for decoding to obtain the super-resolution decoding feature. The original low-resolution polarized mosaic image is subjected to bilinear interpolation to obtain an interpolated image; The super-resolution decoded features are subjected to residual fusion processing with the interpolated image to obtain the high-resolution polarization image.

4. The super-resolution imaging method for multi-dimensional perception according to claim 3, characterized in that, The first Stokes feature and the second Stokes feature are sequentially fused and compressed to obtain polarization fused features, including: Perform channel feature concatenation on the first Stokes feature and the second Stokes feature to obtain the Stokes channel concatenation feature; The Stokes channel splicing features are subjected to multi-layer convolution operations to perform feature dimensionality reduction, resulting in the polarization fusion features.

5. The super-resolution imaging method for multi-dimensional perception according to claim 3, characterized in that, The process of inputting the polarization fusion features and the initial low-resolution features into the deep feature extraction network for processing to obtain demosaic features includes: The polarization fusion feature and the initial low-resolution feature are input together into the first encoder for encoding processing to obtain the first encoded feature; The first encoded feature and the polarization fusion feature are input together into the second encoder for encoding processing to obtain the second encoded feature; Based on the multi-scale large kernel attention mechanism, the second encoded feature is processed to construct an equivalent large receptive field to obtain the equivalent encoded feature. The equivalent coding feature and the polarization fusion feature are input together into the first decoder for decoding to obtain the first decoding feature; The first decoding feature and the polarization fusion feature are input together into the second decoder for decoding processing to obtain the demosaic feature.

6. The super-resolution imaging method for multi-dimensional perception according to claim 5, characterized in that, The first encoder, the second encoder, the first decoder, and the second decoder have the same structure; the step of inputting the polarization fusion feature and the initial low-resolution feature into the first encoder for encoding processing to obtain the first encoded feature includes: The polarization fusion feature is input into the first branch to obtain the first branch feature; the first branch is provided with a first normalization layer and a first convolutional layer connected in sequence. The initial low-resolution features are input into the second branch to obtain the second branch features; the second branch is provided with a multi-head attention layer, a second normalization layer, a second convolutional layer, a gating unit and an activation function layer connected in sequence; the second branch features are the output features of the initial low-resolution features processed sequentially by the multi-head attention layer, the second normalization layer, the second convolutional layer and the gating unit. The second branch feature is concatenated with the first branch feature to obtain the branch concatenation feature. The branch splicing features are added pixel by pixel to the initial low-resolution features to obtain the superimposed features; The superimposed features are input into the activation function layer for processing to obtain the first encoded features.

7. The super-resolution imaging method for multi-dimensional perception according to claim 3, characterized in that, The refinement process for the depth mosaic features to obtain refined features includes: The feature refinement unit performs feature refinement processing on the depth mosaic feature to obtain refined features; wherein, the feature refinement unit is provided with a cascaded first feature refinement subunit, a second feature refinement subunit, and a third feature refinement subunit; The first feature refinement subunit, the second feature refinement subunit, and the third feature refinement subunit have the same structure and are all based on Convolution module and SiLU activation function module are constructed.

8. A super-resolution imaging method for multi-dimensional perception according to claim 5, characterized in that, The de-mosaic polarization fusion features and some features from the deep feature extraction network are input into the super-resolution decoding module for decoding processing to obtain super-resolution decoding features, including: The equivalent coding feature and the demosaic polarization fusion feature are input into the third decoder for decoding to obtain the third decoding feature; The third decoding feature and the de-mosaic polarization fusion feature are input into the fourth decoder for decoding processing to obtain the fourth decoding feature; The fourth decoded feature is convolutionally processed using a depthwise separable convolution module to obtain a depthwise separable convolution feature. The depthwise separable convolutional features are refined to obtain the super-resolution decoding features; The third decoder and the fourth decoder have the same structure as the first decoder.

9. A super-resolution imaging method for multi-dimensional perception according to claim 5, characterized in that, The equivalent large receptive field construction process based on the multi-scale large kernel attention mechanism for the second encoded feature is used to obtain the equivalent encoded feature, including: based on Convolution maps the second encoded feature across channels to obtain the first mapped feature; based on The first mapping feature is processed by depth convolution to obtain the first depth feature; Perform on the first depth feature Dilated convolution processing yields the second depth feature; based on Convolution performs non-linear mapping on the second depth feature to obtain the second mapped feature; The second mapping feature and the second coding feature are subjected to residual fusion processing to obtain the equivalent coding feature.

10. A super-resolution imaging system for multi-dimensional perception, characterized in that, The super-resolution imaging system for multi-dimensional perception includes: an acquisition unit and a polarization reconstruction unit; The acquisition unit is used to acquire the original low-resolution polarized mosaic image captured by the split-plane polarization camera, and to calculate the first Stokes parameter and the second Stokes parameter based on the light intensity components of multiple polarization directions contained in the original low-resolution polarized mosaic image. The polarization reconstruction unit is used to input the original low-resolution polarization mosaic image, the first Stokes parameter, and the second Stokes parameter into the pre-trained polarization reconstruction model to obtain a high-resolution polarization image reconstructed by super-resolution. The pre-trained polarization reconstruction model is constructed using a multi-scale large kernel attention mechanism. This multi-scale large kernel attention mechanism is deployed in the encoder layer of the pre-trained polarization reconstruction model for global feature modeling. The multi-scale large kernel attention mechanism constructs an equivalent large receptive field by cascading depthwise separable convolutions and dilated convolutions of different scales to adaptively fuse local detail features and global polarization distribution features of the image.

Images