A controllable pixel shift driven joint demosaicking and super-resolution reconstruction method

Abstract

This invention discloses a controllable pixel displacement-driven joint demosaicing and super-resolution reconstruction method, comprising: 1. Constructing an original image sequence acquisition system consisting of a piezoelectric displacement module, a control module, and an image acquisition module, acquiring multiple frames of original image sequences through controllable sub-pixel displacement; 2. Constructing a joint demosaicing and super-resolution reconstruction network based on the multiple frame sequences, performing feature extraction and fusion on the multiple frame sequences; 3. Introducing a direction-aware wavelet enhancement module into the network, enhancing edge texture expression through wavelet decomposition, sub-band directional modulation, and reconstruction; 4. Introducing a structure-aware convolutional kernel attention module into the network, adaptively adjusting the convolutional kernel response according to local structural complexity to suppress false colors and structural distortion. This invention utilizes known displacement information to improve the determinism of original image sequence fusion and the detail fidelity and color consistency in high-frequency regions, and is applicable to scenarios such as industrial camera visual inspection and precision measurement.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology. Specifically, it is a method for joint demosaicing and super-resolution reconstruction of multi-frame original image sequences based on controllable pixel displacement. It is specifically used in scenarios such as industrial visual inspection and precision measurement to acquire high-quality, high-resolution images under limited sampling conditions. Background Technology

[0002] High-resolution imaging has significant application value in industrial visual inspection, precision measurement, and micro-defect identification. However, limitations such as the diffraction limit of optical systems, sensor pixel size, sampling bandwidth, and noise often prevent images acquired in a single exposure from simultaneously meeting the requirements of high resolution and high signal-to-noise ratio. To improve spatial resolution, existing technologies typically employ two approaches: one is hardware upgrades through higher pixel density sensors or higher numerical aperture lenses; the other is computational imaging methods to reconstruct low-resolution observations, including single-frame super-resolution and multi-frame super-resolution techniques.

[0003] In practical imaging systems, image sensors commonly employ color filter arrays for sampling, typically Bayer arrays. Bayer sampling results in each pixel recording only single-channel information, requiring de-mosaicing to recover full-color information. Existing systems often use a sequential processing flow of "de-mosaic first, then super-resolution," where single or multiple frames are first interpolated to remove the mosaic effect to obtain a color image, followed by super-resolution reconstruction. However, since de-mosaicing is a typical undersampling reconstruction problem, it easily introduces color artifacts, zipper effects, and structural distortions in areas with strong edges and high-frequency textures. When subsequent multi-frame fusion and super-resolution reconstruction are performed, these errors may be further amplified during cross-frame alignment and fusion, leading to problems such as enhanced false colors, inconsistent edge structures, and blurred details, thus affecting the stability and reliability of industrial inspection and measurement.

[0004] For multi-frame super-resolution reconstruction, existing techniques typically rely on inter-frame motion estimation and registration, such as alignment methods based on optical flow, feature matching, or block matching, before multi-frame fusion reconstruction. While these methods are applicable to natural scenes or handheld shooting conditions, in scenarios like industrial microscopy and precision measurement, the imaging field of view may contain numerous fine textures, periodic structures, and high-contrast edges. Furthermore, they are susceptible to noise, underexposure, lens distortion, micro-vibrations, and plateau errors, leading to instability in motion estimation and a tendency to produce sub-pixel-level alignment errors. Especially during the fusion of multiple original images, the color undersampling caused by Bayer sampling coupled with spatial displacement means that even small alignment deviations can cause significant color inconsistencies and structural distortions, reducing the interpretability and consistency of the reconstruction results.

[0005] To reduce the uncertainty of alignment errors, some systems employ active displacement imaging, using precision actuators such as piezoelectric displacement stages to drive image sensors or imaging components to perform controllable sub-pixel displacement, acquiring multiple frames of sequential images with complementary sampling positions, and providing definite displacement information from the acquisition end. However, existing active displacement multi-frame super-resolution schemes still generally suffer from the following shortcomings: First, most methods still follow the "de-mosaic first, then fusion" processing strategy, making it difficult to fundamentally avoid the accumulation and amplification of de-mosaic errors in multi-frame fusion; Second, in the multi-frame fusion process of joint reconstruction of original image sequences, there is still a lack of a universal solution that balances detail fidelity and color consistency in effectively expressing and enhancing directional high-frequency details while suppressing false colors and structural distortions caused by cross-frame fusion; Third, in areas with dense high-frequency textures or strong edges, the fusion and reconstruction algorithms lack the adaptive modulation capability for directional information and structural complexity, making it difficult to balance detail clarity and structural consistency in the reconstruction results. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by proposing a controllable pixel displacement-driven joint demosaic and super-resolution reconstruction method. This method aims to acquire a multi-frame original image sequence containing displacement priors through controllable sub-pixel displacement at the acquisition end, and to achieve multi-frame feature extraction and fusion. While improving the determinism of cross-frame fusion, it enhances the ability to express high-frequency details and suppresses false colors and structural distortion, thereby obtaining high-quality, high-resolution color images to meet the application needs of industrial visual inspection and precision measurement scenarios.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a controllable pixel displacement-driven joint demosaicing and super-resolution reconstruction method, characterized by the following steps: Step 1: Construct an image acquisition system consisting of a two-dimensional piezoelectric displacement stage, a controller, and an image sensor to acquire the original image sequence containing displacement priors. ,in, Indicates the first Frame low-resolution image, This represents the total number of low-resolution images, and ; Indicates the ratio of a high-resolution image to a low-resolution image; Step 2, based on Construct the direction using equation (1) Cross-frame difference : (1) In equation (1), Indicates direction Cross-frame intensity difference between adjacent low-resolution images, and Indicates along direction Two low-resolution images of displacement. , These represent the horizontal, vertical, and main diagonal directions, respectively. The direction is obtained using equation (2). Intensity difference map on : (2) In equation (2), Indicates direction The number of frame pairs that are differentially processed; Step 3: Construct a joint demosaicing and super-resolution reconstruction network, including: a sequence image integration module, a shallow feature extraction module, a deep feature extraction module, and a reconstruction module, and then... and Feature extraction and cross-frame fusion processing are performed to obtain a high-resolution color image. ; Step 3.1: The sequence image integration module integrates the images through sub-pixel convolution. Mapping to target In a high-resolution mesh, this allows for initial alignment. Multi-resolution stitched images , and These are the length and width of the stitched image, respectively. Multiples; Step 3.2, the shallow feature extraction module will... Rearranged into a four-channel representation Then, it is passed through a convolutional layer. The channel dimension is extended Thus, shallow features are obtained. ,in, Indicates the channel dimension; Step 3.3, the deep feature extraction module consists of... Composed of cascaded RDATB blocks, and for and Process the data to output deep feature sequences. With cross-frame intensity difference sequence ,in, Indicates the first Deep feature maps output from cascaded RDATB blocks Indicates the first Cross-frame intensity difference map output by cascaded RDATB blocks Indicates the number of RDATB blocks; Step 3.4: The reconstruction module uses convolutional layers to... Channel dimension Compress to Then, subpixel convolution is used to upsample the compressed deep feature sequence to obtain a high-resolution result image. ; Step 4: Construct the mean absolute error loss function using equation (3) : (3) Step 5: Iteratively train the joint demosaicing and super-resolution reconstruction network using the backpropagation algorithm, and calculate the mean absolute error loss function. Adjust the network parameters until the mean absolute error loss function is reached. The process continues until convergence, resulting in a trained joint demosaic and super-resolution reconstruction model. This model is then used to process the input original image sequence and obtain the optimal reconstructed color high-resolution image.

[0008] The controllable pixel displacement driven joint demosaicing and super-resolution reconstruction method of the present invention is also characterized in that each RDATB block in the deep feature extraction module of step 3.3 is composed of an STL module, a direction-aware wavelet enhancement module, and a structure-aware convolutional kernel attention module. Step 3.3.1, when At that time, the first The STL module in each RDATB block Processing is performed to obtain the first... One reconstructed feature map ; Will and Enter the number respectively The direction-aware wavelet enhancement module in the first RDATB block processes the data to obtain the first... Wavelet Enhancement Feature Map With the Intensity difference map across frames ; Will Enter the first The structure-aware convolutional kernel attention module in the first RDATB block is processed to obtain the first... Deep feature map ; Step 3.3.2, when At that time, the first Deep feature map With the Intensity difference map across frames Enter the number Processing is performed in the RDATB block to obtain the first... Deep feature map With cross-frame intensity difference map Thus, by the first The output of the RDATB block is the first Deep feature map and the Deep feature map With cross-frame intensity difference map .

[0009] Furthermore, the STL module in step 3.3.1 is obtained by following these steps: One reconstructed feature map : Step a1, using equation (4) Feature extraction is performed to obtain the first... Intermediate feature maps ; (4) In equation (4), For multi-head self-attention operations in windows, For the first normalization layer, Indicates residual connection; Step a2, using equation (5) Feature extraction is performed to obtain the first... One reconstructed feature map ; (5) In equation (5), For the second normalization layer, Indicates residual connection, It is a feedforward transform network.

[0010] Furthermore, the feedforward transform network in step a2 It includes: wavelet decomposition unit, low-frequency subband channel mixing unit, high-frequency subband channel mixing unit, and wavelet reconstruction unit, and obtains the following steps: One reconstructed feature map : Step a2.1: Wavelet decomposition units utilize two-dimensional discrete wavelet transform... Mapped to four feature subbands, including: Low-frequency subband and the A set of high-frequency subbands ; Step a2.2, the low-frequency subband channel mixing unit uses equations (6)-(7) to... Processing is performed to obtain the first... Low-frequency reconfiguration subband : (6) (7) In equations (6) and (7), for The k-th low-frequency subband token sequence obtained by flattening This indicates a rearrangement operation. For the first linear layer, This is the GeLU activation function. For the second linear layer, This represents the inverse operation of rearrangement; Step a2.3: The high-frequency subband channel mixing unit uses equations (8)-(10) to process the first... A set of high-frequency subbands Processing is performed to obtain the first... A set of high-frequency reconstructed subbands : (8) (9) (10) In equations (8)-(10), and These are the k-th first intermediate output feature and the k-th second intermediate output feature, respectively. The first convolutional kernel is Convolution mapping operation, This is a channel-wise depthwise convolution operation. For GELU activation function, The second kernel is Convolutional mapping operations; Step a2.4, the wavelet reconstruction unit... and After performing the inverse wavelet transform, then with After performing residual join, the first... One reconstructed feature map .

[0011] Furthermore, the direction-aware wavelet enhancement module in step 3.3.1 is obtained according to the following steps: Wavelet Enhancement Feature Map intensity difference map across frames k : Step b1, for Perform a two-dimensional discrete wavelet transform to obtain the first... Low-frequency subband With the Horizontal high-frequency sub-band , No. Vertical high-frequency subband , No. High-frequency sub-bands in the main diagonal direction ; Step b2: Construct a direction prediction branch consisting of two convolutional layers, a ReLU activation function, and a Sigmoid activation function, and then... Process and generate the first Horizontal attention map , No. Vertical attention map , No. Attention diagram in the main diagonal direction ; Step b3, and After weighting, the first modulated number is obtained. Horizontal high-frequency sub-band ,Will and After weighting, the first modulated number is obtained. Vertical high-frequency subband ,Will and After weighting, the first modulated number is obtained. High-frequency sub-bands in the main diagonal direction ; Step b4: Apply convolution kernels to the first... -1 cross-frame intensity difference map Divided into the first Horizontal edge prior feature map , No. Prior feature maps of vertical edges , No. Prior feature maps of edges along the main diagonal directions ; Step b5: Using equations (11)-(13) respectively, obtain the first... Modulation coefficients in each horizontal direction , No. Modulation coefficients in each vertical direction , No. Modulation coefficients in each main diagonal direction : (11) (12) (13) In equations (11)-(13), Indicates the activation function; Step b6: Using equations (14)-(16) respectively, obtain the first... Horizontally weighted high-frequency sub-bands , No. Vertically weighted high-frequency sub-bands , No. Each main diagonal weighted high-frequency sub-band ; (14) (15) (16) Step b7: Using two convolutional layers and the ReLU activation function, sequentially apply the following to the first... Low-frequency subband The process is performed to obtain the low-frequency subband after feature enhancement for the kth feature. ; Step b8, , , After concatenation, the result is obtained by a convolutional layer. Intensity difference map across frames ; Step b9, for and Perform inverse wavelet transform to obtain the inverse wavelet transform feature map. Thus, the first equation is obtained using equation (17). Wavelet Enhancement Feature Map ; (17) In equation (17), This indicates a residual connection.

[0012] Furthermore, the structure-aware convolutional kernel attention module in step 3.3.1 is obtained according to the following steps: Deep feature map ; Step c1: Construct a dynamic convolution kernel weight generation branch consisting of convolution and ReLU activation functions, and then... Processing is performed to obtain all pixels and the surrounding pixels of each pixel. The k-th dynamic kernel weight sequence corresponding to each region pixel ; Step c2, using convolution kernels for The convolution operation will The number of channels is compressed to Then, a neighborhood unfolding operation is performed to construct the k-th local neighborhood feature set corresponding to all pixel positions. , ; This represents the number of channels after compression. Step c3, for and After performing weighted summation at each pixel position, the k-th aggregated feature map is obtained. Then, after passing through the convolution kernel, it becomes... The convolution operation will Mapping back to the original channel dimension C yields the k-th deep feature map. .

[0013] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing the controllable pixel displacement driven joint demosaic and super-resolution reconstruction method, and the processor is configured to execute the program stored in the memory.

[0014] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, performs the steps of the controllable pixel displacement driven joint demosaic and super-resolution reconstruction method.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: 1. Prior displacement at the acquisition end enhances the determinism and repeatability of fusion: This invention achieves controllable subpixel displacement acquisition through a two-dimensional piezoelectric displacement module, obtaining an original image sequence containing displacement priors. This transforms multi-frame alignment from traditional post-motion estimation to prior knowledge at the acquisition end, reducing the impact of alignment errors and inter-frame inconsistencies on the fusion results from the source. It significantly improves the determinism, stability, and repeatability of the multi-frame fusion and reconstruction process, and is especially suitable for industrial scenarios with fine textures or rich edges.

[0016] 2. Combined demosaicing and super-resolution to reduce cascaded error amplification: This invention achieves joint demosaicing and super-resolution reconstruction, avoiding the accumulation and amplification of demosaic interpolation errors in the traditional "demosaicing first, super-resolution later" serial process during subsequent multi-frame fusion and reconstruction stages. This improves the color and structural consistency of the reconstructed image and effectively reduces the probability of typical artifacts such as false colors, zipper effect, and structural distortion.

[0017] 3. Adaptive orientation and structure enhancement improve high-frequency detail fidelity: This invention introduces a direction-aware wavelet enhancement module into the joint reconstruction network. By performing wavelet decomposition on features and directional modulation and reconstruction on high-frequency subbands in different directions, it enhances the expressive ability of high-frequency details such as edges and textures. At the same time, a structure-aware convolutional kernel attention module is introduced, which adaptively adjusts the convolutional kernel response according to the local structural complexity and performs differentiated modulation on smooth regions and complex structural regions. This further suppresses false colors and structural distortions caused by cross-frame fusion, making the reconstruction results better in terms of detail clarity and structural consistency.

[0018] 4. Improve project applicability through software and hardware collaboration: This invention combines precise sub-pixel displacement control on the hardware side with joint demosaicing and super-resolution reconstruction on the software side. The hardware provides stable and reliable displacement prior input, while the software fully exploits the complementary information between sequential images to complete high-quality reconstruction. This collaborative approach improves reconstruction resolution and image quality while reducing reliance on complex motion estimation and alignment algorithms, making the solution easier to deploy and apply in scenarios such as industrial camera vision inspection, precision measurement, and microstructure observation. Attached Figure Description

[0019] Figure 1 A schematic diagram of the joint demosaicing and super-resolution reconstruction method driven by controllable pixel displacement; Figure 2 A schematic diagram of a joint demosaicing and super-resolution network structure driven by controllable pixel displacement; Figure 3 This is a schematic diagram of the STL module structure of the present invention; Figure 4 This is a schematic diagram of the Conv-MLP feedforward transform network structure of the present invention; Figure 5 This is a schematic diagram of the direction-aware wavelet enhancement module structure of the present invention; Figure 6 This is a schematic diagram of the structure of the structure-aware convolutional kernel attention module of the present invention; Figure 7 This is a comparison image of the high-resolution color images obtained by the present invention and other reconstruction methods in this embodiment. Detailed Implementation

[0020] This embodiment addresses the problems in existing technologies, such as undersampling of colors due to Bayer array sampling in color image sensors, false colors and structural distortions in strong edge and high-frequency texture regions caused by the traditional "de-mosaic first, then super-resolution" concatenated process, and subpixel alignment errors and fusion uncertainties caused by multi-frame super-resolution relying on post-motion estimation. It proposes a joint de-mosaic and super-resolution reconstruction technique for actively displaced original image sequences. This technique leverages controllable subpixel displacement prior at the acquisition end to improve the determinism of cross-frame fusion while simultaneously achieving joint de-mosaic and super-resolution reconstruction of the original image sequence. Furthermore, it improves detail fidelity and color consistency in high-frequency texture and edge regions to meet the stability, repeatability, and traceability requirements of industrial vision inspection and precision measurement applications. This overcomes the limitations of existing hardware and software strategies in resolution improvement, enabling the acquisition of high-quality, high-resolution images in industrial vision inspection and precision measurement scenarios, achieving more accurate and stable image reconstruction. Specifically, this controllable pixel displacement-driven joint de-mosaic and super-resolution reconstruction method and system, such as... Figure 1 As shown, it includes the following steps: Step 1: Construct an image acquisition system consisting of a two-dimensional piezoelectric displacement stage, a controller, and an image sensor. By inputting a preset displacement trajectory command to the controller, the displacement map drives the image sensor to perform sub-pixel-level displacement, thereby acquiring the original image sequence containing displacement priors. ,in, Indicates the first Frame low-resolution image, This represents the total number of low-resolution images, and ; This indicates the ratio of a high-resolution image to a low-resolution image.

[0021] A two-dimensional piezoelectric displacement stage drives an image sensor to make minute displacements along two mutually perpendicular planes, enabling the image sensor to sequentially complete exposure acquisition at multiple sub-pixel sampling phases. Since the displacement trajectory is actively applied by the controller according to preset instructions, the displacement relationship between adjacent frames is deterministic and repeatable, thus providing explicit displacement priors for subsequent multi-frame information fusion. Furthermore, when the super-resolution is... At that time, collection Frame-resolution images are used to form sufficient subpixel coverage on a high-resolution sampling grid.

[0022] Step 2, based on Construct the direction using equation (1) Cross-frame difference : (1) In equation (1), Indicates direction Intensity difference between adjacent frames, and Indicates along direction Two frames of images showing displacement. , These represent the horizontal, vertical, and main diagonal directions, respectively.

[0023] The direction is obtained using equation (2). Intensity difference map on : (2) In equation (2), Indicates direction The number of frame pairs that are differentially processed; By performing intensity difference analysis on adjacent frames along different displacement directions, a cross-frame intensity difference is constructed. This is used to characterize the local grayscale change trend caused by subpixel displacement. Since the difference responses in the horizontal, vertical, and diagonal directions can reflect the edge and texture structures in different directions of the image, further aggregating multiple difference results in the same direction can yield a more stable intensity difference map. , It can be used as edge prior information in the subsequent deep feature extraction process to guide the network to enhance the directional high-frequency structural response.

[0024] Step 3: Construct a joint demosaic and super-resolution reconstruction network, such as... Figure 2 As shown, it includes: a sequence image integration module, a shallow feature extraction module, a deep feature extraction module, and a reconstruction module, and performs... and Feature extraction and cross-frame fusion processing are performed to obtain a high-resolution color image. ; Unlike the cascaded processing approach that first performs demosaicing and then super-resolution reconstruction, the joint demosaicing and super-resolution reconstruction network in this embodiment directly models the original image sequence containing displacement priors, collaboratively recovering color information and spatial details within a unified framework. This approach reduces the error accumulation problem caused by traditional staged processing and makes fuller use of complementary sampling information between multiple frame sequences.

[0025] like Figure 2As shown, the joint demosaicing and super-resolution reconstruction network operates in the order of "sequence image integration - shallow feature extraction - deep feature extraction - reconstruction output". The sequence image integration module is responsible for mapping multiple frames of original images to a unified high-resolution sampling coordinate system; the shallow feature extraction module is responsible for converting the original sampling structure into a feature representation more suitable for deep network processing; the deep feature extraction module enhances high-frequency information and structural consistency step by step through multiple RDATB blocks; the reconstruction module maps the deep features back to the color high-resolution image; after being combined with the orientation intensity difference map obtained in step 2, the entire network can simultaneously utilize the complementary information of the original sampling and the orientation guidance information to complete the joint reconstruction.

[0026] Step a1: The sequence image integration module integrates the images through sub-pixel convolution. Mapping to target In a high-resolution mesh, preliminary alignment is achieved. Multi-resolution stitched images , and These are the length and width of the stitched image, respectively. It is a multiple.

[0027] Step a2, the shallow feature extraction module will Rearranged into a four-channel representation Then, it is passed through a convolutional layer. The channel dimension is extended Thus, shallow features are obtained. ,in The channel dimension is represented by the fact that the original image is sampled using a Bayer color filter array, with different spatial locations corresponding to different color channel sampling values. Therefore, before entering the deep network, the stitched image is rearranged into a four-channel representation, which can explicitly preserve the sampling period structure of the original color filter array.

[0028] Step a3, the deep feature extraction module consists of... Composed of cascaded RDATB blocks, and for and Process the data to output deep feature sequences. With cross-frame intensity difference sequence ,in, Indicates the first Deep feature maps output from cascaded RDATB blocks Indicates the first Cross-frame intensity difference map output by cascaded RDATB blocks Indicates the number of RDATB blocks.

[0029] Each RDATB block in the deep feature extraction module consists of an STL module, a direction-aware wavelet enhancement module, and a structure-aware convolutional kernel attention module. The first RDATB block uses the orientation intensity difference map obtained in step 2 as initial guidance information, and the second to third RDATB blocks... Each RDATB block then uses the cross-frame intensity difference map output by the previous RDATB block to continue guiding, thus forming a cross-layer propagation direction prior refinement mechanism; Step a3.1, when At that time, the first The STL module in each RDATB block Processing is performed to obtain the first... One reconstructed feature map ; Will and Enter the first The direction-aware wavelet enhancement module in the first RDATB block processes the data to obtain the first... Wavelet Enhancement Feature Map With the Intensity difference map across frames ; Will Enter the first The structure-aware convolutional kernel attention module in the first RDATB block is processed to obtain the first... Deep feature map .

[0030] Step a3.2, when At that time, the first Deep feature map Compared with the cross-frame intensity difference map refined by the previous module Enter the number Processing is performed in the RDATB block to obtain the first... Deep feature map With cross-frame intensity difference map .

[0031] STL modules such as Figure 3 As shown, the following steps were taken to obtain the first... One reconstructed feature map : The STL module first establishes the correlation between different locations within a local window through multi-head self-attention operations. The reason for using windowed self-attention instead of global self-attention is that, on the one hand, a large amount of key structural information in industrial images is usually concentrated in local edges, textures and repeating patterns, and the window mechanism can effectively capture their local correlations; on the other hand, window partitioning can significantly reduce computational complexity and improve training and inference efficiency.

[0032] Step a3.3, using equation (3) to... Feature extraction is performed to obtain the first... Intermediate feature maps ; (3) In equation (3), For multi-head self-attention operations in windows, For the first normalization layer, This indicates a residual connection.

[0033] Step a3.4, using equation (4) to... Feature extraction is performed to obtain the first... One reconstructed feature map ; (4) In equation (4), For the second normalization layer, Indicates residual connection, The feedforward transform network Conv-MLP includes: wavelet decomposition unit, low-frequency subband channel mixing unit, high-frequency subband channel mixing unit, and wavelet reconstruction unit.

[0034] The feedforward transform network Conv-MLP does not directly perform uniform channel mixing on all features. Instead, it first divides the input features into low-frequency and high-frequency sub-bands using wavelet decomposition. This is because the low-frequency sub-band mainly contains the main outline, brightness distribution, and gradually varying structures of the image, making it more suitable for channel-level statistical relationship modeling using linear layers; while the high-frequency sub-band mainly contains edge, texture, and detail variations, making it more suitable for preserving local spatial neighborhood information using convolution. This approach reduces the overhead of mixing tokens across the entire resolution in traditional transformer MLPs. The computational complexity is given by N, where N represents the number of tokens and C represents the number of channels, such as... Figure 4 As shown, the following steps were taken to obtain the first... One reconstructed feature map .

[0035] Step a3.4.1: Wavelet decomposition units utilize two-dimensional discrete wavelet transform... Mapped to four feature subbands, including: Low-frequency subband , No. A set of high-frequency subbands .

[0036] Step a3.4.2, the low-frequency subband channel mixing unit uses equations (6)-(8) to... Processing is performed to obtain the first... Low-frequency reconfiguration subband : (5) (6) In equations (5) and (6), for The k-th low-frequency subband token sequence obtained by flattening This indicates a rearrangement operation. For the first linear layer, This is the GeLU activation function. For the second linear layer, This represents the inverse operation of the rearrangement operation.

[0037] Step a3.4.3, High-frequency subband channel mixing unit for the first A set of high-frequency subbands Processing is performed to obtain the first... A set of high-frequency reconstructed subbands : (7) (8) (9) In equations (7)-(9), and These are the k-th first intermediate output feature and the k-th second intermediate output feature, respectively. The first convolutional kernel is Convolution mapping operation, This is a channel-wise depthwise convolution operation. For GELU activation function, The second kernel is The convolution mapping operation.

[0038] Step a3.4.4, Wavelet Reconstruction Unit Pair and After performing the inverse wavelet transform, then with Perform residual joins to obtain the first... One reconstructed feature map ; Direction-aware wavelet enhancement module, such as Figure 5 As shown, the following steps were taken to obtain the first... Wavelet Enhancement Feature Map intensity difference map across frames k .

[0039] Step a3.5, for the first One reconstructed feature map Perform a two-dimensional discrete wavelet transform to obtain the first... Low-frequency subband With the Horizontal high-frequency sub-band , No. Vertical high-frequency subband , No. High-frequency sub-bands in the main diagonal direction ; By decomposing the reconstructed feature map into a low-frequency sub-band and multiple high-frequency sub-bands using two-dimensional discrete wavelet transform, structural information and detail information can be processed separately. The low-frequency sub-band mainly reflects the overall structure of the image, while the high-frequency sub-bands mainly reflect edge and texture changes. This processing is beneficial for subsequent targeted enhancement of high-frequency information without significantly damaging the stability of the low-frequency structure.

[0040] Step a3.6: Construct a direction prediction branch consisting of two convolutional layers, a ReLU activation function, and a Sigmoid activation function, and then... Process and generate the first Horizontal attention map , No. Vertical attention map , No. Attention diagram in the main diagonal direction ; The orientation prediction branch learns orientation-related response strengths from the current reconstructed feature map through a lightweight convolutional structure and outputs three orientation attention maps. Compared with directly using fixed orientation filters, this approach can adaptively generate different orientation emphasis levels according to the input content, enabling the module to still have good adaptability when complex textures and multi-directional edges coexist.

[0041] Step a3.7, and After weighting, the first modulated number is obtained. Horizontal high-frequency sub-band ,Will and After weighting, the first modulated number is obtained. Vertical high-frequency subband ,Will and After weighting, the first modulated number is obtained. High-frequency sub-bands in the main diagonal direction The attention map generated by the direction prediction branch can be used to preliminarily modulate the high-frequency components in the horizontal, vertical and diagonal directions to highlight the response components that are consistent with the current structural direction.

[0042] Step a3.8: Apply convolution kernels to the first... -1 cross-frame intensity difference map Divided into the first Horizontal edge prior feature map , No. Prior feature maps of vertical edges , No. Prior feature maps of edges along the main diagonal directions .

[0043] Step a3.9: Using equations (10)-(12), we obtain the first... Modulation coefficients in each horizontal direction , No. Modulation coefficients in each vertical direction , No. Modulation coefficients in each main diagonal direction : (10) (11) (12) Modulation coefficients are constructed using the cosine similarity between the high-frequency subband and the prior at the directional edge. This allows for a measurement of the degree of matching between the current high-frequency response and the prior structure from the perspective of directional consistency. When the directional features of the two are highly consistent, the corresponding modulation coefficients are larger, thereby enhancing the detail recovery in that direction. Conversely, high-frequency responses inconsistent with the edge prior are suppressed to reduce noise and false detail propagation.

[0044] Step a3.10: Using equations (13)-(15), we obtain the first... Horizontally weighted high-frequency sub-bands , No. Vertically weighted high-frequency sub-bands , No. Each main diagonal weighted high-frequency sub-band ; (13) (14) (15) Step a3.11: Apply the activation function of two convolutional layers to the first... Low-frequency subband The process is performed to obtain the low-frequency subband after feature enhancement for the kth feature. .

[0045] Step a3.12, update the , , After recombination, the result is processed through a convolutional layer to obtain the... Intensity difference map across frames It serves as the edge guide input for the next layer's RDATB block, enabling the network to continuously refine its directional prior expression capabilities during layer-by-layer propagation.

[0046] Step a3.13, for and Perform inverse wavelet transform to obtain the inverse wavelet transform feature map. Thus, by using equation (16), the first... Wavelet Enhancement Feature Map ; (16) In equation (16), This indicates a residual connection.

[0047] Structure-aware convolutional kernel attention module, such as Figure 6 As shown, the following steps were taken to obtain the first... Deep feature map : like Figure 6 As shown, the structure-aware convolutional kernel attention module does not use the same fixed convolutional kernel for all spatial locations. Instead, it dynamically generates the corresponding convolutional kernel weights based on the input features at the current location. The reason for this is that smooth regions, unidirectional edge regions, and complex texture regions have different requirements for neighborhood information. If a uniform convolutional kernel is used, it is easy to over-enhance smooth regions or under-represent complex regions.

[0048] Step a3.14: Construct a dynamic convolutional kernel weight generation branch consisting of convolution and ReLU activation functions, and then... Processing is performed to obtain each pixel and its surrounding pixels. The k-th dynamic kernel weight corresponding to the region pixel , .

[0049] Step a3.15: Apply convolution kernels to... The convolution operation will The number of channels is compressed to Then, a neighborhood unfolding operation is performed to construct the k-th local neighborhood feature set corresponding to each pixel position. , ; This represents the number of channels after compression.

[0050] Step a3.16, for and After performing position-by-position weighted summation, the k-th aggregated feature map is obtained. After convolution kernel The convolution operation will Mapping back to the original channel dimension C yields the k-th deep feature map. .

[0051] Step a4: The reconstruction module uses convolutional layers to... Channel dimension Compress to Then, subpixel convolution is used to upsample the compressed deep feature sequence to obtain a high-resolution result image. .

[0052] Step 4: Construct the mean absolute error loss function using equation (17) : (17) Step 5: Iteratively train the joint demosaicing and super-resolution reconstruction network using the backpropagation algorithm, and calculate the mean absolute error loss function. Adjust the network parameters until the mean absolute error loss function is reached. The process continues until convergence, resulting in a trained joint demosaic and super-resolution reconstruction model. This model is then used to process the input original image sequence and obtain the optimal reconstructed color high-resolution image.

[0053] In this embodiment, an electronic device includes a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing any of the controllable pixel displacement driven joint demosaic and super-resolution reconstruction methods described above, and the processor is configured to execute the program stored in the memory.

[0054] In this embodiment, a computer-readable storage medium stores a computer program, characterized in that the computer program, when run by a processor, executes the steps of the image joint demosaicing and super-resolution reconstruction method driven by any of the controllable pixel displacements described above.

[0055] The effects of the present invention will be further illustrated below with experimental results: The proposed pixel-shifting image super-resolution network employs a PyTorch architecture and is trained using a single NVIDIA RTX 4090 DGPU. The optimizer uses momentum decay rates of [missing information - likely 1] and [missing information - likely 2]. and Adam. and During training, low-resolution and high-resolution images are cropped separately. and and and The batch size is set to 32. The initial learning rate is set to... The learning rate reaches the target number of training steps. Reduce by half.

[0056] To evaluate the performance of this invention in joint demosaicing and super-resolution, it was compared with several state-of-the-art methods. These methods can be divided into two categories: those that perform demosaicing and super-resolution tasks sequentially, and those that perform joint demosaicing and super-resolution reconstruction. The sequential demosaicing and super-resolution task method can be further divided into single-frame and multi-frame methods based on the number of input frames for the super-resolution task. Through comparison with various state-of-the-art algorithms, this invention highlights the advantage of image information richness brought about by sequential image reconstruction under fixed subpixel displacement. All algorithms were tested on the Set5, Mcm, B100, and Urban100 datasets regenerated using simulated pixel displacement. Furthermore, this invention uses Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as evaluation metrics to comprehensively evaluate the joint demosaicing and super-resolution processing results of sequential images.

[0057] This invention significantly outperforms other methods in reconstructing images based on two key image quality metrics: peak signal-to-noise ratio and structural similarity. This demonstrates that the invention can more effectively improve color accuracy, detail sharpness, and overall structural similarity, thereby generating results that more closely resemble the original high-resolution image. Detailed results are shown in Table 1.

[0058] In Table 1, Kokkinos is the demosaicing method disclosed in the non-patent document "Deep Image Demosaicking Using a Cascade of Convolutional Residual Denoising Networks"; RSTCANet is the demosaicing method disclosed in the non-patent document "Residual Swin Transformer Channel Attention Network for Image Demosaicing"; MAN is the single-image super-resolution method disclosed in the non-patent document "Multi-Scale Attention Network for Single Image Super-Resolution"; SwinIR is the single-image super-resolution method disclosed in the non-patent document "SwinIR: Image Restoration Using Swin Transformer"; TENet is the single-image joint demosaicing and super-resolution method disclosed in the non-patent document "Rethinking Learning-Based Demosaicing, Denoising, and Super-Resolution Pipeline"; and TSCNN-L is the non-patent document "A Two-Stage Convolutional Neural Network for Joint Demosaicking and..." The method disclosed in "Super-Resolution" is a single-image joint demosaic and super-resolution method; ASHSR is a multi-frame image super-resolution method based on active displacement disclosed in the non-patent literature "Super-Resolution Reconstruction of Sequential Images Based on an ActiveShift via a Hybrid Attention Calibration Mechanism"; the method of this invention is a controllable pixel displacement driven joint demosaic and super-resolution reconstruction method. Here, "+" indicates that the corresponding demosaic method and super-resolution reconstruction method are cascaded sequentially in a comparison manner, with demosaic first and then super-resolution reconstruction.

[0059] Table 1 Comparison of Quantization Metrics for Reconstructed Images from Public Datasets

[0060] Figure 7The visual effects of the joint demosaicing and super-resolution reconstruction network based on the original image sequence of this invention are compared with other methods. The real high-resolution images used are selected from the Urban100 dataset. Figure 7 Parts (a) and (b) in the figure demonstrate the different algorithms at a magnification scale of [missing information]. and The corresponding low-resolution image reconstruction results are shown in the figure. It is clear from the figure that the high-resolution image reconstructed by this invention has realistic colors, complete structure, and clear details. The reconstruction quality is far superior to other algorithms and is closer to a real high-resolution image. This invention reduces the errors introduced during the two-stage task through end-to-end de-mosaic and super-resolution processing, while also suppressing false colors. By introducing a fixed sub-pixel displacement sequence image, the network model can utilize richer image information, effectively avoiding the generation of unrealistic textures.

Claims

1. A method for joint demosaicing and super-resolution reconstruction driven by controllable pixel displacement, characterized in that, Includes the following steps: Step 1: Construct an image acquisition system consisting of a two-dimensional piezoelectric displacement stage, a controller, and an image sensor to acquire the original image sequence containing displacement priors. ,in, Indicates the first Frame low-resolution image, This represents the total number of low-resolution images, and ; Indicates the ratio of a high-resolution image to a low-resolution image; Step 2, based on Construct the direction using equation (1) Cross-frame difference : (1) In equation (1), Indicates direction Cross-frame intensity difference between adjacent low-resolution images, and Indicates along direction Two low-resolution images of displacement. , These represent the horizontal, vertical, and main diagonal directions, respectively. The direction is obtained using equation (2). Intensity difference map on : (2) In equation (2), Indicates direction The number of frame pairs that are differentially processed; Step 3: Construct a joint demosaicing and super-resolution reconstruction network, including: a sequence image integration module, a shallow feature extraction module, a deep feature extraction module, and a reconstruction module, and then... and Feature extraction and cross-frame fusion processing are performed to obtain a high-resolution color image. ; Step 3.1: The sequence image integration module integrates the images through sub-pixel convolution. Mapping to target In a high-resolution mesh, this allows for initial alignment. Multi-resolution stitched images , and These are the length and width of the stitched image, respectively. Multiples; Step 3.2, the shallow feature extraction module will... Rearranged into a four-channel representation Then, it is passed through a convolutional layer. The channel dimension is extended Thus, shallow features are obtained. ,in, Indicates the channel dimension; Step 3.3, the deep feature extraction module consists of... Composed of cascaded RDATB blocks, and for and Process the data to output deep feature sequences. With cross-frame intensity difference sequence ,in, Indicates the first Deep feature maps output from cascaded RDATB blocks Indicates the first Cross-frame intensity difference map output by cascaded RDATB blocks Indicates the number of RDATB blocks; Step 3.4: The reconstruction module uses convolutional layers to... Channel dimension Compress to Then, subpixel convolution is used to upsample the compressed deep feature sequence to obtain a high-resolution result image. ; Step 4: Construct the mean absolute error loss function using equation (3) : (3) Step 5: Iteratively train the joint demosaicing and super-resolution reconstruction network using the backpropagation algorithm, and calculate the mean absolute error loss function. Adjust the network parameters until the mean absolute error loss function is reached. The process continues until convergence, resulting in a trained joint demosaic and super-resolution reconstruction model. This model is then used to process the input original image sequence and obtain the optimal reconstructed color high-resolution image.

2. The method for joint demosaicing and super-resolution reconstruction driven by controllable pixel displacement according to claim 1, characterized in that, Each RDATB block in the deep feature extraction module of step 3.3 consists of an STL module, a direction-aware wavelet enhancement module, and a structure-aware convolutional kernel attention module. Step 3.3.1, when At that time, the first The STL module in each RDATB block Processing yields the first... One reconstructed feature map ; Will and Enter the number respectively The direction-aware wavelet enhancement module in the first RDATB block processes the data to obtain the first... Wavelet Enhancement Feature Map With the Intensity difference map across frames ; Will Enter the first The structure-aware convolutional kernel attention module in the first RDATB block is processed to obtain the first... Deep feature map ; Step 3.3.2, when At that time, the first Deep feature map With the Intensity difference map across frames Enter the number Processing is performed in the RDATB block to obtain the first... Deep feature map With cross-frame intensity difference map Thus, by the first The output of the RDATB block is the first Deep feature map and the Deep feature map With cross-frame intensity difference map .

3. The method for joint demosaicing and super-resolution reconstruction driven by controllable pixel displacement according to claim 2, characterized in that, The STL module in step 3.3.1 is obtained by following these steps: One reconstructed feature map : Step a1, using equation (4) Feature extraction is performed to obtain the first... Intermediate feature maps ; (4) In equation (4), For multi-head self-attention operations in windows, For the first normalization layer, Indicates residual connection; Step a2, using equation (5) Feature extraction is performed to obtain the first... One reconstructed feature map ; (5) In equation (5), For the second normalization layer, Indicates residual connection, It is a feedforward transform network.

4. The method for joint demosaicing and super-resolution reconstruction driven by controllable pixel displacement according to claim 3, characterized in that, The feedforward transform network in step a2 It includes: wavelet decomposition unit, low-frequency subband channel mixing unit, high-frequency subband channel mixing unit, and wavelet reconstruction unit, and obtains the following steps: One reconstructed feature map : Step a2.1: Wavelet decomposition units utilize two-dimensional discrete wavelet transform... Mapped to four feature subbands, including: Low-frequency subband and the A set of high-frequency subbands ; Step a2.2, the low-frequency subband channel mixing unit uses equations (6)-(7) to... Processing yields the first... Low-frequency reconfiguration subband : (6) (7) In equations (6) and (7), for The k-th low-frequency subband token sequence obtained by flattening This indicates a rearrangement operation. For the first linear layer, This is the GeLU activation function. For the second linear layer, This represents the inverse operation of rearrangement; Step a2.3: The high-frequency subband channel mixing unit uses equations (8)-(10) to process the first... A set of high-frequency subbands Processing yields the first... A set of high-frequency reconstructed subbands : (8) (9) (10) In equations (8)-(10), and These are the k-th first intermediate output feature and the k-th second intermediate output feature, respectively. The first convolutional kernel is Convolution mapping operation, This is a channel-wise depthwise convolution operation. For GELU activation function, The second kernel is Convolutional mapping operations; Step a2.4, the wavelet reconstruction unit... and After performing the inverse wavelet transform, then with After performing residual join, the first... One reconstructed feature map .

5. The method for joint demosaicing and super-resolution reconstruction driven by controllable pixel displacement according to claim 2, characterized in that, The direction-aware wavelet enhancement module in step 3.3.1 is obtained by following these steps: Wavelet Enhancement Feature Map intensity difference map with the k-th cross-frame : Step b1, for Perform a two-dimensional discrete wavelet transform to obtain the first... Low-frequency subband With the Horizontal high-frequency sub-band , No. Vertical high-frequency subband , No. High-frequency sub-bands in the main diagonal direction ; Step b2: Construct a direction prediction branch consisting of two convolutional layers, a ReLU activation function, and a Sigmoid activation function, and then... Process and generate the first Horizontal attention map , No. Vertical attention map , No. Attention diagram in the main diagonal direction ; Step b3, and After weighting, the first modulated number is obtained. Horizontal high-frequency sub-band ,Will and After weighting, the first modulated number is obtained. Vertical high-frequency subband ,Will and After weighting, the first modulated number is obtained. High-frequency sub-bands in the main diagonal direction ; Step b4: Apply convolution kernels to the first... -1 cross-frame intensity difference map Divided into the first Horizontal edge prior feature map , No. Prior feature maps of vertical edges , No. Prior feature maps of edges along the main diagonal directions ; Step b5: Using equations (11)-(13) respectively, obtain the first... Modulation coefficients in each horizontal direction , No. Modulation coefficients in each vertical direction , No. Modulation coefficients in each main diagonal direction : (11) (12) (13) In equations (11)-(13), Indicates the activation function; Step b6: Using equations (14)-(16) respectively, obtain the first... Horizontally weighted high-frequency sub-bands , No. Vertically weighted high-frequency sub-bands , No. Each main diagonal weighted high-frequency sub-band ; (14) (15) (16) Step b7: Using two convolutional layers and the ReLU activation function, sequentially apply the following to the first... Low-frequency subband The process is performed to obtain the low-frequency subband after feature enhancement for the kth feature. ; Step b8, , , After concatenation, the result is obtained by a convolutional layer. Intensity difference map across frames ; Step b9, for and Perform inverse wavelet transform to obtain the inverse wavelet transform feature map. Thus, the first equation is obtained using equation (17). Wavelet Enhancement Feature Map ; (17) In equation (17), This indicates a residual connection.

6. The method for joint demosaicing and super-resolution reconstruction driven by controllable pixel displacement according to claim 2, characterized in that, The structure-aware convolutional kernel attention module in step 3.3.1 is obtained according to the following steps: Deep feature map ; Step c1: Construct a dynamic convolution kernel weight generation branch consisting of convolution and ReLU activation functions, and then... Processing is performed to obtain all pixels and the surrounding pixels of each pixel. The k-th dynamic kernel weight sequence corresponding to each region pixel ; Step c2, using convolution kernels for The convolution operation will The number of channels is compressed to Then, a neighborhood unfolding operation is performed to construct the k-th local neighborhood feature set corresponding to all pixel positions. , ; This represents the number of channels after compression. Step c3, for and After performing weighted summation at each pixel position, the k-th aggregated feature map is obtained. Then, after passing through the convolution kernel, it becomes... The convolution operation will Mapping back to the original channel dimension C yields the k-th deep feature map. .

7. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the joint demosaicing and super-resolution reconstruction method driven by any of claims 1-6, the processor being configured to execute the program stored in the memory.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the joint demosaic and super-resolution reconstruction method driven by any one of claims 1-6.

Images