Texture perception state space modeling method for image restoration task

Abstract

The invention discloses a texture perception state space modeling method for an image restoration task, and the method comprises the steps: 1, constructing a region selection mechanism based on texture complexity, and enabling the region selection mechanism to be used for distinguishing a flat region and a high-texture region in an image; 2, introducing a texture modulation mechanism, and performing explicit adjustment on a state transition matrix in the state space model; 3, enhancing the context modeling capability of the model through a multi-direction sensing module; and 4, by combining position embedding and a sequence modeling structure, the capability of the model in the aspects of image structure understanding and spatial information maintenance is improved. The method can effectively alleviate the problem of information loss when a traditional image restoration method processes texture details, improves the structure restoration capability of a complex region, gives consideration to the restoration quality and the calculation efficiency, is suitable for multiple image restoration scenes such as image super-resolution, image rain removal, low-light image enhancement and the like, and improves the image restoration efficiency. And the method has good engineering adaptability and actual deployment value.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and specifically to an image restoration method based on state-space modeling, in particular to an image restoration model with a texture perception mechanism and a multi-directional perception structure, which is suitable for tasks such as image super-resolution, image deraining, and low-light image enhancement. Background Art

[0002] The image restoration task aims to restore image quality loss caused by limited imaging conditions, sensor noise interference, or visual degradation processes. It has wide application value in scenarios such as security monitoring, medical imaging, and industrial inspection. The current mainstream methods mainly rely on convolutional neural networks or Transformer structures. The former has a limited receptive field and has difficulty capturing long-range dependencies. Although the latter has global modeling capabilities, it has high computational costs and is not friendly to high-resolution image processing. In addition, most image restoration methods default to a unified modeling strategy for all areas in the image, ignoring the non-uniformity of the image content structure in terms of texture complexity. High-texture areas often contain more details, are easily damaged, and are significantly more difficult to restore than flat areas. Therefore, traditional homogenized processing methods are difficult to strike a balance between efficiency and quality in real scenarios. Summary of the Invention

[0003] The present invention aims to solve the problems of existing image restoration methods, such as insufficient restoration of texture areas, limited global structure perception ability and low computational efficiency. A texture-aware state-space modeling method for image restoration tasks is proposed, in order to focus on modeling complex texture areas and improve the overall image reconstruction quality while maintaining modeling efficiency.

[0004] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0005] The texture perception state space modeling method for image restoration tasks of the present invention is characterized in that it includes the following steps:

[0006] Step 1: Filter and sort the image blocks in the input image pig based on texture complexity to obtain the texture complex area of ​​the input image pig;

[0007] Step 2: Based on the complex texture area, construct the A state-space model of texture modulation with time steps;

[0008] Step 3: Use formula (5) to get Image patch features fused with position information at each time step :

[0009] (5)

[0010] In formula (5), For the The two-dimensional position encoding vector to be learned in time steps; For the Image patch features of time steps;

[0011] Step 4: Based on , get the Image patch modeling features after fusion of time steps ;

[0012] Step 5: Normalization, spatial attention enhancement and convolution mapping are performed in sequence to obtain the first High-resolution reconstructed image patches , thus obtaining the reconstructed image of the input image pig.

[0013] The texture-aware state-space modeling method for image restoration tasks described in the present invention is also characterized in that step 1 includes the following steps:

[0014] Step 1.1: Divide the input image pig into several image blocks of the same size. The image blocks are denoted as , and use formula (1) to calculate The average brightness value :

[0015] (1)

[0016] In formula (1), express The grayscale or eigenvalue of the j-th pixel in ; express The total number of pixels in the

[0017] Step 1.2: Calculate using formula (2) Texture complexity :

[0018] (2)

[0019] Step 1.3, sort all image blocks in descending order according to the texture complexity of each image block, and select the image blocks before sorting. The image blocks are used as the texture complex areas of the input image pig.

[0020] Furthermore, step 2 includes the following steps:

[0021] Step 2.1: Place the first Image blocks Input two sets of linear neural network layers for processing, and obtain two linear features, which are respectively compared with the first Image blocks After multiplying the texture complexity of The input mapping matrix of time steps Pass the exam The state transition scaling factor for each time step ; ;

[0022] Step 2.2: Use formula (3) to calculate the modulated The state transition matrix of time steps :

[0023] (3)

[0024] In formula (4), A is the predefined basic state transfer matrix, Indicates element-by-element exponential mapping of the matrix;

[0025] Step 2.3: Use formula (4) to construct State-space model of texture modulation with time steps:

[0026] , (4)

[0027] In formula (5), Indicates the The hidden state of time steps, Indicates the The hidden state of the time step, when k=1, initialize , and are the 2 output mapping weights, For the Image patch features at each time step.

[0028] Furthermore, step 4 includes the following steps:

[0029] Step 4.1, according to the four scanning directions Input formula (5), and the corresponding modeling features in the horizontal direction are obtained , vertical modeling features , Modeling features in the main diagonal direction Modeling features in the sub-diagonal direction ;

[0030] Step 4.2: Use formula (6) to get Image patch modeling features after fusion of time steps :

[0031] (6).

[0032] Furthermore, step 5 includes the following steps:

[0033] Step 5.1, Perform layer normalization to obtain the normalized Image patch modeling features ;

[0034] Step 5.2: Use formula (7) to get Attention response map :

[0035] (7)

[0036] In formula (8), and They are spatial average pooling and maximum pooling operations respectively; is the convolution operation, Represented as sigmoid activation function;

[0037] Step 5.3: Calculate the attention-enhanced Image patch features ;

[0038] Step 5.4, Perform convolution mapping to obtain the High-resolution reconstructed image patches ;

[0039] Step 5.5, After splicing the high-resolution reconstructed image blocks, the reconstructed image of the input image pig is obtained.

[0040] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the texture perception state space modeling method, and the processor is configured to execute the program stored in the memory.

[0041] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the texture-aware state-space modeling method when executed by a processor.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention divides the input image or intermediate feature map into several image blocks and uses local variance as a measure of texture complexity to screen out high-texture areas, thereby achieving accurate identification and focusing on areas with rich information and difficult detail restoration, thereby bringing about efficient utilization of model resources and significantly improved detail restoration capabilities.

[0044] 2. The present invention adopts a state-space modeling structure for high-texture areas and introduces texture modulation parameters to achieve dynamic adjustment of the state transition amplitude, thereby bringing about adaptive regulation of the model on the complexity of different areas, effectively improving the detail recovery effect and alleviating problems such as overfitting.

[0045] 3. By introducing spatial position encoding, the present invention adds a learnable embedding vector corresponding to the two-dimensional coordinates of each image block, thereby achieving effective fusion of spatial structure information, thereby bringing better spatial consistency and global structure understanding capabilities.

[0046] 4. The present invention realizes comprehensive modeling of multi-angle structural information by adopting four main direction modeling paths, namely horizontal, vertical and main and secondary diagonals, in the state space modeling process, thereby bringing better restoration effects of complex structures and edge details.

[0047] 5. The present invention combines texture complexity guidance, state modulation, spatial position encoding and multi-directional modeling, and weightedly fuses the texture modeling branch with the full-image convolution branch to achieve balanced restoration of high and low texture areas, thereby bringing about a comprehensive improvement in the overall image restoration quality and engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the framework of the image enhancement method based on texture perception state space constructed by the present invention;

[0049] Figure 2 Schematic diagram of the multi-directional structure perception mechanism proposed in this invention. DETAILED DESCRIPTION

[0050] In this embodiment, a texture perception state space modeling method for image restoration tasks is provided. Figure 1 As shown, the following steps are included:

[0051] Step 1: Filter and sort the image blocks in the input image pig based on texture complexity to obtain the texture complex area of ​​the input image pig:

[0052] Step 1.1: Divide the input image pig into several image blocks of the same size. The image blocks are denoted as , and use formula (1) to calculate The average brightness value :

[0053] (1)

[0054] In formula (1), express The grayscale or eigenvalue of the j-th pixel in ; express The total number of pixels in the

[0055] Step 1.2: Calculate using formula (2) Texture complexity :

[0056] (2)

[0057] Step 1.3, sort all image blocks in descending order according to the texture complexity of each image block, and select the image blocks before sorting. The image blocks are used as the texture complex areas of the input image pig.

[0058] Step 2: Based on the results of step 1 A texture-complex image block is constructed to build a state space modeling mechanism for texture modulation, and feature modeling and memory retention are performed on the selected texture area image block:

[0059] Step 2.1: Place the first Image blocks Input two sets of linear neural network layers for processing, and obtain two linear features, which are respectively compared with the first Image blocks After multiplying the texture complexity of The input mapping matrix of time steps Pass the exam The state transition scaling factor for each time step ; :

[0060] , (3)

[0061] in, Indicates the texture complexity of the corresponding image block, obtained by step 1; , are two independent sets of linear mapping functions.

[0062] Step 2.2: Use the scale factor and the basic state transfer matrix , use formula (4) to calculate the state transfer matrix after modulation :

[0063] (4)

[0064] In formula (4), A is the predefined basic state transfer matrix, Represents element-by-element exponential mapping of a matrix.

[0065] Step 2.3: Use formula (5) to construct State-space model of texture modulation with time steps :

[0066] , (5)

[0067] In formula (5), Indicates the The hidden state of time steps, Indicates the The hidden state of the time step, when k=1, initialize , and are the 2 output mapping weights, For the Image patch features at each time step.

[0068] Step 3: Use formula (6) to get Image patch features fused with position information at each time step , position encoding is introduced to enhance its spatial context modeling capability, as follows:

[0069] (6)

[0070] In formula (6), For the The two-dimensional position encoding vector to be learned at each time step.

[0071] Step 4: Based on , perform state space modeling along multiple spatial directions to improve the modeling ability of texture structure and obtain the first Image patch modeling features after fusion of time steps ;

[0072] Step 4.1, according to the four scanning directions Input formula (5), and the corresponding modeling features in the horizontal direction are obtained , vertical modeling features , Modeling features in the main diagonal direction Modeling features in the sub-diagonal direction As shown in formula (7) and formula (8);

[0073] , (7)

[0074] , (8)

[0075] in, represents the state space modeling, as defined in Equation (5). Figure 2 As shown in , four different scanning directions can fully enhance the network's ability to perceive image context, thereby improving the ability to repair.

[0076] Step 4.2: Use formula (9) to get Image patch modeling features after fusion of time steps :

[0077] (9)

[0078] Step 5: Normalization, spatial attention enhancement and convolution mapping are performed in sequence to obtain the first High-resolution reconstructed image patches , thus obtaining the reconstructed image of the input image pig:

[0079] Step 5.1: Use formula (10) to fusion features of each image block Perform layer normalization to enhance feature stability and eliminate inter-channel offsets to obtain a normalized representation:

[0080] (10)

[0081] Step 5.2: Normalize the features Input spatial attention module to model salient regions. This module calculates the average pooling and maximum pooling results of image blocks in spatial dimensions and performs convolution fusion to generate attention response maps. As shown in formula (11), it is used to highlight the salient areas of texture and structure:

[0082] (11)

[0083] In formula (11), and They are spatial average pooling and maximum pooling operations respectively; is the convolution operation, Represented as sigmoid activation function.

[0084] Step 5.3: Calculate the attention-enhanced Image patch features :

[0085] Step 5.4, Perform convolution mapping to obtain the High-resolution reconstructed image patches As shown in formula (12):

[0086] (12)

[0087] Step 5.5, After splicing the high-resolution reconstructed image blocks, the reconstructed image of the input image pig is obtained.

[0088] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0089] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0090] To verify the effectiveness of the method presented in this example, training was performed on the DIV2K dataset and testing was performed on two standard image restoration datasets, Set5 and Set14. The test datasets contain a variety of typical natural scenes, covering both highly textured and flat areas. The test results were evaluated using two metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR measures the overall pixel restoration of an image, while SSIM measures the structural similarity of the image. Four mainstream image restoration methods were selected for comparison with the method presented in this example: Enhanced Deep Residual Networks for Single Image Super-Resolution (EDSR), Image Super-Resolution Using Very Deep Residual Channel Attention Networks (RCAN), and SwinIR: Image Restoration Using Swin Transformer (SwinIR). All methods used 75% of the images for training and 25% for testing. The experimental results are shown in Table 1.

[0091] Table 1 Comparison of the results of the proposed method and the three selected comparison methods on the Set5 and Set14 datasets

[0092]

[0093] From the above experiments, it can be seen that the method of the present invention has achieved optimal performance on multiple public image restoration datasets, is significantly better than existing methods in restoring details in high-texture areas, and has strong engineering practical value.

Claims

1. A texture-aware state-space modeling method for image restoration tasks, characterized by: The following steps are involved: Step 1: Filter and sort the image blocks in the input image pig based on texture complexity to obtain the texture complex area of ​​the input image pig; Step 2: Based on the complex texture area, construct the A state-space model of texture modulation with time steps; Step 3: Use formula (5) to get Image patch features fused with position information at each time step : (5) In formula (5), For the The two-dimensional position encoding vector to be learned in time steps; For the Image patch features of time steps; Step 4: Based on , get the Image patch modeling features after fusion of time steps ; Step 5: Normalization, spatial attention enhancement and convolution mapping are performed in sequence to obtain the first High-resolution reconstructed image patches , thus obtaining the reconstructed image of the input image pig.

2. A texture-aware state-space modeling method for image restoration tasks according to claim 1, characterized in that: Step 1 includes the following steps: Step 1.1: Divide the input image pig into several image blocks of the same size. The image blocks are denoted as , and use formula (1) to calculate The average brightness value : (1) In formula (1), express The grayscale or eigenvalue of the j-th pixel in ; express The total number of pixels in the Step 1.2: Calculate using formula (2) Texture complexity : (2) Step 1.3, sort all image blocks in descending order according to the texture complexity of each image block, and select the image blocks before sorting. The image blocks are used as the texture complex areas of the input image pig.

3. A texture-aware state-space modeling method for image restoration tasks according to claim 2, characterized in that: Step 2 includes the following steps: Step 2.1: Place the first Image blocks Input two sets of linear neural network layers for processing, and obtain two linear features, which are respectively compared with the first Image blocks After multiplying the texture complexity of The input mapping matrix of time steps Pass the exam The state transition scaling factor for each time step ; ; Step 2.2: Use formula (3) to calculate the modulated The state transition matrix of time steps : (3) In formula (4), A is the predefined basic state transfer matrix, Indicates element-by-element exponential mapping of the matrix; Step 2.3: Use formula (4) to construct State-space model of texture modulation with time steps: , (4) In formula (5), Indicates the The hidden state of time steps, Indicates the The hidden state of the time step, when k=1, initialize , and are the 2 output mapping weights, For the Image patch features at each time step.

4. A texture-aware state-space modeling method for image restoration tasks according to claim 3, characterized in that: Step 4 includes the following steps: Step 4.1, according to the four scanning directions Input formula (5), and the corresponding modeling features in the horizontal direction are obtained , vertical modeling features , Modeling features in the main diagonal direction Modeling features in the sub-diagonal direction ; Step 4.2: Use formula (6) to get Image patch modeling features after fusion of time steps : (6)。 5. A texture-aware state-space modeling method for image restoration tasks according to claim 4, characterized in that: Step 5 includes the following steps: Step 5.1, Perform layer normalization to obtain the normalized Image patch modeling features ; Step 5.2: Use formula (7) to get Attention response map : (7) In formula (8), and They are spatial average pooling and maximum pooling operations respectively; is the convolution operation, Represented as sigmoid activation function; Step 5.3: Calculate the attention-enhanced Image patch features ; Step 5.4, Perform convolution mapping to obtain the High-resolution reconstructed image patches ; Step 5.5, After splicing the high-resolution reconstructed image blocks, the reconstructed image of the input image pig is obtained.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports a processor to execute the texture-aware state-space modeling method according to any one of claims 1 to 5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the texture-aware state-space modeling method according to any one of claims 1 to 5 are performed.

Images