An image inpainting method, device, electronic equipment and storage medium

Abstract

The application provides an image inpainting method and device, electronic equipment and storage medium, and relates to the field of image inpainting. In order to improve the image inpainting quality, a trained feature extraction model can be used to extract features of an image to be inpainted to obtain image features, and the image features can be decoded to obtain an inpainted image. The feature extraction model includes multiple feature extraction blocks. Each feature extraction block can obtain image information from the local and global, and the feature extraction blocks can be densely connected. The feature extraction block of the current level can extract the global features of the current layer based on the local features extracted by the current layer and the global features extracted by the previous layer. The local degraded features and the global repair prior collaborative update can be realized. The information flow between different depth levels can be maximized. The deep features can be generated under the guidance of the global summary of all shallow features, the feature extraction effect can be improved, and the image inpainting quality can be improved.

Description

Technical Field

[0001] This invention relates to the field of image restoration, and in particular to an image restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] In industrial settings, high-resolution cameras often suffer from motion blur, random noise, and reduced contrast in captured images due to mechanical vibrations, high-speed movement on production lines, dust obstruction, or lighting fluctuations. Image restoration is required to eliminate these defects.

[0003] In related technologies, image restoration methods have the drawback of being unable to take into account both local details and global structure of an image, thus only achieving poor restoration results. Summary of the Invention

[0004] The purpose of this invention is to provide an image restoration method, apparatus, electronic device, and storage medium that can improve the extraction effect of image features and achieve the collaborative extraction of local and global features, thereby improving the image restoration effect.

[0005] To address the aforementioned technical problems, this invention provides an image restoration method, comprising: The image to be repaired is input into a trained feature extraction model for feature extraction processing to obtain image features. The feature extraction model includes multi-level feature extraction blocks. Each feature extraction block performs local feature extraction on the feature map group of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are combined with the upper-level sequence output by the upper-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output. The image features are decoded to obtain the repaired image.

[0006] Optionally, the feature extraction block includes a channel splitting layer, a feature extraction layer, a channel splicing layer, and a downsampling layer. The feature extraction layer includes K grouping processing branches. The feature extraction block has a first input terminal, a second input terminal, a first output terminal, and a second output terminal. The first input terminal is connected to the first output terminal of the adjacent upper-level feature extraction block of the feature extraction block. The second input terminal is connected to the second output terminals of all upper-level feature extraction blocks of the feature extraction block. The first output terminal is connected to the first input terminal of the adjacent lower-level feature extraction block of the feature extraction block. The second output terminal is connected to the second input terminal of all lower-level feature extraction blocks of the feature extraction block. The feature extraction block extracts local features from the feature map group of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are then combined with the upper-level sequence output by the previous-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are then fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output, including: The channel splitting layer splits the input feature map group into K sub-feature map groups; The first grouping processing branch outputs the first sub-feature map group to the channel stitching layer and the second grouping processing branch; The i-th group processing branch concatenates the outputs of the i-th sub-feature map group and the (i-1)-th group processing branch to obtain the first concatenated feature. Local features are extracted from the first concatenated feature to obtain local features, and the local features are output to the channel concatenation layer and the (i+1)-th group processing branch; where i∈(1,K); The Kth group processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the upper-layer sequence obtained from the second input terminal to obtain the second concatenated feature. The second concatenated feature is then subjected to selective state-space model sequence calculation to obtain the current layer sequence, and the current layer sequence is output to the channel concatenation layer and the second output terminal. The channel splicing layer splices the outputs of each group processing branch to obtain channel splicing features; The downsampling layer downsamples the channel stitching features to obtain the image features, and outputs them to the first output terminal.

[0007] Optionally, local features are obtained by performing local feature extraction on the first splicing features, including: The first spliced ​​feature is convolved using a convolution kernel of a preset size to obtain the local feature.

[0008] Optionally, the Kth grouping processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th grouping processing branch, and the upper-level sequence obtained from the second input to obtain the second concatenated feature, including: The Kth group processing branch adds the output of the Kth sub-feature map group to the output of the (K-1)th group processing branch, and converts the result into a one-dimensional sequence. The upper-level sequence is concatenated to the prefix position of the one-dimensional sequence.

[0009] The step of selectively calculating the state-space model sequence of the second splicing feature to obtain the current layer sequence includes: The original sequence is obtained by selectively calculating the state-space model sequence of the second splicing feature; The sequence segment of a preset length located at the end of the original sequence is taken as the current layer sequence.

[0010] Optionally, when the second input terminal receives multiple upper-level sequences, the method further includes: A spliced ​​sequence is obtained by concatenating multiple upper-level sequences; The spliced ​​sequence is processed using an attention mechanism to obtain a processed sequence of a preset length; The Kth grouping processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th grouping processing branch, and the upper-level sequence obtained from the second input to obtain the second concatenated feature, including: The Kth group processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the processed sequence to obtain the second concatenated feature.

[0011] Optionally, the channel splitting layer splits the input feature map group into K sub-feature map groups, including: The channel splitting layer performs average channel splitting on the input feature map group to obtain the K sub-feature map groups.

[0012] Optionally, the step of decoding the image features to obtain the repaired image includes: Obtain the first image features output by the penultimate feature extraction block and the second image features output by the last feature extraction block; The second image features are subjected to interpolation and upsampling processing to obtain upsampled features; The first image features and the upsampled features are fused, and the fusion result is subjected to deconvolution upsampling processing to obtain the repaired image.

[0013] The present invention also provides an image restoration apparatus, comprising: The feature extraction module is used to input the image to be repaired into a trained feature extraction model for feature extraction processing to obtain image features. The feature extraction model includes multi-level feature extraction blocks. The feature extraction blocks perform local feature extraction on the feature map group of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are combined with the upper-level sequence output by the upper-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output. The decoding module is used to decode the image features to obtain the repaired image.

[0014] The present invention also provides an electronic device, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement the image restoration method as described above.

[0015] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the image restoration method described above.

[0016] This invention provides an image restoration method, comprising: inputting the image to be restored into a trained feature extraction model for feature extraction processing to obtain image features; wherein, the feature extraction model includes multi-level feature extraction blocks, the feature extraction blocks extract local features from feature maps of the image to be restored at different scales to obtain local features at different scales, selectively state-space modeling the local features at the largest scale and the upper-level sequence output by the upper-level feature extraction block to obtain the current layer sequence, fusing the local features at different scales with the current layer sequence to obtain image features, and outputting the image features and the current layer sequence; and decoding the image features to obtain the restored image.

[0017] The beneficial effects of this invention are as follows: This invention improves the image feature extraction process to enhance image restoration quality. This invention provides a feature extraction model comprising multi-level feature extraction blocks. Each feature extraction block can extract local features from the feature map of the image to be restored at different scales, thereby focusing on local details in the image to be restored according to different scales. Furthermore, the feature extraction block can perform selective state-space model sequence calculations on the largest-scale local features and the upper-level sequence output by the previous-level feature extraction block to obtain the current-layer sequence, thereby determining the global features of the current layer based on the local features collected in the current layer and the global features collected in previous layers. Subsequently, the feature extraction block can fuse local features at different scales with the current-layer sequence to obtain image features and output the image features and the current-layer sequence. As can be seen, the feature extraction block can achieve deep collaboration between local multi-scale features and global selective state-space model features. Furthermore, by densely connecting each feature extraction layer, the current layer receives image features and upper-level sequences output from all preceding layers, and continues to extract local features from the image features. It then combines the extracted local features with the upper-level sequences generated by the preceding layers to generate the current layer sequence. This achieves collaborative updates of "local degradation features - global restoration priors," strengthening long-range dependencies across degradation regions. Simultaneously, it maximizes information flow between different network depths, allowing the generation of deep features to be guided by the global summary of all shallow features, thus effectively extracting shallow contour texture information and deep high-level semantic information. Furthermore, after obtaining effectively extracted image features, this invention can decode these features to obtain the restored image, thereby effectively improving image restoration quality.

[0018] The present invention also provides an image restoration device, an electronic device, and a computer-readable storage medium, which have the above-mentioned beneficial effects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A flowchart of an image restoration method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a feature extraction block connection method provided in an embodiment of the present invention; Figure 3 A schematic diagram of an image restoration model provided in an embodiment of the present invention; Figure 4This is a schematic diagram of a feature extraction block structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a sequence fusion calculation method provided in an embodiment of the present invention; Figure 6 This is a structural block diagram of an image restoration device provided in an embodiment of the present invention; Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] In industrial settings, high-resolution cameras often suffer from motion blur, random noise, and decreased contrast in captured images due to mechanical vibrations, high-speed production line movements, dust obstruction, or fluctuating lighting. Image inpainting is necessary to eliminate these defects. However, current image inpainting methods often struggle to balance local details with global structure, resulting in poor restoration outcomes. For example, traditional image inpainting algorithms rely on manual priors or pure convolutional neural networks (CNNs), failing to adequately address both local details and global structure. While VisionTransformer and diffusion models offer high restoration quality, their quadratic complexity cannot meet millisecond-level processing speeds, leading to poor real-time performance. Mamba-type Selective State Space Models (SSMs) provide linear complexity for long-range modeling, offering a novel approach for rapid industrial degradation restoration. However, these SSM methods perform sequential scanning at a single scale, lacking joint modeling of multi-scale, cross-regional industrial degradation and failing to synergize with convolutional local priors. This results in edge ringing and texture distortion in the restoration results, impacting the accuracy of subsequent defect detection tasks such as semantic segmentation and object detection.

[0023] In view of this, regarding the technical problem of how to improve image feature extraction capabilities and thus improve image restoration results, the present invention provides an image restoration method that can improve the image feature extraction effect, realize the collaborative extraction of local and global features, and thus improve the image restoration effect.

[0024] For easier understanding, please refer to Figure 1 , Figure 1 A flowchart of an image restoration method provided in an embodiment of the present invention, the method may include: S10. Input the image to be repaired into the trained feature extraction model for feature extraction processing to obtain image features. The feature extraction model includes multi-level feature extraction blocks. The feature extraction blocks extract local features of the feature map group of the image to be repaired at different scales to obtain local features of different scales. The local features of the largest scale are selected and the upper-level sequence output by the upper-level feature extraction block are used to calculate the current layer sequence through selective state space model. The local features of different scales are fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output.

[0025] In this embodiment, to improve the image feature extraction capability, a feature extraction model can be provided. The feature extraction model may include multi-level feature extraction blocks, each used to extract image information at different levels, such as shallow contour texture information and deep high-level semantic information. Each feature extraction block may have a first input, a second input, a first output, and a second output. The first input is connected to the first output of the adjacent upper-level feature extraction block, the second input is connected to the second output of all upper-level feature extraction blocks, the first output is connected to the first input of the adjacent lower-level feature extraction block, and the second output is connected to the second input of all lower-level feature extraction blocks. Therefore, the feature extraction blocks are connected in a dense connection manner.

[0026] To better understand the connectivity relationships between feature extraction blocks, please refer to... Figure 2 , Figure 2 This is a schematic diagram illustrating a feature extraction block connection method provided in an embodiment of the present invention. Firstly, using... Figure 2Taking feature extraction block 2 as an example, it has a first input terminal, a second input terminal, a first output terminal, and a second output terminal. The first input terminal of feature extraction block 2 is connected to the first output terminal of feature extraction block 1, and is used to receive image features extracted by feature extraction block 1. The image features are fused features of local and global features of the image to be repaired. The second input terminal of feature extraction block 2 is connected to the second output terminal of feature extraction block 1, and is used to receive the upper-layer sequence extracted by feature extraction block 1. The upper-layer sequence is the global feature of the image to be repaired. The first output terminal of feature extraction block 2 is connected to the first output terminal of feature extraction block 3, and is used to send the image features extracted by feature extraction block 2. The second output terminal of feature extraction block 2 is connected to the second input terminal of feature extraction block 3, and is used to send the current layer sequence extracted by feature extraction block 2. It is also worth noting that the second output terminal of feature extraction block 1 is also connected to the second input terminal of feature extraction block 3, and is used to input the upper-layer sequence extracted by feature extraction block 1 into feature extraction block 3. This dense connection method ensures that each layer receives image features from its neighboring preceding layers, as well as the upper-layer sequences from all preceding layers. It then performs local feature extraction on the image features and combines the local features extracted at the current layer with the upper-layer sequences generated by the preceding layers to generate the current layer sequence. This ensures that shallow global information can be accurately transmitted to deeper layers without significantly increasing computation. It enables collaborative updates of "local degradation features - global repair priors," strengthening long-range dependencies across degradation regions. Simultaneously, it maximizes the information flow between different depth layers of the network, allowing the generation of deep features to be guided by the global summary of all shallow features. This effectively extracts shallow contour texture information and deep high-level semantic information.

[0027] In another case, please refer to Figure 3 , Figure 3 This is a schematic diagram of an image restoration model provided in an embodiment of the present invention. In this schematic diagram, the feature extraction model consists of feature extraction blocks 1 to 4. Each feature extraction block is connected not only to its adjacent lower-level feature blocks, but also to all lower-level feature blocks via cross-layer jump connections. For example, feature extraction block 1 is connected to feature extraction blocks 2 to 4 via cross-layer jump connections 1 to 3, feature extraction block 2 is connected to feature extraction blocks 3 and 4 via cross-layer jump connections 4 and 5, and feature extraction block 3 is connected to feature extraction block 4 via cross-layer jump connection 6.

[0028] The specific structure and internal working mode of the feature extraction block are described below. In one implementation, the feature extraction block may include a channel splitting layer, a feature extraction layer, a channel stitching layer, and a downsampling layer. The feature extraction layer contains K group processing branches. The feature extraction block performs local feature extraction on the feature map groups of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are then combined with the upper-level sequence output by the upper-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are then fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output. This may include: S11: The channel splitting layer splits the input feature map group into K sub-feature map groups.

[0029] In this embodiment, the input to the feature extraction block is a multi-channel feature map group of the image to be repaired. Each channel corresponds to one feature map, and multiple feature maps from multiple channels form a feature map group. In this embodiment, the feature map group can be divided into K sub-feature map groups based on the number K of the grouping processing branches. For example, a 100-channel feature map group can be divided into 5 sub-feature map groups, each containing feature maps from 20 channels.

[0030] It should be noted that this embodiment does not limit the value of K; K can be a positive integer greater than 3. The channel splitting layer can split the input feature map group into K sub-feature map groups by average channel splitting.

[0031] S12: The first group processing branch outputs the first sub-feature map group to the channel stitching layer and the second group processing branch.

[0032] S13: The i-th group processing branch concatenates the outputs of the i-th sub-feature map group and the (i-1)-th group processing branch to obtain the first concatenated feature. Local features are extracted from the first concatenated feature to obtain local features, and the local features are output to the channel concatenation layer and the (i+1)-th group processing branch; where i∈(1,K).

[0033] In this embodiment, a convolution kernel of a preset size can be set in the i-th group processing branch. The first spliced ​​feature can be convolved using the convolution kernel of the preset size to obtain local features.

[0034] S14: The Kth group processing branch concatenates the upper-layer sequence obtained from the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the second input terminal to obtain the second concatenated feature. The second concatenated feature is then used to perform selective state-space model sequence calculation to obtain the current layer sequence, and the current layer sequence is output to the channel concatenation layer and the second output terminal.

[0035] S15: The channel splicing layer splices the outputs of each group processing branch to obtain the channel splicing feature.

[0036] S16: The downsampling layer downsamples the channel stitching features to obtain image features and outputs them to the first output terminal.

[0037] In this embodiment, the channel splicing features can be downsampled by 2 times to reduce the amount of computation and accelerate the overall inference process.

[0038] To better understand the internal structure and working principle of the feature extraction block, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of a feature extraction block structure provided in an embodiment of the present invention. Figure 4 by Figure 3 The following explanation uses feature extraction block 3 as an example. Figure 4 In the process, feature extraction block 3 can include a channel splitting layer, a feature extraction layer, a channel splicing layer, and a downsampling layer. The feature extraction layer can consist of four group processing branches.

[0039] The first input to this feature extraction block is the feature map group (i.e., image features, also called channel group) extracted from the upper layer. The channel splitting layer first splits the feature map group into multiple sub-feature map groups. In this embodiment, it can be split into 4 groups: group 1, group 2, group 3, and group 4. Except for the last group, the information of the sub-feature map groups in each other group is passed through two branches to maximize feature reuse.

[0040] Subsequently, different sub-feature map groups are processed through different grouping processing branches.

[0041] Specifically, group 1 enters the channel splicing layer through path 1 in the first group processing branch. This is because group 1 is an identity mapping layer, so no processing is required, and it directly enters the channel splicing layer. Group 1 also enters the second group processing branch through path 2 in the first group processing branch.

[0042] Group 2 is first added to Group 1, and then convolutional processing is performed through the convolutional layer in the second group processing branch to extract local features. Here, a convolutional layer with a window size of N×N is used. Subsequently, the convolution result is passed forward through two branches in the second group processing branch: data from path 3 enters the channel splicing layer, and data from path 4 enters the third group processing branch.

[0043] Group 3 is first added to the convolution result of the second group processing branch, and then convolved through the convolutional layer in the third group processing branch to extract local features. Here, a convolutional layer with a window size of M×M is used. Subsequently, the convolution result is passed forward through two branches in the third group processing branch: data from path 5 enters the channel splicing layer, and data from path 6 enters the fourth group processing branch.

[0044] Group 4 is first added to the convolution result of the third group processing branch and combined with the previous layer sequence. Then, sequence calculation is performed to obtain the current layer sequence. The current layer sequence also passes through two branches for data transfer. The data from path 9 enters the channel splicing layer, while the data from path 8 is output to the second output end of feature extraction block 3 for sequence calculation in the next layer, playing a role in guiding data across layers.

[0045] This embodiment employs a hierarchical feature extraction layer structure to achieve feature reuse within the same layer, improve information utilization, and uncover feature information not noticed in previous groupings. Another crucial reason for using this hierarchical information transmission in image restoration models is that image restoration tasks require comprehensive observation of both global and local aspects of the image. One advantage of channel grouping is the ability to extract information from different receptive fields using different convolutional kernels, combined with the sequential computation of the selective state-space model, thus considering both local and global information. However, since grouping inevitably results in some loss of global information, the sequential computation of the selective state-space model cannot effectively capture all global information. Therefore, based on the specific characteristics of image restoration tasks, this embodiment transmits grouped information layer by layer, enabling the extraction of local features at different scales. For example, the N×N convolutional kernel of the second grouping processing branch can extract local feature information from groups 1 and 2; the M×M convolutional kernel of the third grouping processing branch can extract local feature information from groups 1, 2, and 3; and the fourth grouping processing branch can perform sequential computation based on the local features of groups 1-4. In this way, by processing the connections between branches in groups, this embodiment not only improves the feature reuse rate, but also makes up for the problem of missing global information caused by channel grouping, making the results calculated from the sequence more accurate.

[0046] Furthermore, in sequence calculation, this embodiment first adds the output of the Kth sub-feature map group to the output of the (K-1)th group processing branch, and flattens the addition result from two-dimensional data into one-dimensional sequence data. Then, the one-dimensional sequence is combined with the upper-level sequence input from the second input terminal. The specific combination method is as follows (e.g. Figure 5This method uses the data from the upper-level sequence as a prefix to the one-dimensional sequence. Specifically, it places the data at the beginning of the sequence during concatenation, followed by the current-level sequence, forming a new sequence for regular sequence computation. After the sequence computation is complete, a new set of original sequences is obtained. The sequence segment of a preset length at the end of this original sequence can be used as the current-level sequence.

[0047] Based on this, the Kth grouping processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th grouping processing branch, and the upper-level sequence obtained from the second input to obtain the second concatenated feature, including: S1411: The Kth group processing branch adds the output of the Kth sub-feature map group to the output of the (K-1)th group processing branch, and converts the addition result into a one-dimensional sequence; S1412: Concatenate the upper-level sequence to the prefix position of the one-dimensional sequence.

[0048] Accordingly, the current layer sequence is obtained by selectively calculating the state-space model sequence of the second splicing features, including: S1413: Selective state-space model sequence calculation is performed on the second splicing feature to obtain the original sequence; S1414: Take the sequence segment of preset length located at the end of the original sequence as the current layer sequence.

[0049] Specifically, the sequential computation process of the selective state-space model can be represented as follows: ; ; Where t represents discrete time, which can be understood as the input order of vectors in one-dimensional sequential data. A, B, and C are all learnable matrices in the selective state-space model, representing the state transition weight matrix, input weight matrix, and output weight matrix, respectively. Input representing the current time, This represents the current state determined by the model. This represents the previous time state determined by the model. This represents the current time.

[0050] It is worth noting that, since this embodiment uses the upper-level sequence as a prefix of the one-dimensional sequence, the selective state-space model can perform sequence calculation on the current level's one-dimensional sequence while fully considering the calculation results of the preceding level sequence, which can improve the global feature extraction effect.

[0051] Furthermore, to improve computational efficiency, if the second input of a feature extraction block contains upper-layer sequences from multiple preceding feature extraction blocks, feature filtering based on an attention mechanism can be performed on the upper-layer sequences before computation. This filtering is achieved by suppressing invalid features and highlighting valid features. This embodiment does not limit the specific attention mechanism method; it can be the channel attention method of SENet, or other spatial attention-based methods, etc. Figure 4 As shown on the right, this diagram illustrates the detailed method for filtering upper-level sequences. First, multiple upper-level sequences are concatenated together, then an attention mechanism is used for filtering, and finally, a sequence of a certain length is selected, with the selected length being consistent with the length of the current-level sequence.

[0052] Based on this, when the second input terminal receives multiple upper-level sequences, this method also includes: S17: Concatenate multiple upper-level sequences to obtain a concatenated sequence; S18: Apply attention mechanism to the spliced ​​sequence to obtain a processed sequence of a preset length; The Kth grouping processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th grouping processing branch, and the upper-level sequence obtained from the second input to obtain the second concatenated feature, including: S1421: The Kth group processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the processed sequence to obtain the second concatenated feature.

[0053] S20. Decode the image features to obtain the repaired image.

[0054] This step decodes the image features to obtain the restored image. In the decoding part, considering the need to preserve as much original detail as possible in image restoration, this embodiment obtains the first image features output from the penultimate feature extraction block and the second image features output from the last feature extraction block. The second image features are then interpolated and upsampled to obtain upsampled features. Subsequently, the first image features and the upsampled features are fused. At this point, the fused result contains image features of different levels and finer granularities, which helps improve decoding quality. Furthermore, this embodiment performs deconvolution upsampling on the fused result to obtain the restored image. Using deconvolution for upsampling is more effective in restoring image upsampling details compared to commonly used interpolation methods.

[0055] Based on this, the image features are decoded to obtain the repaired image, which may include: S21: Obtain the first image features output by the penultimate feature extraction block and the second image features output by the last feature extraction block; S22: Perform interpolation upsampling on the second image features to obtain upsampled features; S23: The first image features and the upsampled features are fused together, and the fusion result is deconvolutionally upsampled to obtain the repaired image.

[0056] Furthermore, it is understood that the image restoration model and feature extraction model in this embodiment need to be trained using training data in order to achieve the image restoration effect. This training data should include the image before restoration and the image after restoration. This embodiment does not limit the specific model training process; relevant techniques in the field of machine learning can be referenced.

[0057] Based on the above embodiments, this invention improves the image feature extraction process to enhance image restoration quality. This invention provides a feature extraction model comprising multi-level feature extraction blocks. Each feature extraction block can extract local features from feature maps of the image to be restored at different scales, thereby focusing on local details in the image at different scales. Furthermore, the feature extraction block can perform selective state-space model sequence calculations on the largest-scale local features and the upper-layer sequence obtained from the second input to obtain the current-layer sequence, thus determining the global features of the current layer based on the local features acquired in the current layer and the global features acquired in previous layers. Subsequently, the feature extraction block can fuse local features at different scales with the current-layer sequence to obtain image features and output the image features and the current-layer sequence. As can be seen, the feature extraction block can achieve deep collaboration between local multi-scale features and global selective state-space model features. Furthermore, by densely connecting each feature extraction layer, the current layer receives image features and upper-level sequences output from all preceding layers, and continues to extract local features from the image features. It then combines the extracted local features with the upper-level sequences generated by the preceding layers to generate the current layer sequence. This achieves collaborative updates of "local degradation features - global restoration priors," strengthening long-range dependencies across degradation regions. Simultaneously, it maximizes information flow between different network depths, allowing the generation of deep features to be guided by the global summary of all shallow features, thus effectively extracting shallow contour texture information and deep high-level semantic information. Furthermore, after obtaining effectively extracted image features, this invention can decode these features to obtain the restored image, thereby effectively improving image restoration quality.

[0058] Compared with related technologies, the present invention has the following technical effects: 1. High-quality restoration: Through deep collaboration of local and global features and dense information flow across layers, this method can understand the overall structure and local details of the image, thereby restoring image degradation problems such as large-area and complex artifacts and blurring. The generated results are more reasonable in structure and more realistic in texture.

[0059] 2. Strong information integrity: The feature injection mechanism within the module ensures that the SSM (selective state space model) can obtain complete local information when performing global modeling; the dense state transfer between modules realizes the maximum reuse of features in the network depth, effectively alleviating the gradient vanishing problem in deep networks.

[0060] 3. Computationally efficient: The main body of this invention adopts a selective state-space model with linear complexity for global modeling, which avoids the huge computational bottleneck faced by the Transformer architecture under high-resolution images, and is more suitable for deployment in industrial production environments with strict real-time requirements.

[0061] The image restoration apparatus, electronic device, and computer-readable storage medium provided in the embodiments of the present invention are described below. The image restoration apparatus, electronic device, and computer-readable storage medium described below can be referred to in correspondence with the image restoration method described above.

[0062] Please refer to Figure 6 , Figure 6 This is a structural block diagram of an image restoration device provided in an embodiment of the present invention. The device may include: The feature extraction module 601 is used to input the image to be repaired into a trained feature extraction model for feature extraction processing to obtain image features. The feature extraction model includes multi-level feature extraction blocks. The feature extraction blocks extract local features of the feature map group of the image to be repaired at different scales to obtain local features of different scales. The local features of the largest scale are selected and the upper-level sequence output by the upper-level feature extraction block are used to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features of different scales are fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output. The decoding module 602 is used to decode the image features to obtain the repaired image.

[0063] Optionally, the feature extraction block includes a channel splitting layer, a feature extraction layer, a channel splicing layer, and a downsampling layer. The feature extraction layer includes K grouping processing branches. The feature extraction block has a first input terminal, a second input terminal, a first output terminal, and a second output terminal. The first input terminal is connected to the first output terminal of the adjacent upper-level feature extraction block of the feature extraction block. The second input terminal is connected to the second output terminals of all upper-level feature extraction blocks of the feature extraction block. The first output terminal is connected to the first input terminal of the adjacent lower-level feature extraction block of the feature extraction block. The second output terminal is connected to the second input terminal of all lower-level feature extraction blocks of the feature extraction block. The channel splitting layer is used to split the input feature map into K sub-feature maps; The first grouping processing branch is used to output the first sub-feature map group to the channel stitching layer and the second grouping processing branch; The i-th group processing branch is used to concatenate the outputs of the i-th sub-feature map group and the (i-1)-th group processing branch to obtain the first concatenated feature, extract local features from the first concatenated feature to obtain local features, and output the local features to the channel concatenation layer and the (i+1)-th group processing branch; where i∈(1,K); The Kth group processing branch is used to concatenate the upper-layer sequence obtained from the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the second input terminal to obtain the second concatenated feature. The second concatenated feature is then used to perform selective state space model sequence calculation to obtain the current layer sequence, and the current layer sequence is output to the channel concatenation layer and the second output terminal. The channel splicing layer is used to splice the outputs of each group processing branch to obtain the channel splicing feature. The downsampling layer is used to downsample the channel stitching features to obtain image features, and then output them to the first output terminal.

[0064] Optionally, the i-th group processing branch is also used for: The first concatenated features are convolved using a convolution kernel of a preset size to obtain local features.

[0065] Optionally, the Kth group processing branch is also used for: The output of the Kth sub-feature map group is added to the output of the (K-1)th group processing branch, and the result is converted into a one-dimensional sequence. The upper-level sequence is concatenated to the prefix position of the one-dimensional sequence; The original sequence is obtained by selectively calculating the state-space model sequence of the second splicing feature; The sequence segment of a preset length located at the end of the original sequence is taken as the current layer sequence.

[0066] Optionally, the device may further include: The upper-level sequence splicing module is used to splice multiple upper-level sequences to obtain a spliced ​​sequence; The attention mechanism processing module is used to process the spliced ​​sequence using the attention mechanism to obtain a processed sequence of a preset length; The Kth grouping processing branch is also used for: The second spliced ​​feature is obtained by concatenating the output of the Kth sub-feature map group, the K-1th group processing branch, and the processed sequence.

[0067] Optionally, the channel splitting layer is also used for: The channel splitting layer splits the input feature map group into K sub-feature map groups by average channel splitting.

[0068] Optionally, the decoding module includes: The acquisition submodule is used to acquire the first image features output by the penultimate feature extraction block and the second image features output by the last feature extraction block. The upsampling processing submodule is used to perform interpolation upsampling processing on the second image features to obtain upsampled features; The deconvolution upsampling processing submodule is used to fuse the first image features and the upsampled features, and then perform deconvolution upsampling processing on the fusion result to obtain the repaired image.

[0069] Please refer to Figure 7 , Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein, the memory 12 is used to store a computer program; the processor 11 is used to execute the image restoration method provided in the foregoing embodiment when executing the computer program.

[0070] For details regarding the specific process of the above image restoration method, please refer to the relevant content provided in the foregoing embodiments, which will not be repeated here.

[0071] Furthermore, the memory 12, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0072] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15, and a communication bus 16; wherein, the power supply 13 is used to provide operating voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 15 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0073] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the image restoration method described in the above embodiments.

[0074] Since the embodiments of the computer program product section correspond to the embodiments of the image restoration method section, please refer to the description of the embodiments of the image restoration method section for the embodiments of the computer program product section, and they will not be repeated here.

[0075] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image restoration method described in the above embodiments.

[0076] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the image restoration method portion, the embodiments of the storage medium portion are described in the description of the embodiments of the image restoration method portion, and will not be repeated here.

[0077] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0078] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0079] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0080] The image restoration method, apparatus, electronic device, and storage medium provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. An image restoration method, characterized in that, include: The image to be repaired is input into a trained feature extraction model for feature extraction processing to obtain image features. The feature extraction model includes multi-level feature extraction blocks. Each feature extraction block performs local feature extraction on the feature map group of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are combined with the upper-level sequence output by the upper-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output. The image features are decoded to obtain the repaired image.

2. The image restoration method according to claim 1, characterized in that, The feature extraction block includes a channel splitting layer, a feature extraction layer, a channel splicing layer, and a downsampling layer. The feature extraction layer includes K grouping processing branches. The feature extraction block has a first input terminal, a second input terminal, a first output terminal, and a second output terminal. The first input terminal is connected to the first output terminal of the adjacent upper-level feature extraction block of the feature extraction block. The second input terminal is connected to the second output terminals of all upper-level feature extraction blocks of the feature extraction block. The first output terminal is connected to the first input terminal of the adjacent lower-level feature extraction block of the feature extraction block. The second output terminal is connected to the second input terminal of all lower-level feature extraction blocks of the feature extraction block. The feature extraction block extracts local features from the feature map group of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are then combined with the upper-level sequence output by the previous-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are then fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output, including: The channel splitting layer splits the input feature map group into K sub-feature map groups; The first grouping processing branch outputs the first sub-feature map group to the channel stitching layer and the second grouping processing branch; The i-th group processing branch concatenates the outputs of the i-th sub-feature map group and the (i-1)-th group processing branch to obtain the first concatenated feature. Local features are extracted from the first concatenated feature to obtain local features, and the local features are output to the channel concatenation layer and the (i+1)-th group processing branch; where i∈(1,K); The Kth group processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the upper-layer sequence obtained from the second input terminal to obtain the second concatenated feature. The second concatenated feature is then subjected to selective state-space model sequence calculation to obtain the current layer sequence, and the current layer sequence is output to the channel concatenation layer and the second output terminal. The channel splicing layer splices the outputs of each group processing branch to obtain channel splicing features; The downsampling layer downsamples the channel stitching features to obtain the image features, and outputs them to the first output terminal.

3. The image restoration method according to claim 2, characterized in that, The step of extracting local features from the first spliced ​​features includes: The first spliced ​​feature is convolved using a convolution kernel of a preset size to obtain the local feature.

4. The image restoration method according to claim 2, characterized in that, The Kth grouping processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th grouping processing branch, and the upper-level sequence obtained from the second input to obtain the second concatenated feature, including: The Kth group processing branch adds the output of the Kth sub-feature map group to the output of the (K-1)th group processing branch, and converts the result into a one-dimensional sequence. The upper-level sequence is concatenated to the prefix position of the one-dimensional sequence; The step of selectively calculating the state-space model sequence of the second splicing feature to obtain the current layer sequence includes: The original sequence is obtained by selectively calculating the state-space model sequence of the second splicing feature; The sequence segment of a preset length located at the end of the original sequence is taken as the current layer sequence.

5. The image restoration method according to claim 2, characterized in that, When the second input terminal receives multiple upper-layer sequences, the method further includes: A spliced ​​sequence is obtained by concatenating multiple upper-level sequences; The spliced ​​sequence is processed using an attention mechanism to obtain a processed sequence of a preset length; The Kth grouping processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th grouping processing branch, and the upper-level sequence obtained from the second input to obtain the second concatenated feature, including: The Kth group processing branch concatenates the Kth sub-feature map group, the output of the (K-1)th group processing branch, and the processed sequence to obtain the second concatenated feature.

6. The image restoration method according to claim 2, characterized in that, The channel splitting layer splits the input feature map group into K sub-feature map groups, including: The channel splitting layer performs average channel splitting on the input feature map group to obtain the K sub-feature map groups.

7. The image restoration method according to claim 1, characterized in that, The process of decoding the image features to obtain the repaired image includes: Obtain the first image features output by the penultimate feature extraction block and the second image features output by the last feature extraction block; The second image features are subjected to interpolation and upsampling processing to obtain upsampled features; The first image features and the upsampled features are fused, and the fusion result is subjected to deconvolution upsampling processing to obtain the repaired image.

8. An image restoration device, characterized in that, include: The feature extraction module is used to input the image to be repaired into a trained feature extraction model for feature extraction processing to obtain image features. The feature extraction model includes multi-level feature extraction blocks. The feature extraction blocks perform local feature extraction on the feature map group of the image to be repaired at different scales to obtain local features at different scales. The local features at the largest scale are combined with the upper-level sequence output by the upper-level feature extraction block to perform selective state-space model sequence calculation to obtain the current layer sequence. The local features at different scales are fused with the current layer sequence to obtain image features, and the image features and the current layer sequence are output. The decoding module is used to decode the image features to obtain the repaired image.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the image restoration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the image restoration method as described in any one of claims 1 to 7.

Images