Image processing method and device, model training method and device, electronic equipment and medium

Abstract

The present disclosure provides an image processing method and device, a model training method and device, an electronic device and a medium, relates to the technical field of image processing, and in particular to the technical field of image denoising and image enhancement. The specific implementation scheme is: obtaining a to-be-processed image, wherein the to-be-processed image is a document image with moire, then encoding the to-be-processed image through an encoder to obtain encoding features output by multiple encoding stages of the encoder respectively. Then, the multiple encoding features are fused through a multi-scale interaction module to obtain fused features, and the fused features are decoded through a decoder to obtain a restored image after removing moire from the to-be-processed image. The moire of the document image is removed more accurately.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to the field of image denoising and image enhancement technology. Background Art

[0002] Moiré is a common interference fringe pattern found in photography or scanning. For example, in a mobile phone, camera, or other device with a camera function, when the pixel spatial frequency of the device's photosensitive element is close to the spatial frequency of the object's pattern, moiré will appear in the image captured by the device. For example, when a mobile phone uses its built-in camera to capture an image on a computer screen, the captured image will exhibit moiré. Moiré typically appears as a rainbow-colored, distorted grid pattern or artifact, severely impacting the visual quality of the image. Summary of the Invention

[0003] The present disclosure provides an image processing method, a model training method, an apparatus, an electronic device, and a medium.

[0004] According to a first aspect of the present disclosure, there is provided an image processing method, comprising:

[0005] Acquire an image to be processed, where the image to be processed is a document image with moiré patterns;

[0006] Encoding the image to be processed by an encoder to obtain encoding features output by multiple encoding stages of the encoder;

[0007] The encoding features are fused through the multi-scale interaction module to obtain the fused features;

[0008] The fusion features are decoded by a decoder to obtain a restored image after the moiré pattern is removed from the image to be processed.

[0009] A second aspect of the embodiments of the present disclosure provides a model training method, including:

[0010] Acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns;

[0011] Encoding the sample image by removing the encoder included in the model to obtain encoding features output by multiple encoding stages of the encoder;

[0012] By removing the multi-scale interaction module included in the model, each encoding feature is fused to obtain the fused feature;

[0013] Decoding the fused features by removing the decoder included in the model to obtain decoding features outputted by multiple decoding stages of the decoder;

[0014] Determining a loss value based on the decoded features and the training labels;

[0015] The network parameters of the removal model are adjusted using the loss value, and the step of encoding the sample image by the encoder included in the removal model is returned to, until the removal model converges, and the training is determined to be completed.

[0016] According to a third aspect of the present disclosure, an image processing apparatus is provided, including:

[0017] An acquisition module, configured to acquire an image to be processed, wherein the image to be processed is a document image with moiré patterns;

[0018] An encoding module, configured to encode the image to be processed by an encoder to obtain encoding features outputted respectively by multiple encoding stages of the encoder;

[0019] The fusion module is used to fuse the encoding features through the multi-scale interaction module to obtain the fusion feature;

[0020] The decoding module is used to decode the fusion feature through a decoder to obtain a restored image after removing the moiré pattern from the image to be processed.

[0021] A fourth aspect of the present disclosure provides a model training device, including:

[0022] an acquisition module, configured to acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns;

[0023] an encoding module, configured to encode the sample image by removing an encoder included in the model, and obtain encoding features outputted respectively by a plurality of encoding stages of the encoder;

[0024] A fusion module is used to fuse the encoding features by removing the multi-scale interaction module included in the model to obtain a fusion feature;

[0025] A decoding module, configured to decode the fused features by removing the decoder included in the model to obtain decoding features outputted by the multiple decoding stages of the decoder;

[0026] A determination module, configured to determine a loss value based on the decoded features and the training labels;

[0027] An adjustment module is used to adjust the network parameters of the removal model using the loss value, return to the step of encoding the sample image by the encoder included in the removal model, and determine that the training is completed when the removal model converges.

[0028] According to a fifth aspect of the present disclosure, an electronic device is provided, including:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of the first aspect or the second aspect.

[0032] According to a sixth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any one of the first aspect or the second aspect.

[0033] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any one of the first aspect or the second aspect when executed by a processor.

[0034] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0036] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0037] Figure 2 is a flow chart of a feature encoding method provided by an embodiment of the present disclosure;

[0038] Figure 3 is an exemplary schematic diagram of a processing process of a spatial attention module provided by an embodiment of the present disclosure;

[0039] Figure 4 is a flow chart of a feature fusion method provided by an embodiment of the present disclosure;

[0040] FIG5( a ) is a schematic structural diagram of a first multi-scale interaction module provided by an embodiment of the present disclosure;

[0041] FIG5( b ) is a schematic structural diagram of a second multi-scale interaction module provided by an embodiment of the present disclosure;

[0042] FIG5( c ) is a schematic structural diagram of a third multi-scale interaction module provided by an embodiment of the present disclosure;

[0043] Figure 6 is a flow chart of a feature decoding method provided by an embodiment of the present disclosure;

[0044] Figure 7 is a structural diagram of a reconstruction module provided by an embodiment of the present disclosure;

[0045] Figure 8 This is a schematic diagram of the structure of the conventional ESDNet;

[0046] Figure 9 is a schematic structural diagram of a removal model provided by an embodiment of the present disclosure;

[0047] Figure 10 is a flowchart of a model training method provided by an embodiment of the present disclosure;

[0048] Figure 11 is a structural diagram of an image processing device provided by an embodiment of the present disclosure;

[0049] Figure 12 is a structural diagram of a model training device provided by an embodiment of the present disclosure;

[0050] Figure 13 This is a block diagram of an electronic device used to implement the image processing method and model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0051] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0052] Current solutions for removing moiré from images are mainly targeted at natural images such as portraits or landscapes. However, these solutions are less effective in removing moiré from document images such as books, test papers, certificates, or bills.

[0053] In order to improve the effect of removing moiré from document images, the present disclosure provides an image processing method, which is applied to electronic devices, such as servers, desktop computers, laptop computers, or mobile phones, which have image processing capabilities. Figure 1 As shown, the image processing method provided by the embodiment of the present disclosure includes the following steps:

[0054] S101: Acquire an image to be processed.

[0055] The image to be processed is a document image with moiré patterns. Specifically, the image to be processed can be an image of a document such as a book, test paper, certificate, or receipt captured by a device with image capture capabilities, such as a mobile phone, tablet computer, or camera, or captured by a scanner. For example, the image to be processed can be an image of a test paper displayed on a computer screen captured by a mobile phone.

[0056] Optionally, the image to be processed may be an image captured by the electronic device via a built-in camera, or an image frame from a video captured by the built-in camera. Alternatively, the image to be processed may be an image downloaded by the electronic device via a network. Alternatively, the image to be processed may be an image sent by another device and received by the electronic device. The presently disclosed embodiments do not specifically limit the source of the image to be processed.

[0057] S102: Encode the image to be processed by an encoder to obtain encoding features outputted by the encoder in multiple encoding stages, wherein the sizes of the encoding features are different.

[0058] The encoder consists of multiple encoding stages connected in series, with each stage extracting increasingly deeper features. That is, earlier encoding stages extract shallower image features, such as edges, corners, texture, and color; later encoding stages extract deeper image features, such as local object contours and semantic information. The detailed encoding process is described below.

[0059] S103: Fusing the coding features through a multi-scale interaction module to obtain fused features.

[0060] The multi-scale interaction module may be a multi-scale feature interaction fusion (MSFIF) module. The specific fusion process can be described below.

[0061] S104: Decode the fused features through a decoder to obtain a restored image after removing moiré patterns from the image to be processed.

[0062] Since the fused features are obtained based on coding features of multiple sizes, and different coding features contain different feature depths, the decoder can combine shallow and deep features for decoding, thereby more accurately constructing a restored image after removing moiré patterns from the processed image.

[0063] In the disclosed embodiment, the encoder's different encoding stages can extract features at different depths, so the features extracted at different encoding stages are not completely identical, and the features extracted at each encoding stage collectively cover a wider range of features. Therefore, the disclosed embodiment fuses the encoding features through a multi-scale interaction module and decodes the fused features through a decoder. This allows the decoder to construct a restored image based on the richer features extracted at each encoding stage, thereby gaining a deeper understanding of the image being processed, enabling more accurate moiré removal of the document image and obtaining a more accurate restored image.

[0064] The image processing method provided by the embodiment of the present disclosure is described in detail below:

[0065] In some embodiments of the present disclosure, the encoder in the above S102 includes a first encoding stage, a second encoding stage, and a third encoding stage.

[0066] On this basis, see Figure 2 The above-mentioned S102 encodes the image to be processed by the encoder to obtain the encoding features output by multiple encoding stages of the encoder, including the following steps:

[0067] S201 : Encode the image to be processed through a first encoding stage to obtain a first encoding feature.

[0068] In the embodiment of the present disclosure, the image to be processed may be downsampled and convolved to obtain preprocessing features, and then the preprocessing features are input into the first encoding stage for encoding to obtain first encoding features.

[0069] The downsampling process may be pixel shuffle downsampling, and the convolution process may be performed by a convolutional layer with a 5×5 convolution kernel.

[0070] In the disclosed embodiments, the encoder, multi-scale interaction module, and decoder all constitute a removal model, which can be derived by improving upon the Efficient Spiking Deraining Network (ESDNet). Because ESDNet requires downsampling and convolution processing before encoding an image through the encoder, the disclosed embodiments first downsample and convolute the image to be processed before passing it through the encoder. This minimizes damage to the original ESDNet structure, ensures smooth image processing, and improves the effectiveness of removing moiré from document images.

[0071] In the disclosed embodiment, each encoding stage includes a preset number of spatial attention modules (SAMs) connected in series, where the preset number is greater than 2. For example, each encoding stage may include a dilated residual dense block (DRDB) and four SAMs connected in series. In the first encoding stage, the preprocessed features may be processed sequentially by the DRDB and the four SAMs to obtain the first encoded features.

[0072] See also Figure 3 , SAM's processing of input features includes: r Bilinear downsampling is used to generate F1, F2 and F3, where F is assumed to be rIf the size of F1 is H×W×C, then the size of F2 is H / 2×W / 2×C / 2, and the size of F3 is H / 4×W / 4×C / 4, where H represents the length, W represents the width, and C represents the number of channels. F1 is then processed sequentially through a dilated dense block (DDK) and a convolutional layer with a 1×1 kernel to obtain Y0. F2 is processed sequentially through a DDK, a convolutional layer with a 1×1 kernel, and a 2× bilinear upsampling (2× Bilinear Up) to obtain Y1. F3 is processed sequentially through a DDK, a convolutional layer with a 1×1 kernel, and a 4× bilinear upsampling (4× Bilinear Up) to obtain Y2. Global average pooling (GAP) is performed on Y0, Y1, and Y2, respectively, to obtain v0, v1, and v2. Then, v0, v1, and v2 are concatenated channel-wise (Channel-wise Concatenation), and the concatenated features are input into the Multi-Layer Perceptron (MLP). The output of the MLP is split channel-wise (Channel-wise Split) to obtain w0, w1, and w2. Among them, w0 reflects the importance of Y0 for removing moiré patterns, w1 reflects the importance of Y1 for removing moiré patterns, and w2 reflects the importance of Y2 for removing moiré patterns. Then, Y0, Y1, Y2, w0, w1, and w2 are multiplied channel-wise (Channel-wise Multiplication), and the multiplication result is added to F. r Add channel by channel to get the processing result F out .

[0073] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré patterns, by increasing the number of SAMs included in the encoding stage, the process of removing moiré patterns from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of moiré removal.

[0074] S202: Input the first coding feature into the second coding stage for coding, and obtain a second coding feature output by the second coding stage.

[0075] S203: Input the second coding feature into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

[0076] The processing of the second encoding stage and the third encoding stage is the same as that of the first encoding stage, and reference may be made to the relevant description in S201 , which will not be repeated here.

[0077] Through the above method, the embodiment of the present disclosure can obtain the coding features output by the encoder in three different coding stages. The features included in these three coding features are not exactly the same, so these three coding features cover a wider range of features, making the subsequent moiré removal effect through these three features better.

[0078] In some embodiments of the present disclosure, see Figure 4 The above-mentioned S103 fuses the coding features through the multi-scale interaction module to obtain the fused features, including the following steps:

[0079] S401: Input each coding feature into a first multi-scale interaction module for fusion to obtain a first fused feature.

[0080] Among them, the three multi-scale interaction modules can all be multi-scale feature fusion interaction (MSFIF) modules.

[0081] S402: Input each coding feature into a second multi-scale interaction module for fusion to obtain a second fused feature.

[0082] S403: Input each encoded feature into a third multi-scale interaction module for fusion to obtain a third fused feature. S401 to S403 can be executed serially or in parallel. The embodiment of the present disclosure does not specifically limit the execution order of these three steps.

[0083] Among them, the structures of the first multi-scale interaction module, the second multi-scale interaction module and the third multi-scale interaction module are similar, and the sizes of the first fusion features, the second fusion features and the third fusion features output by the three multi-scale interaction modules are different.

[0084] In the embodiment of the present disclosure, the size of the first fusion feature is larger than the size of the second fusion feature, and the size of the second fusion feature is larger than the size of the third fusion feature.

[0085] Through the above method, the embodiment of the present disclosure can fuse the encoding features through multiple multi-scale interaction modules and obtain three fused features of different sizes, so that the size of each fused feature meets the size requirement of the input feature of a decoding stage of the decoder.

[0086] As shown in Figure 5(a), the first multi-scale interaction module includes three first interaction modules and one second interaction module. Each first interaction module consists of two residual blocks and a squeeze-and-excitation (SE) block. The second interaction module includes two convolutional layers and a squeeze-and-excitation (SE) block between them. Assuming the size of the first encoded feature is h × w × c, the size of the second encoded feature is h / 2 × w / 2 × c / 2, and the size of the third encoded feature is h / 4 × w / 4 × c / 4.

[0087] The method of obtaining the first fusion feature includes the following steps:

[0088] Step 1: Input the first coding feature into the first first interaction module for processing to obtain the first excitation feature.

[0089] Step 2: Input the second coding feature into the second first interaction module for processing to obtain a second excitation feature.

[0090] Step 3: Input the third coding feature into the third first interaction module for processing to obtain a third excitation feature.

[0091] Combining steps 1-3, see Figure 5(a). The first encoded feature is input into the first residual block of the first first interaction module, yielding feature a1 as the output of this residual block. The second encoded feature is input into the first residual block of the second first interaction module, yielding feature a2 as the output of this residual block. The third encoded feature is input into the first residual block of the third first interaction module, yielding feature a3 as the output of this residual block. a1 is convolved with a 1×1 kernel and downsampled to yield a1'. a2 is convolved with a 1×1 kernel and downsampled to yield a2'. a2 is convolved with a 1×1 kernel and upsampled to yield a2''. a3 is convolved with a 1×1 kernel and upsampled to yield a3''. Upsampling and downsampling ensure that the resolution of the processed image is compatible with the number of feature channels. In Figure 5(a), the dashed line represents convolution and downsampling, and the dotted dashed line represents convolution and upsampling.

[0092] Add a1 and a2'' channel by channel, and process the result of the addition through the second residual block and SE block of the first interaction module in sequence to obtain the first excitation feature. The size of the first excitation feature is h×w×c.

[0093] Add a1', a2, and a3'' channel by channel, and process the result of the addition through the second residual block and SE block of the second first interaction module in sequence to obtain the second excitation feature. The size of the second excitation feature is h / 2 × w / 2 × 2c.

[0094] Add a3 and a2' channel by channel, and process the result of the addition sequentially through the second residual block and SE block of the third first interaction module to obtain the third excitation feature. The size of the third excitation feature is h / 4×w / 4×4c.

[0095] Step 4: Upsample the second excitation feature and the third excitation feature respectively to obtain two upsampled features.

[0096] Referring to FIG5(a), the second excitation feature can be convolved with a 1×1 convolution kernel and upsampled to obtain an upsampled feature. The third excitation feature can also be convolved with a 1×1 convolution kernel and upsampled to obtain an upsampled feature.

[0097] Step 5: Concatenate the first excitation feature and the two upsampled features, and input the concatenated features into the second interaction module to obtain the first interaction feature.

[0098] As shown in Figure 5(a), the first excitation feature and the two upsampled features can be concatenated channel by channel, resulting in a concatenated feature size of h × w × 7c. The concatenated feature is then input into the first convolutional layer with a 3 × 3 kernel, included in the second interaction module. The output of this convolutional layer has an output size of h × w × c. The output of this convolutional layer is then input into the SE block included in the second interaction module. The output of the SE block is then input into the second convolutional layer with a 3 × 3 kernel, included in the second interaction module, to obtain the first interaction feature output by this convolutional layer.

[0099] Step 6: Concatenate the first interaction feature with each coding feature to obtain the first fusion feature.

[0100] Referring to FIG5(a), the first interaction feature and each encoding feature can be added channel by channel to obtain a first fusion feature, and the size of the first fusion feature is h×w×c.

[0101] Through the above method, the disclosed embodiments can obtain more comprehensive and richer fused features by fusing coding features of different scales. This fused feature helps the subsequent decoder capture more image details and contextual information, thereby improving the accuracy of the subsequent restored image.

[0102] As shown in Figure 5(b), the second multi-scale interaction module includes three third interaction modules and one fourth interaction module. The third interaction module includes two residual blocks and a compression and excitation block. The fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers. Assuming that the size of the first encoded feature is h×w×c, the size of the second encoded feature is h / 2×w / 2×c / 2, and the size of the third encoded feature is h / 4×w / 4×c / 4. In Figure 5(b), the short dashed line represents convolution processing and downsampling, and the dot-dashed line represents convolution processing and upsampling.

[0103] The method of obtaining the second fusion feature includes the following steps:

[0104] Step 1′: input the first coding feature into the first third interaction module for processing to obtain the fourth excitation feature.

[0105] Step 2': input the second coding feature into the second third interaction module for processing to obtain the fifth excitation feature.

[0106] Step 3': input the third coding feature into the third third interaction module for processing to obtain a sixth excitation feature.

[0107] The implementation of step 1' to step 3' is the same as the implementation of step 1 to step 3 above, and reference may be made to the above description, which will not be repeated here.

[0108] 5( b ), the size of the fourth excitation feature is h×w×c, the size of the fifth excitation feature is h / 2×w / 2×2c, and the size of the sixth excitation feature is h / 4×w / 4×4c.

[0109] Step 4': upsample the fourth excitation feature to obtain a downsampled feature.

[0110] Referring to FIG5( b ), the fourth excitation feature may be convolved with a convolution kernel of 1×1 and downsampled to obtain a downsampled feature.

[0111] Step 5': downsample the sixth excitation feature to obtain an upsampled feature.

[0112] 5( b ), the sixth excitation feature may be subjected to convolution processing with a convolution kernel of 1×1 and upsampled to obtain an upsampled feature.

[0113] Step 6': concatenate the up-sampled feature, the fifth excitation feature, and the down-sampled feature, and input the concatenated feature into the fourth interaction module to obtain a second interaction feature.

[0114] As shown in Figure 5(b), the upsampled features, the fifth excitation features, and the downsampled features can be concatenated channel by channel. The size of the concatenated features is h / 2 × w / 2 × 7c. The concatenated features are then input into the first convolutional layer with a 3 × 3 convolution kernel included in the fourth interactive module. The output of this convolutional layer has a size of h / 2 × w / 2 × 2c. The output of this convolutional layer is then input into the SE block included in the fourth interactive module. The output of the SE block is then input into the second convolutional layer with a 3 × 3 convolution kernel included in the fourth interactive module to obtain the second interactive features output by this convolutional layer.

[0115] Step 7': concatenate the second interactive feature with each coding feature to obtain a second fusion feature.

[0116] Referring to FIG5( b ), the second interaction feature can be added to each encoding feature channel by channel to obtain a second fusion feature, and the size of the second fusion feature is h / 2×w / 2×2c.

[0117] Through the above method, the disclosed embodiments can obtain more comprehensive and richer fused features by fusing coding features of different scales. This fused feature helps the subsequent decoder capture more image details and contextual information, thereby improving the accuracy of the subsequent restored image.

[0118] As shown in Figure 5(c), the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module. The fifth interaction module includes two residual blocks and a compression and excitation block. The sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers. In Figure 5(c), the short dashed line represents convolution processing with downsampling, and the dot-dashed line represents convolution processing with upsampling.

[0119] The method of obtaining the third fusion feature includes the following steps:

[0120] Step 1'': input the first coding feature into the first fifth interaction module for processing to obtain the seventh excitation feature.

[0121] Step 2'': input the second coding feature into the second fifth interaction module for processing to obtain the eighth excitation feature.

[0122] Step 3'': input the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature.

[0123] The implementation method of step 1'' to step 3'' is the same as the implementation method of the above steps 1 to step 3. Please refer to the above description and will not be repeated here.

[0124] 5( c ), the size of the seventh excitation feature is h×w×c, the size of the eighth excitation feature is h / 2×w / 2×2c, and the size of the ninth excitation feature is h / 4×w / 4×4c.

[0125] Step 4'': downsample the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features.

[0126] 5(c), the seventh excitation feature can be convolved with a 1×1 convolution kernel and downsampled to obtain a downsampled feature. The eighth excitation feature can also be convolved with a 1×1 convolution kernel and downsampled to obtain a downsampled feature.

[0127] Step 5'': concatenate the two downsampled features and the ninth excitation feature, and input the concatenated features into the sixth interaction module to obtain the third interaction feature.

[0128] As shown in Figure 5(c), the ninth excitation feature and the two downsampled features can be concatenated channel by channel, resulting in a concatenated feature size of h / 4 × w / 4 × 7c. The concatenated feature is then input into the first 3×3 convolution layer included in the sixth interactive module. The output of this convolution layer has an output size of h / 4 × w / 4 × 4c. The output of this convolution layer is then input into the SE block included in the sixth interactive module. The output of the SE block is then input into the second 3×3 convolution layer included in the sixth interactive module, resulting in the third interactive feature output by this convolution layer.

[0129] Step 6'': concatenate the third interactive feature with each coding feature to obtain a third fusion feature.

[0130] Referring to FIG5(c), the third interaction feature can be added to each encoding feature channel by channel to obtain a third fusion feature. The size of the first fusion feature is h / 4×w / 4×4c.

[0131] Through the above method, the disclosed embodiments can obtain more comprehensive and richer fused features by fusing coding features of different scales. This fused feature helps the subsequent decoder capture more image details and contextual information, thereby improving the accuracy of the subsequent restored image.

[0132] In some embodiments of the present disclosure, the decoder in the above S104 includes a first decoding stage, a second decoding stage, and a third decoding stage.

[0133] On this basis, see Figure 6 The above S104 decodes the fusion features through a decoder to obtain a restored image after removing moiré patterns from the image to be processed, including the following steps:

[0134] S601: Input the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature.

[0135] In the disclosed embodiment, each decoding stage includes a preset number of SAMs connected in series, where the preset number is greater than 2. For example, each decoding stage may include a DRDB and four SAMs connected in series. In the first decoding stage, the first fused feature may be processed sequentially by the DRDB and the four SAMs to obtain a first decoding feature.

[0136] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré patterns, by increasing the number of SAMs included in the decoding stage, the process of removing moiré patterns from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of moiré removal.

[0137] S602: Decode the second fusion feature and the first decoding feature in a second decoding stage to obtain a second decoding feature.

[0138] The first decoded feature can be input into a bilinear upsampling layer for upsampling, and the upsampling result can be concatenated with the second fused feature. For example, the upsampling result and the second fused feature can be added channel by channel to obtain a concatenated feature. Alternatively, the second fused feature can be directly concatenated with the first decoded feature to obtain a concatenated feature.

[0139] The concatenated feature is then input into the second decoding stage for decoding to obtain a second decoding feature output by the second decoding stage, wherein the second decoding stage includes a DRDB and a preset number of SAMs connected in series.

[0140] S603 : Decode the third fusion feature and the second decoding feature through a third decoding stage to obtain a third decoding feature.

[0141] The second decoded feature can be input into a bilinear upsampling layer for upsampling, and the upsampling result is concatenated with the third fused feature channel by channel. The concatenated result is input into a third decoding stage, and processed by a DRDB and a preset number of SAMs included in the third decoding stage to obtain a third decoded feature. The third decoding stage includes a DRDB and a preset number of SAMs connected in series.

[0142] Alternatively, the third decoding feature may be determined by the method described below.

[0143] S604: Obtain a restored image based on the third decoding feature.

[0144] The third decoded feature may be subjected to convolution processing and pixel shuffle upsample processing to obtain a restored image.

[0145] Through the above method, when performing feature decoding in the decoding stage in the embodiment of the present disclosure, it can not only be based on the processing results of the previous decoding stage, but also based on the fusion features obtained by fusing the encoding features of the three encoding stages, so that the decoding stage can have more features to refer to, and the understanding of the document image is deeper, thereby improving the moiré removal effect of the document image.

[0146] The method of obtaining the third decoding feature in S603 includes the following steps:

[0147] Step 1: Perform convolution processing on the second decoding feature to obtain a convolution feature.

[0148] The convolution process is performed by a convolution layer with a convolution kernel of 1×1.

[0149] Step 2: Reconstruct the convolution features through the reconstruction module to obtain the reconstructed features.

[0150] Step 3: Splice the third fusion feature and the reconstructed feature to obtain a spliced ​​feature.

[0151] The third fusion feature and the reconstruction feature can be added channel by channel to obtain the splicing feature.

[0152] Step 4: Input the splicing features into the third decoding stage for decoding to obtain the third decoding features output by the third decoding stage.

[0153] The embodiment of the present disclosure adds convolution and feature reconstruction processing between the second decoding stage and the third decoding stage to obtain deeper features from the second decoding features, improve the feature extraction capability of the removal model, and ensure the effect of removing moiré patterns from document images.

[0154] In some embodiments of the present disclosure, the reconstruction module includes a plurality of multi-scale residual group modules (MSRGMs) connected in series. Figure 7 , Figure 7 The reconstruction module shown includes N residual group modules (RGMs). For example, the reconstruction module includes three MSRGMs connected in series.

[0155] The structure of each multi-scale residual group module is the same, and each multi-scale residual group module includes: a first residual group module, a second residual group module, a third residual group module and a convolutional layer.

[0156] The processing of input features by each multi-scale residual group module includes the following steps:

[0157] Step (1): Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features.

[0158] The input features are input into the first RGM, and the first RGM is used to perform feature extraction based on the channel attention mechanism on the input features to obtain the first branch features.

[0159] When the MSRGM is the first MSRGM included in the reconstruction module, the input feature is the convolution feature; when the MSRGM is not the first MSRGM included in the reconstruction module, the input feature is the output result of the previous MSRGM.

[0160] Step (2): perform a first downsampling process on the input feature to obtain a first downsampling feature.

[0161] See also Figure 7 , perform DePixelShuffle for different scales on the input features to achieve feature downsampling and obtain the first downsampled features.

[0162] Step (3): input the first down-sampled feature into the second residual group module for feature extraction based on the channel attention mechanism, and perform the first up-sampling process on the feature extraction result to obtain the second branch feature.

[0163] See also Figure 7 , the first down-sampled feature is input into the second RGM, and the second RGM is used to perform feature extraction based on the channel attention mechanism on the first down-sampled feature, and the feature extraction results are subjected to pixel shuffle for different scales to achieve feature upsampling and obtain the second branch feature.

[0164] The sampling multiples of the first downsampling process and the first upsampling process may be the same, for example, the first downsampling process is 2x downsampling, and the first upsampling process is 2x upsampling.

[0165] Step (4): perform a second downsampling process on the input features to obtain second downsampling features.

[0166] See also Figure 7, performing a DePixelShuffle for different scales on the input features to achieve feature downsampling and obtain second downsampled features. The first downsampling process and the second downsampling process have different sampling magnifications, resulting in different sizes of the first downsampled features and the second downsampled features. For example, the first downsampling process is a 2x downsampling process, resulting in each dimension of the downsampled features being 1 / 2 of the corresponding dimension of the features before downsampling; the second downsampling process is a 4x downsampling process, resulting in each dimension of the downsampled features being 1 / 4 of the corresponding dimension of the features before downsampling.

[0167] Step (5): input the second down-sampled features into the third residual group module for feature extraction based on the channel attention mechanism, and perform a second up-sampling process on the feature extraction results to obtain the third branch features.

[0168] See also Figure 7 , the second down-sampled features are input into the third RGM, and the third RGM is used to perform feature extraction based on the channel attention mechanism on the second down-sampled features, and the feature extraction results are subjected to PixelShuffle fordifferent scale to achieve feature upsampling and obtain the third branch features.

[0169] The sampling multiples of the second downsampling process and the second upsampling process may be the same, for example, the second downsampling process is 4-fold downsampling, and the second upsampling process is 4-fold upsampling.

[0170] Step (6) concatenates the input features, the first branch features, the second branch features, and the third branch features, and performs convolution processing on the concatenated features using a convolution layer to obtain a processing result.

[0171] See also Figure 7 The input features, first branch features, second branch features, and third branch features are concatenated channel by channel. The concatenated features are then fed into a 1×1 convolution layer, which performs convolution processing on the concatenated features to obtain the processing results. If the MSRGM is not the last MSRGM included in the reconstruction module, the processing results will serve as the input features of the next MSRGM; if the MSRGM is the last MSRGM included in the reconstruction module, the processing results will serve as the output of the reconstruction module.

[0172] Since the residual group module can perform feature extraction based on the channel attention mechanism, the embodiment of the present disclosure can pay more attention to the feature areas that are conducive to moiré removal during the feature extraction process, thereby improving the accuracy of moiré removal. Moreover, the embodiment of the present disclosure performs different degrees of downsampling on the input features before the second residual group module and the third residual group module perform feature extraction, so that the three residual group modules can perform feature extraction on input features of different scales, thereby coping with the problem of moiré removal of document images of various resolutions. Moreover, experiments have shown that the reconstruction module provided by the embodiment of the present disclosure has a better effect on removing noise such as reflections, low light and shadows in document images, improves the accuracy of the color and content of the restored image, and improves the image quality of the restored image.

[0173] The following describes the removal model structure provided by the embodiment of the present disclosure and the conventional ESDNet structure in combination with actual application scenarios:

[0174] The conventional ESDNet structure is as follows Figure 8 As shown in Figure 1, ESDNet includes: downsampling layer, first convolutional layer, encoder, decoder, second convolutional layer and pixel shuffling upsampling layer.

[0175] The encoder and decoder both consist of three stages connected in series, with each stage consisting of a DRDB and a SAM connected in series. A bilinear upsampling layer is also included between each two decoding stages of the decoder.

[0176] Each encoding stage of the encoder also has a skip connection with a decoding stage of the decoder. Specifically, there is a skip connection between the first encoding stage and the third decoding stage, a skip connection between the second encoding stage and the second decoding stage, and a skip connection between the third encoding stage and the first decoding stage.

[0177] The removal model structure provided by the embodiment of the present disclosure is as follows Figure 9 As shown, the removal model includes: a downsampling layer, a first convolutional layer, an encoder, a first multi-scale interaction module, a second multi-scale interaction module, a third multi-scale interaction module, a decoder, a second convolutional layer and a pixel shuffling upsampling layer.

[0178] The encoder and decoder both include three stages connected in series, and each stage includes a DRDB and four SAMs connected in series. Figure 9 Only one SAM is shown in each stage, but each stage may actually include four SAMs connected in series.

[0179] The first multi-scale interaction module is connected to the three encoding stages and the first decoding stage respectively; the second multi-scale interaction module is connected to the three encoding stages and the second decoding stage respectively; and the third multi-scale interaction module is connected to the three encoding stages and the third decoding stage respectively.

[0180] There is a convolution layer with a 1×1 convolution kernel and a reconstruction module in series between the second decoding stage and the third decoding stage of the encoder.

[0181] By inputting a document image with moiré patterns into a removal model, a restored image after removing the moiré patterns output by the removal model can be obtained.

[0182] By comparison, it can be seen that compared with the conventional ESDNet, in the removal model provided by the embodiment of the present disclosure, each stage of the encoder and decoder includes more SAMs, which can more effectively extract multi-scale features aligned at the same semantic level, and at the same time allocate attention to different features, thereby selectively fusing multi-scale features and improving the feature extraction capability of the removal model.

[0183] Furthermore, skip connections do not process the features extracted in the encoding stage; instead, they are directly input into a single decoding stage, and the decoding stage can only receive the processing results of a single encoding stage via a single skip connection. The removal model in the disclosed embodiment utilizes a multi-scale interaction module to fuse the processing results of the three encoding stages and input them into each decoding stage. This means that each decoding stage can obtain the fused results of the three encoding stages, enriching the features used by the decoding stage, improving the accuracy of image restoration, and enhancing the effectiveness of moiré removal.

[0184] In addition, the embodiment of the present disclosure also adds a reconstruction module between the second decoding stage and the third decoding stage, which can improve the processing effect of noise such as reflections, low light, shadows, etc. in the image, and further improve the image quality of the restored image.

[0185] Based on the same inventive concept, the embodiment of the present disclosure also provides a model training method, which is applied to an electronic device, wherein the electronic device to which the model training method is applied and the electronic device to which the above-mentioned image processing method is applied can be the same electronic device, or different electronic devices. Figure 10 As shown, the model training method provided by the embodiment of the present disclosure includes the following steps:

[0186] S1001. Obtain a sample image and a training label of the sample image.

[0187] The sample image is a document image with moiré patterns, and the training label is a document image without moiré patterns.

[0188] For example, multiple images without moiré patterns may be obtained and used as training labels, and moiré patterns may be added to these images as sample images corresponding to the training labels.

[0189] Alternatively, multiple images with moiré patterns may be obtained as sample images, and images obtained by removing the moiré patterns from these images using other methods or manually may be obtained as training labels for the sample images.

[0190] Alternatively, the sample images and their training labels may be obtained by other means, which are not specifically limited in the embodiments of the present disclosure.

[0191] Traditionally, ESDNet training uses sample images with a fixed resolution. For example, images from the UHDM4k dataset are used as sample images. The images in the UHDM4k dataset all have a 4k resolution, meaning they have approximately 4,000 pixels horizontally. For example, the resolution of a 4k image is 3840×2160 or 4096×2160.

[0192] To improve the effectiveness of the trained removal model in removing moiré patterns from document images, sample images of various resolutions can be obtained, thereby improving the effectiveness of the removal model in removing moiré patterns from document images of various resolutions. For example, the sample images obtained in the embodiments of the present disclosure can have resolutions ranging from 200×200 to 4k.

[0193] S1002: Encode the sample image by removing the encoder included in the model to obtain encoding features output by multiple encoding stages of the encoder, wherein the sizes of the encoding features are different.

[0194] The specific implementation method of S1002 is the same as the above S102. Please refer to the relevant description of S102 and will not be repeated here.

[0195] S1003 , fusing the encoding features by removing the multi-scale interaction module included in the model to obtain a fused feature.

[0196] The specific implementation method of S1003 is the same as that of the above S103. Please refer to the relevant description of S103 and will not be repeated here.

[0197] S1004: Decode the fused features by removing the decoder included in the model to obtain decoding features output by multiple decoding stages of the decoder.

[0198] The structure and specific processing process of the decoder can be referred to the relevant description in the above S104, which will not be repeated here.

[0199] S1005: Determine a loss value based on the decoded features and the training labels.

[0200] The decoded features output by each decoding stage can be processed using a convolutional layer and a pixel-shuffling upsampling layer to produce a predicted image. The decoded features output by the third decoding stage are processed by the convolutional layer and the pixel-shuffling upsampling layer to produce a predicted image, also known as a restored image.

[0201] For example, the predicted image and training labels can be substituted into the following loss function to calculate the loss value.

[0202]

[0203] in, represents the loss value, Indicates the calculation of mean absolute error, Indicates the calculation of perceptual loss (Perceptual Loss), is the predicted image obtained based on the decoding features of the i-th decoding stage, for The corresponding label, The corresponding label can be set to the resolution of the training label The image obtained after the resolution is Indicates preset parameters, such as .

[0204] S1006. Use the loss value to adjust the network parameters of the removal model, return to S1002, and encode the sample image through the encoder included in the removal model until the removal model converges, and determine that the training is completed.

[0205] It can be determined whether the removal mode has converged. If not, the network parameters of the removal model are adjusted using the loss value through back propagation and gradient descent, and return to S1002 to continue training the removal model based on the next batch of sample images and their training labels. If converged, it is determined that the training is complete. Among them, if the number of iterations reaches a preset number, it can be determined that it has converged; otherwise, it is determined that it has not converged. Alternatively, if the loss value calculated in this iteration is less than a preset threshold, it can be determined that it has converged; otherwise, it is determined that it has not converged. Alternatively, if the error between the loss values ​​calculated in the last N iterations is less than a preset error, it can be determined that it has converged; otherwise, it is determined that it has not converged. Alternatively, it can be determined that whether the removal model has converged can be determined by other means, and the embodiments of the present disclosure do not specifically limit this.

[0206] In the disclosed embodiment, the depth of features that can be extracted by the encoder of the removal model at different encoding stages is different, so the features extracted at different encoding stages are not exactly the same, and the features extracted at each encoding stage cover a wider range of features as a whole. Therefore, the disclosed embodiment fuses the encoding features through the multi-scale interaction module of the removal model, and decodes the fused features through the decoder of the removal model, so that when the decoder constructs the restored image, it can be based on the rich features extracted at each encoding stage, thereby having a deeper understanding of the text image, so that the removal model can more accurately remove moiré patterns from the text image.

[0207] In some embodiments of the present disclosure, the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the above-mentioned S1002 encodes the sample image by removing the encoder included in the model to obtain the encoding features output by the multiple encoding stages of the encoder, including the following steps:

[0208] Encoding the sample image through a first encoding stage to obtain a first encoding feature;

[0209] Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage;

[0210] The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

[0211] Through the above method, the embodiment of the present disclosure can obtain the coding features output by the encoder in three different coding stages. The features included in these three coding features are not exactly the same, so these three coding features cover a wider range of features, making the subsequent moiré removal effect through these three features better.

[0212] In some embodiments of the present disclosure, each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0213] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré patterns, by increasing the number of SAMs included in the encoding stage, the process of removing moiré patterns from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of moiré removal.

[0214] In some embodiments of the present disclosure, the above-mentioned S1003 removes the multi-scale interaction module included in the model to fuse the coding features to obtain the fused features, including the following steps:

[0215] Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature;

[0216] Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature;

[0217] Each encoded feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

[0218] Through the above method, the embodiment of the present disclosure can fuse the encoding features through multiple multi-scale interaction modules and obtain three fused features of different sizes, so that the size of each fused feature meets the size requirements of the input features of a decoding stage of the decoder.

[0219] In some embodiments of the present disclosure, the first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the above-mentioned method of inputting each encoding feature into the first multi-scale interaction module to obtain the first fusion feature includes the following steps:

[0220] Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature;

[0221] Inputting the second coding feature into the second first interaction module for processing to obtain a second excitation feature;

[0222] Inputting the third coding feature into the third first interaction module for processing to obtain a third excitation feature;

[0223] Upsampling the second excitation feature and the third excitation feature respectively to obtain two upsampled features;

[0224] The first excitation feature and the two upsampled features are concatenated, and the concatenated features are input into the second interaction module to obtain a first interaction feature;

[0225] The first interaction feature is concatenated with each coding feature to obtain a first fusion feature.

[0226] Through the above method, the disclosed embodiments can obtain more comprehensive and richer fused features by fusing coding features of different scales. This fused feature helps the subsequent decoder capture more image details and contextual information, thereby improving the accuracy of the subsequent restored image.

[0227] In some embodiments of the present disclosure, the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and a compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the above-mentioned method of inputting each encoded feature into the second multi-scale interaction module to obtain the second fused feature includes the following steps:

[0228] Inputting the first coding feature into the first third interaction module for processing to obtain a fourth excitation feature;

[0229] Inputting the second coding feature into the second third interaction module for processing to obtain a fifth excitation feature;

[0230] Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature;

[0231] Upsampling the fourth excitation feature to obtain a downsampled feature;

[0232] Downsampling the sixth excitation feature to obtain an upsampled feature;

[0233] The up-sampled feature, the fifth excitation feature, and the down-sampled feature are concatenated, and the concatenated feature is input into the fourth interaction module to obtain a second interaction feature;

[0234] The second interaction feature is concatenated with each coding feature to obtain a second fusion feature.

[0235] Through the above method, the disclosed embodiments can obtain more comprehensive and richer fused features by fusing coding features of different scales. This fused feature helps the subsequent decoder capture more image details and contextual information, thereby improving the accuracy of the subsequent restored image.

[0236] In some embodiments of the present disclosure, the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the above-mentioned method of inputting each encoded feature into the third multi-scale interaction module to obtain the third fused feature includes the following steps:

[0237] Inputting the first coding feature into the first fifth interaction module for processing to obtain a seventh excitation feature;

[0238] Inputting the second coding feature into the second fifth interaction module for processing to obtain an eighth excitation feature;

[0239] Inputting the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature;

[0240] Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features;

[0241] The two downsampled features and the ninth excitation feature are concatenated, and the concatenated features are input into the sixth interaction module to obtain a third interaction feature;

[0242] The third interactive feature is concatenated with each coding feature to obtain a third fusion feature.

[0243] Through the above method, the disclosed embodiments can obtain more comprehensive and richer fused features by fusing coding features of different scales. This fused feature helps the subsequent decoder capture more image details and contextual information, thereby improving the accuracy of the subsequent restored image.

[0244] In some embodiments of the present disclosure, the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; the above-mentioned S1004 decodes the fusion features by removing the decoder included in the model to obtain the decoding features output by the multiple decoding stages of the decoder, including the following steps:

[0245] Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature;

[0246] Decoding the second fusion feature and the first decoding feature in a second decoding stage to obtain a second decoding feature;

[0247] The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

[0248] Through the above method, when performing feature decoding in the decoding stage in the embodiment of the present disclosure, it can not only be based on the processing results of the previous decoding stage, but also based on the fusion features obtained by fusing the encoding features of the three encoding stages, so that the decoding stage can have more features to refer to, and the understanding of the document image is deeper, thereby improving the moiré removal effect of the document image.

[0249] In some embodiments of the present disclosure, the method of decoding the third fusion feature and the second decoding feature in the third decoding stage to obtain the third decoding feature includes the following steps:

[0250] Performing convolution processing on the second decoding feature to obtain a convolution feature;

[0251] The convolution feature is reconstructed through the reconstruction module to obtain the reconstructed feature;

[0252] Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature;

[0253] The concatenated features are input into the third decoding stage for decoding to obtain the third decoding features output by the third decoding stage.

[0254] The embodiment of the present disclosure adds convolution and feature reconstruction processing between the second decoding stage and the third decoding stage to obtain deeper features from the second decoding features, improve the feature extraction capability of the removal model, and ensure the effect of removing moiré patterns from document images.

[0255] In some embodiments of the present disclosure, each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0256] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré patterns, by increasing the number of SAMs included in the decoding stage, the process of removing moiré patterns from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of moiré removal.

[0257] In some embodiments of the present disclosure, the reconstruction module includes multiple multi-scale residual group modules connected in series, each of which includes a first residual group module, a second residual group module, a third residual group module, and a convolutional layer. The multi-scale residual group module processes input features by the following steps:

[0258] Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features;

[0259] Performing a first downsampling process on the input feature to obtain a first downsampling feature;

[0260] The first down-sampled features are input into the second residual group module for feature extraction based on the channel attention mechanism, and the feature extraction results are subjected to a first up-sampling process to obtain second branch features;

[0261] Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes;

[0262] The second down-sampled features are input into the third residual group module for feature extraction based on the channel attention mechanism, and the feature extraction results are subjected to a second up-sampling process to obtain the third branch features;

[0263] The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using a convolution layer to obtain the processing results.

[0264] Since the residual group module can perform feature extraction based on the channel attention mechanism, the embodiment of the present disclosure can pay more attention to the feature areas that are conducive to moiré removal during the feature extraction process, thereby improving the accuracy of moiré removal. Moreover, the embodiment of the present disclosure performs different degrees of downsampling on the input features before the second residual group module and the third residual group module perform feature extraction, so that the three residual group modules can perform feature extraction on input features of different scales, thereby coping with the problem of moiré removal of document images of various resolutions. Moreover, experiments have shown that the reconstruction module provided by the embodiment of the present disclosure has a better effect on removing noise such as reflections, low light and shadows in document images, improves the accuracy of the color and content of the restored image, and improves the image quality of the restored image.

[0265] In some embodiments of the present disclosure, the method of encoding the sample image in the first encoding stage to obtain the first encoding feature includes the following steps:

[0266] Perform downsampling and convolution processing on the sample image to obtain preprocessing features;

[0267] The preprocessed features are input into the first encoding stage for encoding to obtain the first encoded features.

[0268] Since ESDNet needs to undergo downsampling and convolution processing before encoding the image through the encoder, the embodiment of the present disclosure first downsamples and convolutions the image to be processed, and then processes it through the encoder, thereby reducing the damage to the original structure of ESDNet, ensuring the smoothness of the image processing process, and improving the effect of removing moiré on document images.

[0269] In the model training method, the specific implementation of each step can be referred to the description of the relevant steps of the above-mentioned image processing method, which will not be repeated here.

[0270] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of document images involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0271] It should be noted that the document image in this embodiment is not a document image for a specific user and cannot reflect the personal information of a specific user.

[0272] It should be noted that the document image in this embodiment may come from a public dataset.

[0273] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure further provides an image processing device, such as Figure 11As shown, it includes: an acquisition module 1101, an encoding module 1102, a fusion module 1103 and a decoding module 1104;

[0274] An acquisition module 1101 is configured to acquire an image to be processed, where the image to be processed is a document image with moiré patterns.

[0275] An encoding module 1102 is configured to encode the image to be processed by an encoder to obtain encoding features outputted by multiple encoding stages of the encoder;

[0276] A fusion module 1103 is used to fuse the coding features through a multi-scale interaction module to obtain a fused feature;

[0277] The decoding module 1104 is configured to decode the fused features through a decoder to obtain a restored image after removing moiré patterns from the image to be processed.

[0278] In some embodiments of the present disclosure, the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the encoding module 1102 is specifically configured to:

[0279] Encoding the image to be processed through the first encoding stage to obtain a first encoding feature;

[0280] Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage;

[0281] The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

[0282] In some embodiments of the present disclosure, each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0283] In some embodiments of the present disclosure, the fusion module 1103 is specifically configured to:

[0284] Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature;

[0285] Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature;

[0286] Each encoded feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

[0287] In some embodiments of the present disclosure, the first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1103 is specifically used to:

[0288] Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature;

[0289] Inputting the second coding feature into the second first interaction module for processing to obtain a second excitation feature;

[0290] Inputting the third coding feature into the third first interaction module for processing to obtain a third excitation feature;

[0291] Upsampling the second excitation feature and the third excitation feature respectively to obtain two upsampled features;

[0292] The first excitation feature and the two upsampled features are concatenated, and the concatenated features are input into the second interaction module to obtain a first interaction feature;

[0293] The first interaction feature is concatenated with each coding feature to obtain a first fusion feature.

[0294] In some embodiments of the present disclosure, the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and a compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1103 is specifically configured to:

[0295] Inputting the first coding feature into the first third interaction module for processing to obtain a fourth excitation feature;

[0296] Inputting the second coding feature into the second third interaction module for processing to obtain a fifth excitation feature;

[0297] Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature;

[0298] Upsampling the fourth excitation feature to obtain a downsampled feature;

[0299] Downsampling the sixth excitation feature to obtain an upsampled feature;

[0300] The up-sampled feature, the fifth excitation feature, and the down-sampled feature are concatenated, and the concatenated feature is input into the fourth interaction module to obtain a second interaction feature;

[0301] The second interaction feature is concatenated with each coding feature to obtain a second fusion feature.

[0302] In some embodiments of the present disclosure, the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1103 is specifically used to:

[0303] Inputting the first coding feature into the first fifth interaction module for processing to obtain a seventh excitation feature;

[0304] Inputting the second coding feature into the second fifth interaction module for processing to obtain an eighth excitation feature;

[0305] Inputting the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature;

[0306] Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features;

[0307] The two downsampled features and the ninth excitation feature are concatenated, and the concatenated features are input into the sixth interaction module to obtain a third interaction feature;

[0308] The third interactive feature is concatenated with each coding feature to obtain a third fusion feature.

[0309] In some embodiments of the present disclosure, the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; the decoding module 1104 is specifically configured to:

[0310] Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature;

[0311] Decoding the second fusion feature and the first decoding feature in a second decoding stage to obtain a second decoding feature;

[0312] Decoding the third fusion feature and the second decoding feature through a third decoding stage to obtain a third decoding feature;

[0313] Based on the third decoding feature, a restored image is obtained.

[0314] In some embodiments of the present disclosure, the decoding module 1104 is specifically configured to:

[0315] Performing convolution processing on the second decoding feature to obtain a convolution feature;

[0316] The convolution feature is reconstructed through the reconstruction module to obtain the reconstructed feature;

[0317] Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature;

[0318] The concatenated features are input into the third decoding stage for decoding to obtain the third decoding features output by the third decoding stage.

[0319] In some embodiments of the present disclosure, each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0320] In some embodiments of the present disclosure, the reconstruction module includes multiple multi-scale residual group modules connected in series, and the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the multi-scale residual group module processes the input features, including the following steps:

[0321] Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features;

[0322] Performing a first downsampling process on the input feature to obtain a first downsampling feature;

[0323] The first down-sampled features are input into the second residual group module for feature extraction based on the channel attention mechanism, and the feature extraction results are subjected to a first up-sampling process to obtain second branch features;

[0324] Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes;

[0325] The second down-sampled features are input into the third residual group module for feature extraction based on the channel attention mechanism, and the feature extraction results are subjected to a second up-sampling process to obtain the third branch features;

[0326] The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using a convolution layer to obtain the processing results.

[0327] In some embodiments of the present disclosure, the encoding module 1102 is specifically configured to:

[0328] Perform downsampling and convolution processing on the image to be processed to obtain preprocessing features;

[0329] The preprocessed features are input into the first encoding stage for encoding to obtain the first encoded features.

[0330] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure also provides a model training device, such as Figure 12As shown, the apparatus includes: an acquisition module 1201, an encoding module 1202, a fusion module 1203, a decoding module 1204, a determination module 1205 and an adjustment module 1206;

[0331] An acquisition module 1201 is configured to acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns and the training label is a document image without moiré patterns;

[0332] An encoding module 1202 is configured to encode the sample image by removing the encoder included in the model to obtain encoding features outputted by the encoder at multiple encoding stages;

[0333] A fusion module 1203 is configured to fuse the encoding features by removing the multi-scale interaction module included in the model to obtain a fused feature;

[0334] A decoding module 1204 is configured to decode the fused features by removing the decoder included in the model to obtain decoded features outputted by the multiple decoding stages of the decoder;

[0335] A determination module 1205 is configured to determine a loss value based on the decoded features and the training labels;

[0336] The adjustment module 1206 is used to adjust the network parameters of the removal model using the loss value, and return to the step of encoding the sample image through the encoder included in the removal model until the removal model converges, and determines that the training is completed.

[0337] In some embodiments of the present disclosure, the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the encoding module 1202 is specifically configured to:

[0338] Encoding the sample image through a first encoding stage to obtain a first encoding feature;

[0339] Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage;

[0340] The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

[0341] In some embodiments of the present disclosure, each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0342] In some embodiments of the present disclosure, the fusion module 1203 is specifically configured to:

[0343] Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature;

[0344] Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature;

[0345] Each encoded feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

[0346] In some embodiments of the present disclosure, the first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1203 is specifically configured to:

[0347] Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature;

[0348] Inputting the second coding feature into the second first interaction module for processing to obtain a second excitation feature;

[0349] Inputting the third coding feature into the third first interaction module for processing to obtain a third excitation feature;

[0350] Upsampling the second excitation feature and the third excitation feature respectively to obtain two upsampled features;

[0351] The first excitation feature and the two upsampled features are concatenated, and the concatenated features are input into the second interaction module to obtain a first interaction feature;

[0352] The first interaction feature is concatenated with each coding feature to obtain a first fusion feature.

[0353] In some embodiments of the present disclosure, the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1203 is specifically configured to:

[0354] Inputting the first coding feature into the first third interaction module for processing to obtain a fourth excitation feature;

[0355] Inputting the second coding feature into the second third interaction module for processing to obtain a fifth excitation feature;

[0356] Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature;

[0357] Upsampling the fourth excitation feature to obtain a downsampled feature;

[0358] Downsampling the sixth excitation feature to obtain an upsampled feature;

[0359] The up-sampled feature, the fifth excitation feature, and the down-sampled feature are concatenated, and the concatenated feature is input into the fourth interaction module to obtain a second interaction feature;

[0360] The second interaction feature is concatenated with each coding feature to obtain a second fusion feature.

[0361] In some embodiments of the present disclosure, the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1203 is specifically configured to:

[0362] Inputting the first coding feature into the first fifth interaction module for processing to obtain a seventh excitation feature;

[0363] Inputting the second coding feature into the second fifth interaction module for processing to obtain an eighth excitation feature;

[0364] Inputting the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature;

[0365] Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features;

[0366] The two downsampled features and the ninth excitation feature are concatenated, and the concatenated features are input into the sixth interaction module to obtain a third interaction feature;

[0367] The third interactive feature is concatenated with each coding feature to obtain a third fusion feature.

[0368] In some embodiments of the present disclosure, the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; the decoding module 1204 is specifically configured to:

[0369] Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature;

[0370] Decoding the second fusion feature and the first decoding feature in a second decoding stage to obtain a second decoding feature;

[0371] The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

[0372] In some embodiments of the present disclosure, the decoding module 1204 is specifically configured to:

[0373] Performing convolution processing on the second decoding feature to obtain a convolution feature;

[0374] The convolution feature is reconstructed through the reconstruction module to obtain the reconstructed feature;

[0375] Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature;

[0376] The concatenated features are input into the third decoding stage for decoding to obtain the third decoding features output by the third decoding stage.

[0377] In some embodiments of the present disclosure, each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0378] In some embodiments of the present disclosure, the reconstruction module includes multiple multi-scale residual group modules connected in series, and the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the multi-scale residual group module processes the input features, including the following steps:

[0379] Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features;

[0380] Performing a first downsampling process on the input feature to obtain a first downsampling feature;

[0381] The first down-sampled features are input into the second residual group module for feature extraction based on the channel attention mechanism, and the feature extraction results are subjected to a first up-sampling process to obtain second branch features;

[0382] Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes;

[0383] The second down-sampled features are input into the third residual group module for feature extraction based on the channel attention mechanism, and the feature extraction results are subjected to a second up-sampling process to obtain the third branch features;

[0384] The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using a convolution layer to obtain the processing results.

[0385] In some embodiments of the present disclosure, the encoding module 1202 is specifically configured to:

[0386] Perform downsampling and convolution processing on the sample image to obtain preprocessing features;

[0387] The preprocessed features are input into the first encoding stage for encoding to obtain the first encoded features.

[0388] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0389] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0390] like Figure 13 As shown, electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. RAM 1303 may also store various programs and data required for the operation of electronic device 1300. Computing unit 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to bus 1304.

[0391] Multiple components in electronic device 1300 are connected to I / O interface 1305, including: an input unit 1306, such as a keyboard, mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a magnetic disk, optical disk, etc.; and a communication unit 1309, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1309 allows electronic device 1300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0392] The computing unit 1301 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the image processing method and the model training method. For example, in some embodiments, the image processing method and the model training method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the image processing method and the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the image processing method or the model training method in any other appropriate manner (for example, by means of firmware).

[0393] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0394] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0395] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0396] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0397] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0398] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0399] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0400] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: Acquire an image to be processed, where the image to be processed is a document image with moiré patterns; Encoding the image to be processed by an encoder to obtain encoding features output by multiple encoding stages of the encoder, each encoding feature including: a first encoding feature, a second encoding feature, and a third encoding feature; The coding features are fused through the multi-scale interaction module to obtain a fused feature, wherein the fused feature includes: a first fused feature obtained by fusion of the first multi-scale interaction module; Decoding the fused features by a decoder to obtain a restored image after removing moiré patterns from the image to be processed; The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the first fusion feature is obtained by the following steps: Inputting the first encoded feature into a first residual block of a first interaction module for processing to obtain a first feature; Inputting the second encoded feature into the first residual block of the second first interaction module for processing to obtain a second feature; Inputting the third encoded feature into the first residual block of the third first interaction module for processing to obtain a third feature; Performing channel-by-channel addition based on the first feature and the second feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the first first interaction module to obtain a first excitation feature; performing channel-by-channel addition based on the first feature, the second feature, and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the second first interaction module to obtain a second excitation feature; performing channel-by-channel addition based on the second feature and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the third first interaction module to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interaction feature is concatenated with each coding feature to obtain the first fusion feature.

2. The method according to claim 1, wherein The encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; encoding the image to be processed by the encoder to obtain encoding features output by the multiple encoding stages of the encoder, including: Encoding the image to be processed through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

3. The method according to claim 2, wherein: Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

4. The method according to claim 2, wherein: The multi-scale interaction module is used to fuse the coding features to obtain the fused features, including: Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature; Each coding feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

5. The method according to claim 4, wherein The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and a compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the second multi-scale interaction module for fusion to obtain the second fused feature includes: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature, and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interaction feature is concatenated with each coding feature to obtain the second fusion feature.

6. The method according to claim 4, wherein: The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the third multi-scale interaction module for fusion to obtain a third fused feature includes: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two downsampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interaction feature is concatenated with each coding feature to obtain the third fusion feature.

7. The method according to claim 4, wherein: The decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; decoding the fusion features by the decoder to obtain a restored image after removing moiré patterns from the image to be processed includes: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; Decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature; The restored image is obtained based on the third decoding feature.

8. The method according to claim 7, wherein: The decoding of the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature includes: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

9. The method according to claim 7 or 8, wherein Each decoding stage includes a preset number of spatial attention modules connected in series, where the preset number is greater than 2.

10. The method according to claim 8, wherein The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the multi-scale residual group module processes the input features, including the following steps: Inputting the input features into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain first branch features; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled features into the second residual group module for feature extraction based on the channel attention mechanism, and performing a first up-sampling process on the feature extraction results to obtain second branch features; Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled features into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction results to obtain third branch features; The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using the convolution layer to obtain a processing result.

11. The method according to any one of claims 2 to 8, wherein: The step of encoding the image to be processed in the first encoding stage to obtain a first encoding feature includes: Performing downsampling and convolution processing on the image to be processed to obtain preprocessing features; The preprocessing features are input into the first encoding stage for encoding to obtain the first encoding features.

12. A model training method comprising: Acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns; Encoding the sample image by removing the encoder included in the model to obtain encoding features output by multiple encoding stages of the encoder, each encoding feature including: a first encoding feature, a second encoding feature, and a third encoding feature; The encoding features are fused by removing the multi-scale interaction module included in the model to obtain a fused feature, wherein the fused feature includes: a first fused feature obtained by fusing the first multi-scale interaction module; Decoding the fused features by removing the decoder included in the model to obtain decoding features outputted by multiple decoding stages of the decoder; Determining a loss value based on the decoded features and the training labels; adjusting the network parameters of the removal model using the loss value, returning to the step of encoding the sample image by the encoder included in the removal model, and determining that the training is completed when the removal model converges; The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the first fusion feature is obtained by the following steps: Inputting the first encoded feature into a first residual block of a first interaction module for processing to obtain a first feature; Inputting the second encoded feature into the first residual block of the second first interaction module for processing to obtain a second feature; Inputting the third encoded feature into the first residual block of the third first interaction module for processing to obtain a third feature; Performing channel-by-channel addition based on the first feature and the second feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the first first interaction module to obtain a first excitation feature; performing channel-by-channel addition based on the first feature, the second feature, and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the second first interaction module to obtain a second excitation feature; performing channel-by-channel addition based on the second feature and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the third first interaction module to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interaction feature is concatenated with each coding feature to obtain the first fusion feature.

13. The method according to claim 12, wherein: The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; The step of encoding the sample image by removing the encoder included in the model to obtain encoding features outputted by multiple encoding stages of the encoder includes: Encoding the sample image through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

14. The method according to claim 13, wherein Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

15. The method according to claim 13, wherein The multi-scale interaction module included in the removal model is removed to fuse the encoding features to obtain the fused features, including: Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature; Each coding feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

16. The method according to claim 15, wherein The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and a compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the second multi-scale interaction module for fusion to obtain the second fused feature includes: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature, and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interaction feature is concatenated with each coding feature to obtain the second fusion feature.

17. The method according to claim 15, wherein: The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the third multi-scale interaction module for fusion to obtain a third fused feature includes: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two downsampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interaction feature is concatenated with each coding feature to obtain the third fusion feature.

18. The method according to claim 15, wherein The decoder includes a first decoding stage, a second decoding stage and a third decoding stage; The decoding of the fusion feature by removing the decoder included in the model to obtain decoding features outputted by multiple decoding stages of the decoder respectively includes: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

19. The method according to claim 18, wherein The decoding of the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature includes: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

20. The method according to claim 18 or 19, wherein Each decoding stage includes a preset number of spatial attention modules connected in series, where the preset number is greater than 2.

21. The method according to claim 19, wherein The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the multi-scale residual group module processes the input features, including the following steps: Inputting the input features into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain first branch features; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled features into the second residual group module for feature extraction based on the channel attention mechanism, and performing a first up-sampling process on the feature extraction results to obtain second branch features; Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled features into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction results to obtain third branch features; The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using the convolution layer to obtain a processing result.

22. The method according to any one of claims 13 to 19, wherein: The step of encoding the sample image in the first encoding stage to obtain a first encoding feature includes: Performing downsampling and convolution processing on the sample image to obtain preprocessing features; The preprocessing features are input into the first encoding stage for encoding to obtain the first encoding features.

23. An image processing device comprising: An acquisition module, configured to acquire an image to be processed, wherein the image to be processed is a document image with moiré patterns; an encoding module, configured to encode the image to be processed by an encoder to obtain encoding features outputted respectively by a plurality of encoding stages of the encoder, wherein each encoding feature comprises: a first encoding feature, a second encoding feature, and a third encoding feature; A fusion module is used to fuse the coding features through the multi-scale interaction module to obtain a fusion feature, wherein the fusion feature includes: a first fusion feature obtained by fusion of the first multi-scale interaction module; A decoding module, configured to decode the fused features through a decoder to obtain a restored image after removing moiré patterns from the image to be processed; The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first encoded feature into a first residual block of a first interaction module for processing to obtain a first feature; Inputting the second encoded feature into the first residual block of the second first interaction module for processing to obtain a second feature; Inputting the third encoded feature into the first residual block of the third first interaction module for processing to obtain a third feature; Performing channel-by-channel addition based on the first feature and the second feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the first first interaction module to obtain a first excitation feature; performing channel-by-channel addition based on the first feature, the second feature, and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the second first interaction module to obtain a second excitation feature; performing channel-by-channel addition based on the second feature and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the third first interaction module to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interaction feature is concatenated with each coding feature to obtain the first fusion feature.

24. The device according to claim 23, wherein The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; the encoding module is specifically used to: Encoding the image to be processed through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

25. The apparatus according to claim 24, wherein Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

26. The apparatus according to claim 24, wherein The fusion module is specifically used to: Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature; Each coding feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

27. The device according to claim 26, wherein The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and a compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature, and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interaction feature is concatenated with each coding feature to obtain the second fusion feature.

28. The apparatus according to claim 26, wherein The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two downsampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interaction feature is concatenated with each coding feature to obtain the third fusion feature.

29. The apparatus according to claim 26, wherein The decoder includes a first decoding stage, a second decoding stage and a third decoding stage; the decoding module is specifically used to: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; Decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature; The restored image is obtained based on the third decoding feature.

30. The apparatus according to claim 29, wherein The decoding module is specifically used to: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

31. The apparatus according to claim 29 or 30, wherein Each decoding stage includes a preset number of spatial attention modules connected in series, where the preset number is greater than 2.

32. The apparatus according to claim 30, wherein The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the multi-scale residual group module processes the input features, including the following steps: Inputting the input features into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain first branch features; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled features into the second residual group module for feature extraction based on the channel attention mechanism, and performing a first up-sampling process on the feature extraction results to obtain second branch features; Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled features into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction results to obtain third branch features; The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using the convolution layer to obtain a processing result.

33. The device according to any one of claims 24 to 30, wherein: The encoding module is specifically used to: Performing downsampling and convolution processing on the image to be processed to obtain preprocessing features; The preprocessing features are input into the first encoding stage for encoding to obtain the first encoding features.

34. A model training device comprising: an acquisition module, configured to acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns; an encoding module, configured to encode the sample image by removing an encoder included in the model, and obtain encoding features outputted by a plurality of encoding stages of the encoder, wherein each encoding feature comprises: a first encoding feature, a second encoding feature, and a third encoding feature; A fusion module, configured to fuse the encoding features by removing the multi-scale interaction module included in the model to obtain a fused feature, wherein the fused feature includes: a first fused feature obtained by fusion of the first multi-scale interaction module; A decoding module, configured to decode the fused features by removing the decoder included in the model to obtain decoding features outputted by the multiple decoding stages of the decoder; A determination module, configured to determine a loss value based on the decoded features and the training labels; an adjustment module, configured to adjust network parameters of the removal model using the loss value, and return to the step of encoding the sample image by the encoder included in the removal model until the removal model converges, thereby determining that the training is completed; The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first encoded feature into a first residual block of a first interaction module for processing to obtain a first feature; Inputting the second encoded feature into the first residual block of the second first interaction module for processing to obtain a second feature; Inputting the third encoded feature into the first residual block of the third first interaction module for processing to obtain a third feature; Performing channel-by-channel addition based on the first feature and the second feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the first first interaction module to obtain a first excitation feature; performing channel-by-channel addition based on the first feature, the second feature, and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the second first interaction module to obtain a second excitation feature; performing channel-by-channel addition based on the second feature and the third feature, and processing the addition result sequentially through the second residual block and the compression and excitation block of the third first interaction module to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interaction feature is concatenated with each coding feature to obtain the first fusion feature.

35. The apparatus of claim 34, wherein: The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; the encoding module is specifically used to: Encoding the sample image through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for encoding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

36. The apparatus of claim 35, wherein: Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

37. The apparatus of claim 35, wherein: The fusion module is specifically used to: Inputting each coding feature into the first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each coding feature into the second multi-scale interaction module for fusion to obtain a second fused feature; Each coding feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

38. The apparatus according to claim 37, wherein The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and a compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature, and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interaction feature is concatenated with each coding feature to obtain the second fusion feature.

39. The apparatus according to claim 37, wherein The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and a compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two downsampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interaction feature is concatenated with each coding feature to obtain the third fusion feature.

40. The apparatus of claim 37, wherein The decoder includes a first decoding stage, a second decoding stage and a third decoding stage; the decoding module is specifically used to: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

41. The apparatus according to claim 40, wherein The decoding module is specifically used to: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstructed feature to obtain a spliced ​​feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

42. The apparatus according to claim 40 or 41, wherein Each decoding stage includes a preset number of spatial attention modules connected in series, where the preset number is greater than 2.

43. The apparatus according to claim 41, wherein The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the multi-scale residual group module processes the input features, including the following steps: Inputting the input features into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain first branch features; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled features into the second residual group module for feature extraction based on the channel attention mechanism, and performing a first up-sampling process on the feature extraction results to obtain second branch features; Performing a second downsampling process on the input feature to obtain a second downsampling feature, where the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled features into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction results to obtain third branch features; The input features, the first branch features, the second branch features, and the third branch features are spliced, and the spliced ​​features are convolved using the convolution layer to obtain a processing result.

44. The device according to any one of claims 35 to 41, wherein The encoding module is specifically used to: Performing downsampling and convolution processing on the sample image to obtain preprocessing features; The preprocessing features are input into the first encoding stage for encoding to obtain the first encoding features.

45. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 22.

46. ​​A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-22.

47. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 22.

Images