Image processing method and device, model training method and device, electronic equipment and medium

Abstract

The invention provides an image processing method, a model training method and device, electronic equipment and a medium, and relates to the technical field of image processing, in particular to the technical field of image denoising and image enhancement. According to the specific implementation scheme, a to-be-processed image is obtained, the to-be-processed image is a document image with moire patterns, then the to-be-processed image is encoded through an encoder, and encoding features output by the encoder in multiple encoding stages are obtained. And fusing the coding features through a multi-scale interaction module to obtain fused features, and decoding the fused features through a decoder to obtain a restored image after moire removal of the to-be-processed image. Moire pattern removal can be carried out on the document image more accurately.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to the field of image denoising and image enhancement technology. Background Art

[0002] Moiré is an interference fringe pattern that is common in photography or scanning. Taking photography as an example, for mobile phones, cameras or other devices with shooting functions, when the pixel spatial frequency of the photosensitive element of the device is close to the spatial frequency of the pattern of the object being photographed, moiré will appear in the image captured by the device. For example, when a mobile phone uses a built-in camera to capture a picture on a computer screen, the captured image has moiré. Among them, moiré usually appears as a rainbow-colored and distorted grid pattern or artifact, which has a serious impact on the visual quality of the image. Summary of the invention

[0003] The present disclosure provides an image processing method, a model training method, a device, an electronic device and a medium.

[0004] A first aspect of the embodiments of the present disclosure provides an image processing method, including:

[0005] Acquire an image to be processed, where the image to be processed is a document image with moiré patterns;

[0006] Encoding the image to be processed by an encoder to obtain encoding features outputted by multiple encoding stages of the encoder respectively;

[0007] Each encoding feature is fused through a multi-scale interaction module to obtain a fused feature;

[0008] The fusion feature is decoded by a decoder to obtain a restored image after the moiré pattern is removed from the image to be processed.

[0009] A second aspect of the embodiments of the present disclosure provides a model training method, including:

[0010] Acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns;

[0011] Encoding the sample image by removing the encoder included in the model to obtain encoding features outputted by multiple encoding stages of the encoder respectively;

[0012] By removing the multi-scale interaction module included in the model, each encoding feature is fused to obtain a fused feature;

[0013] Decoding the fused features by removing the decoder included in the model to obtain decoding features respectively output by multiple decoding stages of the decoder;

[0014] Determining a loss value based on the decoded features and the training labels;

[0015] The network parameters of the removal model are adjusted using the loss value, and the step of encoding the sample image by the encoder included in the removal model is returned to until the removal model converges, and the training is determined to be completed.

[0016] According to a third aspect of the embodiments of the present disclosure, there is provided an image processing device, including:

[0017] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed is a document image with moiré patterns;

[0018] An encoding module, used for encoding the image to be processed by an encoder to obtain encoding features outputted by multiple encoding stages of the encoder respectively;

[0019] A fusion module is used to fuse the encoding features through a multi-scale interaction module to obtain a fusion feature;

[0020] The decoding module is used to decode the fusion feature through a decoder to obtain a restored image after the moiré pattern is removed from the image to be processed.

[0021] A fourth aspect of the embodiments of the present disclosure provides a model training device, including:

[0022] An acquisition module, used for acquiring a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns;

[0023] An encoding module, used for encoding the sample image by removing the encoder included in the model, and obtaining encoding features respectively output by multiple encoding stages of the encoder;

[0024] A fusion module, used to fuse the encoding features by removing the multi-scale interaction module included in the model to obtain a fusion feature;

[0025] A decoding module, used for decoding the fused features by removing the decoder included in the model, and obtaining decoding features respectively output by multiple decoding stages of the decoder;

[0026] A determination module, configured to determine a loss value based on the decoded features and the training labels;

[0027] An adjustment module is used to adjust the network parameters of the removal model using the loss value, return to the step of encoding the sample image through the encoder included in the removal model, and determine that the training is completed when the removal model converges.

[0028] According to a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of the first aspect or the second aspect.

[0032] According to a sixth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods of the first aspect or the second aspect.

[0033] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method according to any one of the first aspect or the second aspect.

[0034] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0036] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present disclosure;

[0037] Figure 2 is a flow chart of a feature encoding method provided by an embodiment of the present disclosure;

[0038] Figure 3 is an exemplary schematic diagram of a processing process of a spatial attention module provided by an embodiment of the present disclosure;

[0039] Figure 4 is a flow chart of a feature fusion method provided by an embodiment of the present disclosure;

[0040] FIG5( a ) is a schematic diagram of the structure of a first multi-scale interaction module provided by an embodiment of the present disclosure;

[0041] FIG5( b ) is a schematic diagram of the structure of a second multi-scale interaction module provided by an embodiment of the present disclosure;

[0042] FIG5( c ) is a schematic diagram of the structure of a third multi-scale interaction module provided by an embodiment of the present disclosure;

[0043] Figure 6 is a flow chart of a feature decoding method provided by an embodiment of the present disclosure;

[0044] Figure 7 is a structural schematic diagram of a reconstruction module provided by an embodiment of the present disclosure;

[0045] Figure 8 This is a schematic diagram of the structure of the conventional ESDNet;

[0046] Fig. 9 is a structural schematic diagram of a removal model provided by an embodiment of the present disclosure;

[0047] Fig.10 is a flow chart of a model training method provided by an embodiment of the present disclosure;

[0048] Fig.11 is a structural schematic diagram of an image processing device provided by an embodiment of the present disclosure;

[0049] Fig.12 is a structural schematic diagram of a model training device provided by an embodiment of the present disclosure;

[0050] Fig.13 It is a block diagram of an electronic device used to implement the image processing method and model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0051] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0052] The current solution for removing moiré from images is mainly aimed at natural images such as portraits or landscapes, but the solution is less effective in removing moiré from document images such as books, test papers, certificates or bills.

[0053] In order to improve the effect of removing moiré from document images, the present disclosure provides an image processing method, which is applied to electronic devices, such as servers, desktop computers, laptop computers, or mobile phones, etc., which have image processing capabilities. Figure 1 As shown, the image processing method provided by the embodiment of the present disclosure includes the following steps:

[0054] S101: Obtain an image to be processed.

[0055] The image to be processed is a document image with moiré. That is, the image to be processed can be an image obtained by photographing a document such as a book, a test paper, a certificate, or a receipt through a device with an image shooting function such as a mobile phone, a tablet computer, or a camera, or through a scanner. For example, the image to be processed can be an image obtained by photographing a test paper displayed on a computer screen through a mobile phone.

[0056] Optionally, the image to be processed may be an image captured by a built-in camera of the electronic device, or an image frame in a video captured by a built-in camera. Alternatively, the image to be processed may be an image downloaded by the electronic device through a network. Alternatively, the image to be processed may be an image sent by other devices and received by the electronic device, etc. The embodiment of the present disclosure does not specifically limit the source of the image to be processed.

[0057] S102, encoding the image to be processed by an encoder to obtain encoding features outputted by multiple encoding stages of the encoder, wherein the sizes of the encoding features are different.

[0058] The encoder includes multiple encoding stages connected in series, and the feature depth extracted in each encoding stage increases successively. That is, the earlier the encoding stage, the shallower the image features can be extracted, such as the edges, corners, textures, and colors of the image; the later the encoding stage, the deeper the image features can be extracted, such as the local contours and semantic information of the object. The specific encoding process can be referred to the description below.

[0059] S103, fusing the encoding features through a multi-scale interaction module to obtain a fused feature.

[0060] The multi-scale interaction module may be a multi-scale feature interaction fusion (MSFIF) module. The specific fusion process may refer to the following description.

[0061] S104: Decode the fused features through a decoder to obtain a restored image after removing the moiré pattern from the image to be processed.

[0062] Since the fused features are obtained based on coding features of multiple sizes, and different coding features contain different feature depths, the decoder can combine shallow and deep features for decoding, thereby more accurately constructing a restored image after removing moiré from the processed image.

[0063] In the disclosed embodiment, the feature depths that can be extracted by the encoder at different encoding stages are different, so the features extracted at different encoding stages are not exactly the same, and the features extracted at each encoding stage cover a wider range of features as a whole. Therefore, the disclosed embodiment fuses the encoding features through a multi-scale interaction module, and decodes the fused features through a decoder, so that when the decoder constructs a restored image, it can be based on the richer features extracted at each encoding stage, so that the understanding of the image to be processed is deeper, so that the document image can be more accurately processed for moiré removal, and a more accurate restored image can be obtained.

[0064] The following is a detailed description of the image processing method provided by the embodiment of the present disclosure:

[0065] In some embodiments of the present disclosure, the encoder in the above S102 includes a first encoding stage, a second encoding stage and a third encoding stage.

[0066] On this basis, see Figure 2 The above S102 encodes the image to be processed by the encoder to obtain the encoding features output by multiple encoding stages of the encoder, including the following steps:

[0067] S201. Encode the image to be processed through a first encoding stage to obtain a first encoding feature.

[0068] In the embodiment of the present disclosure, the image to be processed may be downsampled and convolved to obtain preprocessing features, and then the preprocessing features may be input into the first encoding stage for encoding to obtain first encoding features.

[0069] The downsampling process may be pixel shuffle downsampling. The convolution process may be performed by a convolutional layer with a convolution kernel of 5×5.

[0070] In the disclosed embodiment, the encoder, the multi-scale interaction module and the decoder all belong to the removal model, and the removal model can be obtained by improving the efficient spiking deraining network (ESDNet). Since ESDNet needs to undergo downsampling and convolution processing before encoding the image through the encoder, the disclosed embodiment first performs downsampling and convolution processing on the image to be processed, and then processes it through the encoder, thereby reducing the damage to the original structure of ESDNet, ensuring the smoothness of the image processing process, and improving the effect of removing moiré on the document image.

[0071] In the disclosed embodiment, each encoding stage includes a preset number of spatial attention modules (SAMs) connected in series, and the preset number is greater than 2. For example, each encoding stage may include a dilated residual dense block (DRDB) and 4 SAMs connected in series. In the first encoding stage, the preprocessed features may be processed by the DRDB and the 4 SAMs in sequence to obtain the first encoded features.

[0072] See also Figure 3 , SAM's processing of input features includes: r Bilinear downsampling is used to generate F1, F2 and F3, where F is assumed to be rThe size of is H×W×C, then the size of F1 is H×W×C, then the size of F2 is H / 2×W / 2×C / 2, and the size of F3 is H / 4×W / 4×C / 4, where H represents the length, W represents the width, and C represents the number of channels. Then F1 is processed by the dilated dease block (DDK) and the convolution layer with a convolution kernel of 1×1 to obtain Y0. F2 is processed by DDK, a convolution layer with a convolution kernel of 1×1, and 2×Bilinear Up to obtain Y1. F3 is processed by DDK, a convolution layer with a convolution kernel of 1×1, and 4×Bilinear Up to obtain Y2. Global Average Pooling (GAP) is performed on Y0, Y1, and Y2 respectively to obtain v0, v1, and v2. Then, v0, v1, and v2 are concatenated channel by channel (Channel-wise Concatenation), and the concatenated features are input into the Multi-Layer Perceptron (MLP). The output of the MLP is split channel by channel (Channel-wise Split) to obtain w0, w1, and w2. Among them, w0 reflects the importance of Y0 for removing moiré, w1 reflects the importance of Y1 for removing moiré, and w2 reflects the importance of Y2 for removing moiré. Then Y0, Y1, Y2, w0, w1, and w2 are multiplied channel by channel (Channel-wise Multiplication), and the multiplication result is added to F r Add each channel to get the processing result F out .

[0073] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré, by increasing the number of SAMs included in the encoding stage, the process of removing moiré from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of removing moiré.

[0074] S202: Input the first coding feature into the second coding stage for coding, and obtain a second coding feature output by the second coding stage.

[0075] S203: Input the second coding feature into the third coding stage for coding, and obtain a third coding feature output by the third coding stage.

[0076] The processing of the second encoding stage and the third encoding stage is the same as that of the first encoding stage, and reference may be made to the relevant description in S201, which will not be repeated here.

[0077] Through the above method, the embodiment of the present disclosure can obtain the coding features output by the encoder in three different coding stages. The features included in these three coding features are not exactly the same, so these three coding features cover a wider range of features, making the subsequent moiré removal effect through these three features better.

[0078] In some embodiments of the present disclosure, see Figure 4 The above S103 fuses the coding features through the multi-scale interaction module to obtain the fused features, including the following steps:

[0079] S401: Input each coding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature.

[0080] Among them, the three multi-scale interaction modules can all be multi-scale feature fusion interaction (Multi-Scale Feature Interaction Fusion, MSFIF) modules.

[0081] S402: Input each coded feature into a second multi-scale interaction module for fusion to obtain a second fused feature.

[0082] S403: Input each coded feature into a third multi-scale interaction module for fusion to obtain a third fused feature. S401 to S403 may be executed serially or in parallel, and the embodiment of the present disclosure does not specifically limit the execution order of these three steps.

[0083] Among them, the structures of the first multi-scale interaction module, the second multi-scale interaction module and the third multi-scale interaction module are similar, and the sizes of the first fusion feature, the second fusion feature and the third fusion feature output by the three multi-scale interaction modules are different.

[0084] In the disclosed embodiment, the size of the first fusion feature is larger than the size of the second fusion feature, and the size of the second fusion feature is larger than the size of the third fusion feature.

[0085] Through the above method, the embodiment of the present disclosure can fuse the encoding features through multiple multi-scale interaction modules and obtain three fused features of different sizes, so that the size of each fused feature meets the size requirement of the input feature in a decoding stage of the decoder.

[0086] Referring to Figure 5 (a), the first multi-scale interaction module includes: three first interaction modules and one second interaction module. Each first interaction module includes two residual blocks and a squeeze-and-excitation (SE) block, and the second interaction module includes two convolutional layers and a squeeze-and-excitation block between the two convolutional layers. Assuming that the size of the first encoding feature is h×w×c, the size of the second encoding feature is h / 2×w / 2×c / 2, and the size of the third encoding feature is h / 4×w / 4×c / 4.

[0087] The method of obtaining the first fusion feature includes the following steps:

[0088] Step 1: Input the first coding feature into the first first interaction module for processing to obtain the first excitation feature.

[0089] Step 2: Input the second coding feature into the second first interaction module for processing to obtain a second excitation feature.

[0090] Step 3: Input the third coding feature into the third first interaction module for processing to obtain a third excitation feature.

[0091] Combined with steps 1 to 3, see Figure 5 (a), the first encoded feature is input into the first residual block of the first first interaction module to obtain the feature a1 output by the residual block. The second encoded feature is input into the first residual block of the second first interaction module to obtain the feature a2 output by the residual block. The third encoded feature is input into the first residual block of the third first interaction module to obtain the feature a3 output by the residual block. A1 is convolved with a convolution kernel of 1×1 and downsampled to obtain a1'. A2 is convolved with a convolution kernel of 1×1 and downsampled to obtain a2', and a2 is convolved with a convolution kernel of 1×1 and upsampled to obtain a2''. A3 is convolved with a convolution kernel of 1×1 and upsampled to obtain a3''. The upsampling and downsampling processes can make the resolution of the image to be processed compatible with the number of channels of the feature. In Figure 5 (a), the short dashed line represents convolution processing and downsampling, and the dotted dashed line represents convolution processing and upsampling.

[0092] A1 and a2'' are added channel by channel, and the addition result is processed by the second residual block and SE block of the first first interaction module in turn to obtain the first excitation feature. The size of the first excitation feature is h×w×c.

[0093] A1', a2 and a3'' are added channel by channel, and the addition result is processed by the second residual block and SE block of the second first interaction module in turn to obtain the second excitation feature. The size of the second excitation feature is h / 2×w / 2×2c.

[0094] A3 and a2' are added channel by channel, and the addition result is processed by the second residual block and SE block of the third first interaction module in sequence to obtain the third excitation feature. The size of the third excitation feature is h / 4×w / 4×4c.

[0095] Step 4: Upsample the second excitation feature and the third excitation feature respectively to obtain two upsampled features.

[0096] Referring to FIG. 5( a ), the second excitation feature may be convolved with a convolution kernel of 1×1 and upsampled to obtain an upsampled feature. Also, the third excitation feature may be convolved with a convolution kernel of 1×1 and upsampled to obtain an upsampled feature.

[0097] Step 5: Concatenate the first excitation feature and the two up-sampled features, and input the concatenated features into the second interaction module to obtain the first interaction feature.

[0098] As shown in Figure 5 (a), the first excitation feature and the two upsampled features can be concatenated channel by channel, and the size of the concatenated features is h×w×7c. The concatenated features are then input into the first convolution layer with a convolution kernel of 3×3 included in the second interactive module, and the size of the output result of the convolution layer is h×w×c. The output result of the convolution layer is then input into the SE block included in the second interactive module, and the output result of the SE block is input into the second convolution layer with a convolution kernel of 3×3 included in the second interactive module to obtain the first interactive feature output by the convolution layer.

[0099] Step 6: Concatenate the first interactive feature with each coding feature to obtain a first fusion feature.

[0100] Referring to FIG. 5( a ), the first interaction feature and each encoding feature may be added channel by channel to obtain a first fusion feature, and the size of the first fusion feature is h×w×c.

[0101] Through the above method, the embodiment of the present disclosure can obtain more comprehensive and richer fusion features by fusing coding features of different scales. Such fusion features help the subsequent decoder capture more image details and context information, thereby improving the accuracy of the subsequent generation of restored images.

[0102] Referring to FIG5(b), the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and one compression and excitation block between the two convolutional layers. Assuming that the size of the first encoding feature is h×w×c, the size of the second encoding feature is h / 2×w / 2×c / 2, and the size of the third encoding feature is h / 4×w / 4×c / 4. In FIG5(b), the short dashed line indicates convolution processing and downsampling, and the dotted dashed line indicates convolution processing and upsampling.

[0103] The method of obtaining the second fusion feature includes the following steps:

[0104] Step 1': input the first coding feature into the first third interaction module for processing to obtain the fourth excitation feature.

[0105] Step 2': input the second coding feature into the second third interaction module for processing to obtain the fifth excitation feature.

[0106] Step 3': input the third coding feature into the third third interaction module for processing to obtain the sixth excitation feature.

[0107] The implementation method of step 1' to step 3' is the same as the implementation method of the above steps 1 to 3, and reference may be made to the above description, which will not be repeated here.

[0108] 5( b ), the size of the fourth excitation feature is h×w×c, the size of the fifth excitation feature is h / 2×w / 2×2c, and the size of the sixth excitation feature is h / 4×w / 4×4c.

[0109] Step 4': upsample the fourth excitation feature to obtain a downsampled feature.

[0110] Referring to FIG. 5( b ), the fourth excitation feature may be convolved with a convolution kernel of 1×1 and downsampled to obtain a downsampled feature.

[0111] Step 5': downsample the sixth excitation feature to obtain an upsampled feature.

[0112] Referring to FIG. 5( b ), the sixth excitation feature may be subjected to convolution processing with a convolution kernel of 1×1 and upsampled to obtain an upsampled feature.

[0113] Step 6': concatenate the up-sampled feature, the fifth excitation feature and the down-sampled feature, and input the concatenated feature into the fourth interaction module to obtain the second interaction feature.

[0114] Referring to FIG. 5 (b), the up-sampled feature, the fifth excitation feature, and the down-sampled feature can be concatenated channel by channel, and the size of the concatenated feature is h / 2×w / 2×7c. The concatenated feature is then input into the first convolution layer with a convolution kernel of 3×3 included in the fourth interactive module, and the size of the output result of the convolution layer is h / 2×w / 2×2c. The output result of the convolution layer is then input into the SE block included in the fourth interactive module, and the output result of the SE block is input into the second convolution layer with a convolution kernel of 3×3 included in the fourth interactive module to obtain the second interactive feature output by the convolution layer.

[0115] Step 7': concatenate the second interactive feature with each coding feature to obtain a second fusion feature.

[0116] Referring to FIG. 5( b ), the second interaction feature can be added to each encoding feature channel by channel to obtain a second fused feature, and the size of the second fused feature is h / 2×w / 2×2c.

[0117] Through the above method, the embodiment of the present disclosure can obtain more comprehensive and richer fusion features by fusing coding features of different scales. Such fusion features help the subsequent decoder capture more image details and context information, thereby improving the accuracy of the subsequent generation of restored images.

[0118] Referring to FIG5(c), the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and one compression and excitation block between the two convolutional layers. In FIG5(c), the short dashed line indicates convolution processing and downsampling, and the dotted dashed line indicates convolution processing and upsampling.

[0119] The method of obtaining the third fusion feature includes the following steps:

[0120] Step 1'': input the first coding feature into the first fifth interaction module for processing to obtain the seventh excitation feature.

[0121] Step 2'': input the second coding feature into the second fifth interaction module for processing to obtain the eighth excitation feature.

[0122] Step 3'': input the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature.

[0123] The implementation method of step 1''~step 3'' is the same as the implementation method of the above steps 1~step 3, and can refer to the above description, which will not be repeated here.

[0124] 5( c ), the size of the seventh excitation feature is h×w×c, the size of the eighth excitation feature is h / 2×w / 2×2c, and the size of the ninth excitation feature is h / 4×w / 4×4c.

[0125] Step 4'': downsample the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features.

[0126] Referring to FIG5(c), the seventh excitation feature may be convolved with a convolution kernel of 1×1 and downsampled to obtain a downsampled feature. Also, the eighth excitation feature may be convolved with a convolution kernel of 1×1 and downsampled to obtain a downsampled feature.

[0127] Step 5'': concatenate the two downsampled features and the ninth excitation feature, and input the concatenated features into the sixth interaction module to obtain the third interaction feature.

[0128] Referring to FIG. 5 (c), the ninth excitation feature and the two downsampled features can be concatenated channel by channel, and the size of the concatenated features is h / 4×w / 4×7c. The concatenated features are then input into the first convolution layer with a convolution kernel of 3×3 included in the sixth interactive module, and the size of the output result of the convolution layer is h / 4×w / 4×4c. The output result of the convolution layer is then input into the SE block included in the sixth interactive module, and the output result of the SE block is input into the second convolution layer with a convolution kernel of 3×3 included in the sixth interactive module to obtain the third interactive feature output by the convolution layer.

[0129] Step 6'': concatenate the third interactive feature with each coding feature to obtain a third fusion feature.

[0130] Referring to FIG. 5( c ), the third interaction feature can be added to each encoding feature channel by channel to obtain a third fused feature, and the size of the first fused feature is h / 4×w / 4×4c.

[0131] Through the above method, the embodiment of the present disclosure can obtain more comprehensive and richer fusion features by fusing coding features of different scales. Such fusion features help the subsequent decoder capture more image details and context information, thereby improving the accuracy of the subsequent generation of restored images.

[0132] In some embodiments of the present disclosure, the decoder in the above S104 includes a first decoding stage, a second decoding stage and a third decoding stage.

[0133] On this basis, see Figure 6 The above S104 decodes the fusion features through a decoder to obtain a restored image after removing the moiré pattern from the processed image, including the following steps:

[0134] S601: Input the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature.

[0135] In the disclosed embodiment, each decoding stage includes a preset number of SAMs connected in series, and the preset number is greater than 2. For example, each decoding stage may include a DRDB and 4 SAMs connected in series, and in the first decoding stage, the first fused feature may be processed by the DRDB and the 4 SAMs in sequence to obtain the first decoding feature.

[0136] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré, by increasing the number of SAMs included in the decoding stage, the process of removing moiré from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of removing moiré.

[0137] S602: Decode the second fusion feature and the first decoding feature in a second decoding stage to obtain a second decoding feature.

[0138] The first decoded feature may be input into a bilinear upsampling layer for upsampling, and the upsampling result may be concatenated with the second fused feature, for example, the upsampling result may be added to the second fused feature channel by channel to obtain a concatenated feature. Alternatively, the second fused feature may be directly concatenated with the first decoded feature to obtain a concatenated feature.

[0139] Then the splicing feature is input into the second decoding stage for decoding to obtain the second decoding feature output by the second decoding stage. The second decoding stage includes a DRDB and a preset number of SAMs connected in series.

[0140] S603: Decode the third fusion feature and the second decoding feature through a third decoding stage to obtain a third decoding feature.

[0141] The second decoded feature can be input into the bilinear upsampling layer for upsampling, and the upsampling result can be spliced ​​with the third fusion feature channel by channel, and the splicing result can be input into the third decoding stage, and processed by the DRDB and the preset number of SAMs included in the third decoding stage to obtain the third decoded feature. The third decoding stage includes a DRDB and a preset number of SAMs connected in series.

[0142] Alternatively, the third decoding feature may be determined by the method described below.

[0143] S604: Obtain a restored image based on the third decoding feature.

[0144] The third decoded feature may be subjected to convolution processing and pixel shuffle upsample processing to obtain a restored image.

[0145] Through the above method, when performing feature decoding in the decoding stage in the embodiment of the present disclosure, it can be based not only on the processing results of the previous decoding stage, but also on the fused features obtained by fusing the encoding features of the three encoding stages, so that the decoding stage can have more features to refer to, and the understanding of the document image is deeper, thereby improving the moiré removal effect on the document image.

[0146] The method of obtaining the third decoding feature in S603 includes the following steps:

[0147] Step 1: Perform convolution processing on the second decoding feature to obtain a convolution feature.

[0148] The convolution process is performed by using a convolution layer with a convolution kernel of 1×1.

[0149] Step 2: Reconstruct the convolutional features through the reconstruction module to obtain the reconstructed features.

[0150] Step 3: Splice the third fusion feature and the reconstructed feature to obtain a spliced ​​feature.

[0151] The third fusion feature and the reconstruction feature may be added channel by channel to obtain a splicing feature.

[0152] Step 4: Input the concatenated features into the third decoding stage for decoding, and obtain the third decoding features output by the third decoding stage.

[0153] The disclosed embodiment adds convolution and feature reconstruction processing between the second decoding stage and the third decoding stage to obtain deeper features from the second decoding features, improve the feature extraction capability of the removal model, and ensure the effect of removing moiré patterns from document images.

[0154] In some embodiments of the present disclosure, the reconstruction module includes a plurality of multi-scale residual group modules (MSRGMs) connected in series. Figure 7 , Figure 7 The reconstruction module shown includes N residual group modules (RGMs). For example, the reconstruction module includes 3 MSRGMs connected in series.

[0155] The structure of each multi-scale residual group module is the same, and each multi-scale residual group module includes: a first residual group module, a second residual group module, a third residual group module and a convolution layer.

[0156] The processing of input features by each multi-scale residual group module includes the following steps:

[0157] Step (1), inputting the input feature into the first residual group module for feature extraction processing based on the channel attention mechanism to obtain the first branch feature.

[0158] The input features are input into the first RGM, and the first RGM is used to perform feature extraction processing on the input features based on the channel attention mechanism to obtain the first branch features.

[0159] When the MSRGM is the first MSRGM included in the reconstruction module, the input feature is the convolution feature; when the MSRGM is not the first MSRGM included in the reconstruction module, the input feature is the output result of the previous MSRGM.

[0160] Step (2), performing a first downsampling process on the input feature to obtain a first downsampling feature.

[0161] See also Figure 7 , DePixelShuffle for different scales is performed on the input features to achieve feature downsampling and obtain the first downsampling features.

[0162] Step (3), inputting the first down-sampled feature into the second residual group module for feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain the second branch feature.

[0163] See also Figure 7 , the first down-sampled feature is input into the second RGM, the second RGM is used to perform feature extraction based on the channel attention mechanism on the first down-sampled feature, and the feature extraction result is shuffled at different scales (PixelShuffle for different scales) to achieve feature upsampling and obtain the second branch feature.

[0164] The sampling multiples of the first downsampling process and the first upsampling process may be the same, for example, the first downsampling process is 2 times downsampling, and the first upsampling process is 2 times upsampling.

[0165] Step (4), performing a second downsampling process on the input features to obtain second downsampling features.

[0166] See also Figure 7, perform DePixelShuffle for different scales on the input features to achieve feature downsampling and obtain second downsampled features. The sampling multiples of the first downsampling process and the second downsampling process are different, so that the sizes of the first downsampled features and the second downsampled features are different. For example, the first downsampling process is 2 times downsampling, so that each dimension of the downsampled feature is 1 / 2 of the corresponding dimension of the feature before downsampling; the second downsampling process is 4 times downsampling, so that each dimension of the downsampled feature is 1 / 4 of the corresponding dimension of the feature before downsampling.

[0167] Step (5), input the second down-sampled features into the third residual group module for feature extraction based on the channel attention mechanism, and perform a second up-sampling process on the feature extraction results to obtain the third branch features.

[0168] See also Figure 7 , the second down-sampled features are input into the third RGM, the third RGM is used to perform feature extraction based on the channel attention mechanism on the second down-sampled features, and the feature extraction results are PixelShuffled fordifferent scale to achieve feature upsampling and obtain the third branch features.

[0169] The sampling multiples of the second down-sampling process and the second up-sampling process may be the same, for example, the second down-sampling process is 4-fold down-sampling, and the second up-sampling process is 4-fold up-sampling.

[0170] Step (6), concatenate the input features, the first branch features, the second branch features, and the third branch features, and use a convolution layer to perform convolution processing on the concatenated features to obtain a processing result.

[0171] See also Figure 7 , the input features, the first branch features, the second branch features and the third branch features are connected in series channel by channel, and the connected features are input into a convolution layer with a convolution kernel of 1×1, and the convolution layer is used to perform convolution processing on the connected features to obtain the processing result. When the MSRGM is not the last MSRGM included in the reconstruction module, the processing result will be used as the input feature of the next MSRGM; when the MSRGM is the last MSRGM included in the reconstruction module, the processing result will be used as the output result of the reconstruction module.

[0172] Since the residual group module can perform feature extraction based on the channel attention mechanism, the embodiment of the present disclosure can pay more attention to the feature areas that are conducive to moiré removal during the feature extraction process, thereby improving the accuracy of moiré removal. Moreover, before the second residual group module and the third residual group module perform feature extraction, the embodiment of the present disclosure performs different degrees of downsampling processing on the input features, so that the three residual group modules can extract features from input features of different scales, thereby coping with the problem of moiré removal of document images of various resolutions. Moreover, experiments have shown that the reconstruction module provided by the embodiment of the present disclosure has a better effect on removing noise such as reflections, low light and shadows in document images, improves the accuracy of the color and content of the restored image, and improves the image quality of the restored image.

[0173] The following describes the removal model structure and the conventional ESDNet structure provided by the embodiment of the present disclosure in combination with actual application scenarios:

[0174] The conventional ESDNet structure is as follows Figure 8 As shown, ESDNet includes: a downsampling layer, a first convolutional layer, an encoder, a decoder, a second convolutional layer, and a pixel shuffling upsampling layer.

[0175] The encoder and decoder both include three stages connected in series, and each stage includes a DRDB and a SAM connected in series. A bilinear upsampling layer is also included between every two decoding stages of the decoder.

[0176] There is also a skip connection between each encoding stage of the encoder and a decoding stage of the decoder. There is a skip connection between the first encoding stage and the third decoding stage, there is a skip connection between the second encoding stage and the second decoding stage, and there is a skip connection between the third encoding stage and the first decoding stage.

[0177] The removal model structure provided by the embodiment of the present disclosure is as follows Fig. 9 As shown, the removal model includes: a downsampling layer, a first convolutional layer, an encoder, a first multi-scale interaction module, a second multi-scale interaction module, a third multi-scale interaction module, a decoder, a second convolutional layer and a pixel shuffling upsampling layer.

[0178] The encoder and decoder both include three stages connected in series, and each stage includes a DRDB and four SAMs connected in series. Fig. 9 Only one SAM is shown in each stage, but each stage may actually include four SAMs connected in series.

[0179] The first multi-scale interaction module is respectively connected to the three encoding stages and the first decoding stage; the second multi-scale interaction module is respectively connected to the three encoding stages and the second decoding stage; the third multi-scale interaction module is respectively connected to the three encoding stages and the third decoding stage.

[0180] There is a convolution layer with a convolution kernel of 1×1 and a reconstruction module connected in series between the second decoding stage and the third decoding stage of the encoder.

[0181] By inputting the document image with moiré patterns into the removal model, a restored image after the moiré patterns are removed output by the removal model can be obtained.

[0182] By comparison, it can be seen that compared with the conventional ESDNet, in the removal model provided by the embodiment of the present disclosure, each stage of the encoder and decoder includes more SAMs, which can more effectively extract multi-scale features aligned at the same semantic level, while allocating attention to different features, thereby selectively fusing multi-scale features and improving the feature extraction capability of the removal model.

[0183] Moreover, the jump connection does not process the features extracted in the encoding stage, that is, it is directly input into a decoding stage, and the decoding stage can only receive the processing result of one encoding stage through a jump connection. The removal model of the embodiment of the present disclosure uses a multi-scale interaction module to fuse the processing results of the three encoding stages and input them into each decoding stage, that is, each decoding stage can obtain the fusion results of the three encoding stages, so that the features based on the decoding stage are richer, the accuracy of image restoration is improved, and the removal effect of moiré is improved.

[0184] In addition, the embodiment of the present disclosure also adds a reconstruction module between the second decoding stage and the third decoding stage, which can improve the processing effect of noise such as reflections, low light, shadows, etc. in the image, and further improve the image quality of the restored image.

[0185] Based on the same inventive concept, the embodiment of the present disclosure also provides a model training method, which is applied to an electronic device, wherein the electronic device to which the model training method is applied and the electronic device to which the above-mentioned image processing method is applied can be the same electronic device, or different electronic devices. Fig.10 As shown, the model training method provided by the embodiment of the present disclosure includes the following steps:

[0186] S1001. Obtain sample images and training labels of the sample images.

[0187] The sample image is a document image with moiré patterns, and the training label is a document image without moiré patterns.

[0188] For example, multiple images without moiré patterns may be obtained and used as training labels respectively, and moiré patterns may be added to these images respectively as sample images corresponding to the training labels.

[0189] Alternatively, multiple images with moiré patterns may be obtained as sample images, and images from which the moiré patterns have been removed manually or in other ways may be obtained as training labels for the sample images.

[0190] Alternatively, the sample images and their training labels may be obtained by other means, which are not specifically limited in the embodiments of the present disclosure.

[0191] In the traditional training process of ESDNet, sample images with fixed resolution are usually used. For example, images in the UHDM4k dataset are used as sample images. The resolution of images in the UHDM 4k dataset is 4k, that is, there are about 4000 pixels in the horizontal direction of the image. For example, the resolution of a 4k image is: 3840×2160 or 4096×2160.

[0192] In order to improve the moiré removal effect of the trained removal model on document images, sample images of various resolutions can be obtained, thereby improving the moiré removal effect of the removal model on document images of various resolutions. For example, the resolution of each sample image obtained in the embodiment of the present disclosure can cover a range from 200×200 to 4k.

[0193] S1002: Encode the sample image by removing the encoder included in the model to obtain encoding features output by multiple encoding stages of the encoder, wherein the sizes of the encoding features are different.

[0194] The specific implementation method of S1002 is the same as the above S102. Please refer to the relevant description of S102 and will not be repeated here.

[0195] S1003. The encoding features are fused by removing the multi-scale interaction module included in the model to obtain a fused feature.

[0196] The specific implementation method of S1003 is the same as the above S103. Please refer to the relevant description of S103 and will not be repeated here.

[0197] S1004. Decoding the fused features by removing the decoder included in the model to obtain decoding features output by multiple decoding stages of the decoder.

[0198] The structure and specific processing process of the decoder can refer to the relevant description in the above S104, which will not be repeated here.

[0199] S1005: Determine a loss value based on the decoded features and the training labels.

[0200] The decoded features outputted from each decoding stage can be processed using a convolution layer and a pixel shuffle upsampling layer to obtain a predicted image. The decoded features outputted from the third decoding stage are processed by the convolution layer and the pixel shuffle upsampling layer, and the obtained predicted image can also be called a restored image.

[0201] For example, the predicted image and training label can be substituted into the following loss function to calculate the loss value.

[0202]

[0203] in, represents the loss value, It means calculating the mean absolute error. Indicates the calculation of perceptual loss (Perceptual Loss), is the predicted image obtained based on the decoding features of the i-th decoding stage, for The corresponding label, The corresponding label can be set to the resolution of the training label The image obtained after the resolution is Indicates preset parameters, such as .

[0204] S1006. Use the loss value to adjust the network parameters of the removal model, return to S1002, and encode the sample image through the encoder included in the removal model until the removal model converges, and determine that the training is completed.

[0205] It can be determined whether the removal mode has converged. If not, the network parameters of the removal model are adjusted using the loss value through back propagation and gradient descent, and return to S1002 to continue training the removal model based on the next batch of sample images and their training labels. If converged, the training is determined to be complete. Among them, if the number of iterations reaches a preset number, it can be determined that it has converged; otherwise, it is determined that it has not converged. Alternatively, if the loss value calculated in this iteration is less than a preset threshold, it can be determined that it has converged; otherwise, it is determined that it has not converged. Alternatively, if the error between the loss values ​​calculated in the most recent N iterations is less than a preset error, it can be determined that it has converged; otherwise, it is determined that it has not converged. Alternatively, it can be determined that whether the removal model has converged can also be determined by other methods, and the embodiments of the present disclosure do not specifically limit this.

[0206] In the disclosed embodiment, the feature depths that can be extracted by the encoder of the removal model at different encoding stages are different, so the features extracted at different encoding stages are not exactly the same, and the features extracted at each encoding stage cover a wider range of features as a whole. Therefore, the disclosed embodiment fuses the encoding features through the multi-scale interaction module of the removal model, and decodes the fused features through the decoder of the removal model, so that when the decoder constructs the restored image, it can be based on the rich features extracted at each encoding stage, so that the understanding of the text image is deeper, so that the removal model can more accurately remove the moiré processing of the text image.

[0207] In some embodiments of the present disclosure, the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the above S1002 encodes the sample image by removing the encoder included in the model to obtain the encoding features output by the multiple encoding stages of the encoder, including the following steps:

[0208] Encoding the sample image through a first encoding stage to obtain a first encoding feature;

[0209] Inputting the first coding feature into the second coding stage for coding, and obtaining a second coding feature output by the second coding stage;

[0210] The second coding feature is input into the third coding stage for coding, so as to obtain a third coding feature outputted by the third coding stage.

[0211] Through the above method, the embodiment of the present disclosure can obtain the coding features output by the encoder in three different coding stages. The features included in these three coding features are not exactly the same, so these three coding features cover a wider range of features, making the subsequent moiré removal effect through these three features better.

[0212] In some embodiments of the present disclosure, each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0213] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré, by increasing the number of SAMs included in the encoding stage, the process of removing moiré from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of removing moiré.

[0214] In some embodiments of the present disclosure, the above S1003 removes the multi-scale interaction module included in the model to fuse the encoding features to obtain the fused features, including the following steps:

[0215] Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature;

[0216] Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature;

[0217] Each coded feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

[0218] Through the above method, the embodiment of the present disclosure can fuse various coding features through multiple multi-scale interaction modules and obtain three fused features of different sizes, so that the size of each fused feature meets the requirements of a decoding stage of the decoder for the size of the input feature.

[0219] In some embodiments of the present disclosure, the first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and one compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the above-mentioned method of inputting each coded feature into the first multi-scale interaction module to obtain the first fusion feature includes the following steps:

[0220] Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature;

[0221] Inputting the second coding feature into the second first interaction module for processing to obtain a second excitation feature;

[0222] Inputting the third coding feature into the third first interaction module for processing to obtain a third excitation feature;

[0223] Upsampling the second excitation feature and the third excitation feature respectively to obtain two upsampled features;

[0224] The first excitation feature and the two up-sampled features are concatenated, and the concatenated features are input into the second interaction module to obtain a first interaction feature;

[0225] The first interactive feature is concatenated with each coding feature to obtain a first fusion feature.

[0226] Through the above method, the embodiment of the present disclosure can obtain more comprehensive and richer fusion features by fusing coding features of different scales. Such fusion features help the subsequent decoder capture more image details and context information, thereby improving the accuracy of the subsequent generation of restored images.

[0227] In some embodiments of the present disclosure, the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the above-mentioned method of inputting each coded feature into the second multi-scale interaction module to obtain the second fusion feature includes the following steps:

[0228] Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature;

[0229] Inputting the second coding feature into the second third interaction module for processing to obtain a fifth excitation feature;

[0230] Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature;

[0231] Upsampling the fourth excitation feature to obtain a downsampled feature;

[0232] Downsampling the sixth excitation feature to obtain an upsampled feature;

[0233] The up-sampled feature, the fifth excitation feature and the down-sampled feature are concatenated, and the concatenated feature is input into the fourth interaction module to obtain a second interaction feature;

[0234] The second interactive feature is concatenated with each encoding feature to obtain a second fusion feature.

[0235] Through the above method, the embodiment of the present disclosure can obtain more comprehensive and richer fusion features by fusing coding features of different scales. Such fusion features help the subsequent decoder capture more image details and context information, thereby improving the accuracy of the subsequent generation of restored images.

[0236] In some embodiments of the present disclosure, the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the above-mentioned method of inputting each coded feature into the third multi-scale interaction module to obtain the third fusion feature includes the following steps:

[0237] Inputting the first coding feature into the first fifth interaction module for processing to obtain a seventh excitation feature;

[0238] Inputting the second coding feature into the second fifth interaction module for processing to obtain an eighth excitation feature;

[0239] Inputting the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature;

[0240] Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features;

[0241] The two down-sampled features and the ninth excitation feature are concatenated, and the concatenated features are input into the sixth interaction module to obtain a third interaction feature;

[0242] The third interactive feature is concatenated with each coding feature to obtain a third fusion feature.

[0243] Through the above method, the embodiment of the present disclosure can obtain more comprehensive and richer fusion features by fusing coding features of different scales. Such fusion features help the subsequent decoder capture more image details and context information, thereby improving the accuracy of the subsequent generation of restored images.

[0244] In some embodiments of the present disclosure, the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; the above S1004 decodes the fusion features by removing the decoder included in the model to obtain the decoding features output by the multiple decoding stages of the decoder, including the following steps:

[0245] Inputting the first fused feature into the first decoding stage for decoding to obtain a first decoding feature;

[0246] The second fusion feature and the first decoding feature are decoded in a second decoding stage to obtain a second decoding feature;

[0247] The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

[0248] Through the above method, when performing feature decoding in the decoding stage in the embodiment of the present disclosure, it can be based not only on the processing results of the previous decoding stage, but also on the fused features obtained by fusing the encoding features of the three encoding stages, so that the decoding stage can have more features to refer to, and the understanding of the document image is deeper, thereby improving the moiré removal effect on the document image.

[0249] In some embodiments of the present disclosure, the method of decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain the third decoding feature includes the following steps:

[0250] Performing convolution processing on the second decoding feature to obtain a convolution feature;

[0251] The convolution feature is reconstructed through the reconstruction module to obtain the reconstructed feature;

[0252] Splicing the third fused feature and the reconstructed feature to obtain a spliced ​​feature;

[0253] The concatenated features are input into the third decoding stage for decoding, so as to obtain the third decoding features output by the third decoding stage.

[0254] The disclosed embodiment adds convolution and feature reconstruction processing between the second decoding stage and the third decoding stage to obtain deeper features from the second decoding features, improve the feature extraction capability of the removal model, and ensure the effect of removing moiré patterns from document images.

[0255] In some embodiments of the present disclosure, each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0256] Since SAM can focus on the importance of different spatial positions in the input features for removing moiré, by increasing the number of SAMs included in the decoding stage, the process of removing moiré from document images can increase the attention paid to important areas and reduce the attention paid to unimportant areas, thereby improving the accuracy of removing moiré.

[0257] In some embodiments of the present disclosure, the reconstruction module includes a plurality of multi-scale residual group modules connected in series, and the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolution layer. The processing process of the multi-scale residual group module on the input feature includes the following steps:

[0258] Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features;

[0259] Performing a first downsampling process on the input feature to obtain a first downsampling feature;

[0260] Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature;

[0261] Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes;

[0262] The second down-sampled feature is input into the third residual group module for feature extraction based on the channel attention mechanism, and the feature extraction result is subjected to a second up-sampling process to obtain a third branch feature;

[0263] The input features, the first branch features, the second branch features and the third branch features are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain the processing results.

[0264] Since the residual group module can perform feature extraction based on the channel attention mechanism, the embodiment of the present disclosure can pay more attention to the feature areas that are conducive to moiré removal during the feature extraction process, thereby improving the accuracy of moiré removal. Moreover, before the second residual group module and the third residual group module perform feature extraction, the embodiment of the present disclosure performs different degrees of downsampling processing on the input features, so that the three residual group modules can extract features from input features of different scales, thereby coping with the problem of moiré removal of document images of various resolutions. Moreover, experiments have shown that the reconstruction module provided by the embodiment of the present disclosure has a better effect on removing noise such as reflections, low light and shadows in document images, improves the accuracy of the color and content of the restored image, and improves the image quality of the restored image.

[0265] In some embodiments of the present disclosure, the method of encoding the sample image in the first encoding stage to obtain the first encoding feature includes the following steps:

[0266] Downsample and convolve the sample image to obtain preprocessing features;

[0267] The preprocessed features are input into the first encoding stage for encoding to obtain first encoded features.

[0268] Since ESDNet needs to undergo downsampling and convolution processing before encoding the image through the encoder, the embodiment of the present disclosure first downsamples and convolutions the image to be processed, and then processes it through the encoder, thereby reducing the damage to the original structure of ESDNet, ensuring the smoothness of the image processing process, and improving the effect of removing moiré on document images.

[0269] In the model training method, the specific implementation method of each step can refer to the description of the relevant steps of the above-mentioned image processing method, which will not be repeated here.

[0270] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of document images involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0271] It should be noted that the document image in this embodiment is not a document image for a specific user and cannot reflect the personal information of a specific user.

[0272] It should be noted that the document image in this embodiment may come from a public dataset.

[0273] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure also provides an image processing device, such as Fig.11As shown, it includes: an acquisition module 1101, an encoding module 1102, a fusion module 1103 and a decoding module 1104;

[0274] An acquisition module 1101 is used to acquire an image to be processed, where the image to be processed is a document image with moiré patterns;

[0275] An encoding module 1102 is used to encode the image to be processed through an encoder to obtain encoding features outputted by multiple encoding stages of the encoder;

[0276] A fusion module 1103 is used to fuse the coding features through a multi-scale interaction module to obtain a fusion feature;

[0277] The decoding module 1104 is used to decode the fused features through a decoder to obtain a restored image after the moiré pattern is removed from the image to be processed.

[0278] In some embodiments of the present disclosure, the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the encoding module 1102 is specifically configured to:

[0279] The image to be processed is encoded through the first encoding stage to obtain a first encoding feature;

[0280] Inputting the first coding feature into the second coding stage for coding, and obtaining a second coding feature output by the second coding stage;

[0281] The second coding feature is input into the third coding stage for coding, so as to obtain a third coding feature outputted by the third coding stage.

[0282] In some embodiments of the present disclosure, each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0283] In some embodiments of the present disclosure, the fusion module 1103 is specifically configured to:

[0284] Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature;

[0285] Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature;

[0286] Each coded feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

[0287] In some embodiments of the present disclosure, the first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and one compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1103 is specifically used to:

[0288] Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature;

[0289] Inputting the second coding feature into the second first interaction module for processing to obtain a second excitation feature;

[0290] Inputting the third coding feature into the third first interaction module for processing to obtain a third excitation feature;

[0291] Upsampling the second excitation feature and the third excitation feature respectively to obtain two upsampled features;

[0292] The first excitation feature and the two up-sampled features are concatenated, and the concatenated features are input into the second interaction module to obtain a first interaction feature;

[0293] The first interactive feature is concatenated with each coding feature to obtain a first fusion feature.

[0294] In some embodiments of the present disclosure, the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1103 is specifically used to:

[0295] Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature;

[0296] Inputting the second coding feature into the second third interaction module for processing to obtain a fifth excitation feature;

[0297] Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature;

[0298] Upsampling the fourth excitation feature to obtain a downsampled feature;

[0299] Downsampling the sixth excitation feature to obtain an upsampled feature;

[0300] The up-sampled feature, the fifth excitation feature and the down-sampled feature are concatenated, and the concatenated feature is input into the fourth interaction module to obtain a second interaction feature;

[0301] The second interactive feature is concatenated with each coding feature to obtain a second fusion feature.

[0302] In some embodiments of the present disclosure, the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1103 is specifically used to:

[0303] Inputting the first coding feature into the first fifth interaction module for processing to obtain a seventh excitation feature;

[0304] Inputting the second coding feature into the second fifth interaction module for processing to obtain an eighth excitation feature;

[0305] Inputting the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature;

[0306] Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features;

[0307] The two down-sampled features and the ninth excitation feature are concatenated, and the concatenated features are input into the sixth interaction module to obtain a third interaction feature;

[0308] The third interactive feature is concatenated with each coding feature to obtain a third fusion feature.

[0309] In some embodiments of the present disclosure, the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; the decoding module 1104 is specifically used to:

[0310] Inputting the first fused feature into the first decoding stage for decoding to obtain a first decoding feature;

[0311] The second fusion feature and the first decoding feature are decoded in a second decoding stage to obtain a second decoding feature;

[0312] The third fusion feature and the second decoding feature are decoded in a third decoding stage to obtain a third decoding feature;

[0313] Based on the third decoding feature, a restored image is obtained.

[0314] In some embodiments of the present disclosure, the decoding module 1104 is specifically configured to:

[0315] Performing convolution processing on the second decoding feature to obtain a convolution feature;

[0316] The convolution feature is reconstructed through the reconstruction module to obtain the reconstructed feature;

[0317] Splicing the third fused feature and the reconstructed feature to obtain a spliced ​​feature;

[0318] The concatenated features are input into the third decoding stage for decoding, so as to obtain the third decoding features output by the third decoding stage.

[0319] In some embodiments of the present disclosure, each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0320] In some embodiments of the present disclosure, the reconstruction module includes a plurality of multi-scale residual group modules connected in series, the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolution layer; the processing process of the multi-scale residual group module on the input feature includes the following steps:

[0321] Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features;

[0322] Performing a first downsampling process on the input feature to obtain a first downsampling feature;

[0323] Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature;

[0324] Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes;

[0325] The second down-sampled feature is input into the third residual group module for feature extraction based on the channel attention mechanism, and the feature extraction result is subjected to a second up-sampling process to obtain a third branch feature;

[0326] The input features, the first branch features, the second branch features and the third branch features are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain the processing results.

[0327] In some embodiments of the present disclosure, the encoding module 1102 is specifically configured to:

[0328] Down-sample and convolve the image to be processed to obtain pre-processed features;

[0329] The preprocessed features are input into the first encoding stage for encoding to obtain first encoded features.

[0330] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure also provides a model training device, such as Fig.12As shown, the device includes: an acquisition module 1201, an encoding module 1202, a fusion module 1203, a decoding module 1204, a determination module 1205 and an adjustment module 1206;

[0331] An acquisition module 1201 is used to acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is a document image without moiré patterns;

[0332] An encoding module 1202, configured to encode the sample image by removing the encoder included in the model, and obtain encoding features respectively output by multiple encoding stages of the encoder;

[0333] A fusion module 1203 is used to fuse the encoding features by removing the multi-scale interaction module included in the model to obtain a fusion feature;

[0334] A decoding module 1204 is used to decode the fused features by removing the decoder included in the model to obtain decoding features outputted by multiple decoding stages of the decoder respectively;

[0335] A determination module 1205, for determining a loss value based on the decoded features and the training labels;

[0336] The adjustment module 1206 is used to adjust the network parameters of the removal model using the loss value, and return to the step of encoding the sample image through the encoder included in the removal model until the removal model converges, and determines that the training is completed.

[0337] In some embodiments of the present disclosure, the encoder includes a first encoding stage, a second encoding stage, and a third encoding stage; the encoding module 1202 is specifically used to:

[0338] Encoding the sample image through a first encoding stage to obtain a first encoding feature;

[0339] Inputting the first coding feature into the second coding stage for coding, and obtaining a second coding feature output by the second coding stage;

[0340] The second coding feature is input into the third coding stage for coding, so as to obtain a third coding feature outputted by the third coding stage.

[0341] In some embodiments of the present disclosure, each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0342] In some embodiments of the present disclosure, the fusion module 1203 is specifically used to:

[0343] Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature;

[0344] Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature;

[0345] Each coded feature is input into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the first fused feature, the second fused feature and the third fused feature have different sizes.

[0346] In some embodiments of the present disclosure, the first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and one compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1203 is specifically used to:

[0347] Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature;

[0348] Inputting the second coding feature into the second first interaction module for processing to obtain a second excitation feature;

[0349] Inputting the third coding feature into the third first interaction module for processing to obtain a third excitation feature;

[0350] Upsampling the second excitation feature and the third excitation feature respectively to obtain two upsampled features;

[0351] The first excitation feature and the two up-sampled features are concatenated, and the concatenated features are input into the second interaction module to obtain a first interaction feature;

[0352] The first interactive feature is concatenated with each coding feature to obtain a first fusion feature.

[0353] In some embodiments of the present disclosure, the second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1203 is specifically used to:

[0354] Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature;

[0355] Inputting the second coding feature into the second third interaction module for processing to obtain a fifth excitation feature;

[0356] Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature;

[0357] Upsampling the fourth excitation feature to obtain a downsampled feature;

[0358] Downsampling the sixth excitation feature to obtain an upsampled feature;

[0359] The up-sampled feature, the fifth excitation feature and the down-sampled feature are concatenated, and the concatenated feature is input into the fourth interaction module to obtain a second interaction feature;

[0360] The second interactive feature is concatenated with each coding feature to obtain a second fusion feature.

[0361] In some embodiments of the present disclosure, the third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module 1203 is specifically used to:

[0362] Inputting the first coding feature into the first fifth interaction module for processing to obtain a seventh excitation feature;

[0363] Inputting the second coding feature into the second fifth interaction module for processing to obtain an eighth excitation feature;

[0364] Inputting the third coding feature into the third fifth interaction module for processing to obtain a ninth excitation feature;

[0365] Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features;

[0366] The two down-sampled features and the ninth excitation feature are concatenated, and the concatenated features are input into the sixth interaction module to obtain a third interaction feature;

[0367] The third interactive feature is concatenated with each coding feature to obtain a third fusion feature.

[0368] In some embodiments of the present disclosure, the decoder includes a first decoding stage, a second decoding stage, and a third decoding stage; the decoding module 1204 is specifically used to:

[0369] Inputting the first fused feature into the first decoding stage for decoding to obtain a first decoding feature;

[0370] The second fusion feature and the first decoding feature are decoded in a second decoding stage to obtain a second decoding feature;

[0371] The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

[0372] In some embodiments of the present disclosure, the decoding module 1204 is specifically configured to:

[0373] Performing convolution processing on the second decoding feature to obtain a convolution feature;

[0374] The convolution feature is reconstructed through the reconstruction module to obtain the reconstructed feature;

[0375] Splicing the third fused feature and the reconstructed feature to obtain a spliced ​​feature;

[0376] The concatenated features are input into the third decoding stage for decoding, so as to obtain the third decoding features output by the third decoding stage.

[0377] In some embodiments of the present disclosure, each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

[0378] In some embodiments of the present disclosure, the reconstruction module includes a plurality of multi-scale residual group modules connected in series, the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the processing process of the multi-scale residual group module on the input feature includes the following steps:

[0379] Input the input features into the first residual group module for feature extraction based on the channel attention mechanism to obtain the first branch features;

[0380] Performing a first downsampling process on the input feature to obtain a first downsampling feature;

[0381] Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature;

[0382] Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes;

[0383] The second down-sampled feature is input into the third residual group module for feature extraction based on the channel attention mechanism, and the feature extraction result is subjected to a second up-sampling process to obtain a third branch feature;

[0384] The input features, the first branch features, the second branch features and the third branch features are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain the processing results.

[0385] In some embodiments of the present disclosure, the encoding module 1202 is specifically configured to:

[0386] Downsample and convolve the sample image to obtain preprocessing features;

[0387] The preprocessed features are input into the first encoding stage for encoding to obtain first encoded features.

[0388] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0389] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0390] like Fig.13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0391] Multiple components in the electronic device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the electronic device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0392] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as image processing methods and model training methods. For example, in some embodiments, the image processing method and the model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the image processing method and the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the image processing method and the model training method in any other appropriate manner (for example, by means of firmware).

[0393] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0394] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0395] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0396] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0397] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0398] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0399] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0400] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Acquire an image to be processed, where the image to be processed is a document image with moiré patterns; Encoding the image to be processed by an encoder to obtain encoding features outputted by multiple encoding stages of the encoder respectively; Each encoding feature is fused through a multi-scale interaction module to obtain a fused feature; The fusion feature is decoded by a decoder to obtain a restored image after the moiré pattern is removed from the image to be processed.

2. The method according to claim 1, wherein: The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; the encoding of the image to be processed by the encoder to obtain encoding features output by the multiple encoding stages of the encoder respectively includes: Encoding the image to be processed through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for coding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

3. The method according to claim 2, wherein: Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

4. The method according to claim 2 or 3, wherein: The fusion features are obtained by fusing the encoding features through the multi-scale interaction module, including: Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature; Input each coded feature into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the sizes of the first fused feature, the second fused feature and the third fused feature are different.

5. The method according to claim 4, wherein: The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the first multi-scale interaction module for fusion to obtain the first fused feature includes: Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature; Inputting the second coding feature into a second first interaction module for processing to obtain a second excitation feature; Inputting the third coding feature into a third first interaction module for processing to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interactive feature is concatenated with each encoding feature to obtain the first fusion feature.

6. The method according to claim 4, wherein: The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the second multi-scale interaction module for fusion to obtain the second fused feature includes: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interactive feature is concatenated with each encoding feature to obtain the second fusion feature.

7. The method according to claim 4, wherein: The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the third multi-scale interaction module for fusion to obtain the third fused feature includes: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two down-sampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interactive feature is concatenated with each coding feature to obtain the third fusion feature.

8. The method according to any one of claims 4 to 7, wherein: The decoder includes a first decoding stage, a second decoding stage and a third decoding stage; the decoding of the fusion feature by the decoder to obtain a restored image after removing the moiré pattern from the image to be processed includes: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; Decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature; Based on the third decoding feature, the restored image is obtained.

9. The method according to claim 8, wherein: The step of decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature includes: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstruction feature to obtain the splicing feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

10. The method according to claim 8 or 9, wherein: Each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

11. The method according to claim 9, wherein: The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the processing process of the multi-scale residual group module on the input feature includes the following steps: Inputting the input feature into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain a first branch feature; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature; Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled feature into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction result to obtain a third branch feature; The input feature, the first branch feature, the second branch feature and the third branch feature are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain a processing result.

12. The method according to any one of claims 2 to 11, wherein: The step of encoding the image to be processed in the first encoding stage to obtain a first encoding feature includes: Performing down sampling and convolution processing on the image to be processed to obtain preprocessing features; The preprocessing feature is input into the first encoding stage for encoding to obtain the first encoding feature.

13. A model training method, comprising: Acquire a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns; Encoding the sample image by removing the encoder included in the model to obtain encoding features outputted by multiple encoding stages of the encoder respectively; By removing the multi-scale interaction module included in the model, each encoding feature is fused to obtain a fused feature; Decoding the fused features by removing the decoder included in the model to obtain decoding features respectively output by multiple decoding stages of the decoder; Determining a loss value based on the decoded features and the training labels; The network parameters of the removal model are adjusted using the loss value, and the step of encoding the sample image by the encoder included in the removal model is returned to until the removal model converges, and the training is determined to be completed.

14. The method according to claim 13, wherein: The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; The step of encoding the sample image by removing the encoder included in the model to obtain encoding features respectively output by multiple encoding stages of the encoder includes: Encoding the sample image through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for coding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

15. The method according to claim 14, wherein: Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

16. The method according to claim 14 or 15, wherein: The multi-scale interaction module included in the removal model is removed to fuse the encoding features to obtain the fused features, including: Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature; Input each coded feature into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the sizes of the first fused feature, the second fused feature and the third fused feature are different.

17. The method according to claim 16, wherein: The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and a compression and excitation block, and the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the first multi-scale interaction module for fusion to obtain the first fused feature includes: Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature; Inputting the second coding feature into a second first interaction module for processing to obtain a second excitation feature; Inputting the third coding feature into a third first interaction module for processing to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interactive feature is concatenated with each encoding feature to obtain the first fusion feature.

18. The method according to claim 16, wherein: The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the second multi-scale interaction module for fusion to obtain the second fused feature includes: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interactive feature is concatenated with each encoding feature to obtain the second fusion feature.

19. The method according to claim 16, wherein: The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the inputting each coded feature into the third multi-scale interaction module for fusion to obtain the third fused feature includes: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two down-sampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interactive feature is concatenated with each coding feature to obtain the third fusion feature.

20. The method according to any one of claims 16 to 19, wherein: The decoder comprises a first decoding stage, a second decoding stage and a third decoding stage; The step of decoding the fused features by removing the decoder included in the model to obtain decoding features respectively output by multiple decoding stages of the decoder includes: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

21. The method according to claim 20, wherein: The step of decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature includes: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstruction feature to obtain the splicing feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

22. The method according to claim 20 or 21, wherein: Each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

23. The method according to claim 21, wherein: The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the processing process of the multi-scale residual group module on the input feature includes the following steps: Inputting the input feature into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain a first branch feature; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature; Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled feature into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction result to obtain a third branch feature; The input feature, the first branch feature, the second branch feature and the third branch feature are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain a processing result.

24. The method according to any one of claims 14 to 23, wherein: The step of encoding the sample image in the first encoding stage to obtain a first encoding feature includes: Performing downsampling and convolution processing on the sample image to obtain preprocessing features; The preprocessing feature is input into the first encoding stage for encoding to obtain the first encoding feature.

25. An image processing device, comprising: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed is a document image with moiré patterns; An encoding module, used for encoding the image to be processed by an encoder to obtain encoding features outputted by multiple encoding stages of the encoder respectively; A fusion module is used to fuse the encoding features through a multi-scale interaction module to obtain a fusion feature; The decoding module is used to decode the fusion feature through a decoder to obtain a restored image after the moiré pattern is removed from the image to be processed.

26. The device according to claim 25, wherein The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; the encoding module is specifically used for: Encoding the image to be processed through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for coding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

27. The device according to claim 26, wherein: Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

28. The device according to claim 26 or 27, wherein The fusion module is specifically used for: Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature; Input each coded feature into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the sizes of the first fused feature, the second fused feature and the third fused feature are different.

29. The device according to claim 28, wherein The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and one compression and excitation block, the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature; Inputting the second coding feature into a second first interaction module for processing to obtain a second excitation feature; Inputting the third coding feature into a third first interaction module for processing to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interactive feature is concatenated with each encoding feature to obtain the first fusion feature.

30. The device according to claim 28, wherein The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interactive feature is concatenated with each encoding feature to obtain the second fusion feature.

31. The apparatus of claim 28, wherein: The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two down-sampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interactive feature is concatenated with each coding feature to obtain the third fusion feature.

32. The device according to any one of claims 28 to 31, wherein: The decoder includes a first decoding stage, a second decoding stage and a third decoding stage; the decoding module is specifically used for: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; Decoding the third fusion feature and the second decoding feature through the third decoding stage to obtain a third decoding feature; Based on the third decoding feature, the restored image is obtained.

33. The device according to claim 32, wherein: The decoding module is specifically used for: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstruction feature to obtain the splicing feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

34. The device according to claim 32 or 33, wherein: Each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

35. The device according to claim 33, wherein The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the processing process of the multi-scale residual group module on the input feature includes the following steps: Inputting the input feature into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain a first branch feature; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature; Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled feature into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction result to obtain a third branch feature; The input feature, the first branch feature, the second branch feature and the third branch feature are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain a processing result.

36. The device according to any one of claims 25 to 33, wherein: The encoding module is specifically used for: Performing down sampling and convolution processing on the image to be processed to obtain preprocessing features; The preprocessing feature is input into the first encoding stage for encoding to obtain the first encoding feature.

37. A model training device, comprising: An acquisition module, used for acquiring a sample image and a training label of the sample image, wherein the sample image is a document image with moiré patterns, and the training label is the document image without moiré patterns; An encoding module, used for encoding the sample image by removing the encoder included in the model, and obtaining encoding features respectively output by multiple encoding stages of the encoder; A fusion module, used to fuse the encoding features by removing the multi-scale interaction module included in the model to obtain a fusion feature; A decoding module, used for decoding the fused features by removing the decoder included in the model, and obtaining decoding features respectively output by multiple decoding stages of the decoder; A determination module, configured to determine a loss value based on the decoded features and the training labels; An adjustment module is used to adjust the network parameters of the removal model using the loss value, return to the step of encoding the sample image through the encoder included in the removal model, and determine that the training is completed when the removal model converges.

38. The device according to claim 37, wherein The encoder includes a first encoding stage, a second encoding stage and a third encoding stage; the encoding module is specifically used for: Encoding the sample image through the first encoding stage to obtain a first encoding feature; Inputting the first coding feature into the second coding stage for coding, thereby obtaining a second coding feature output by the second coding stage; The second coding feature is input into the third coding stage for coding to obtain a third coding feature output by the third coding stage.

39. The device according to claim 38, wherein Each encoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

40. The device according to claim 38 or 39, wherein: The fusion module is specifically used for: Inputting each encoding feature into a first multi-scale interaction module for fusion to obtain a first fusion feature; Inputting each encoded feature into a second multi-scale interaction module for fusion to obtain a second fused feature; Input each coded feature into a third multi-scale interaction module for fusion to obtain a third fused feature; wherein the sizes of the first fused feature, the second fused feature and the third fused feature are different.

41. The device according to claim 40, wherein The first multi-scale interaction module includes three first interaction modules and one second interaction module, the first interaction module includes two residual blocks and one compression and excitation block, the second interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first first interaction module for processing to obtain a first excitation feature; Inputting the second coding feature into a second first interaction module for processing to obtain a second excitation feature; Inputting the third coding feature into a third first interaction module for processing to obtain a third excitation feature; performing upsampling processing on the second excitation feature and the third excitation feature respectively to obtain two upsampled features; splicing the first excitation feature and the two up-sampled features, and inputting the spliced ​​features into the second interaction module to obtain a first interaction feature; The first interactive feature is concatenated with each encoding feature to obtain the first fusion feature.

42. The device according to claim 40, wherein: The second multi-scale interaction module includes three third interaction modules and one fourth interaction module, the third interaction module includes two residual blocks and one compression and excitation block, and the fourth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first third interaction module for processing to obtain a fourth excitation feature; Inputting the second coding feature into a second third interaction module for processing to obtain a fifth excitation feature; Inputting the third coding feature into a third third interaction module for processing to obtain a sixth excitation feature; Upsampling the fourth excitation feature to obtain a downsampled feature; Downsampling the sixth excitation feature to obtain an upsampled feature; splicing the up-sampled feature, the fifth excitation feature and the down-sampled feature, and inputting the spliced ​​feature into the fourth interaction module to obtain a second interaction feature; The second interactive feature is concatenated with each encoding feature to obtain the second fusion feature.

43. The device according to claim 40, wherein: The third multi-scale interaction module includes three fifth interaction modules and one sixth interaction module, the fifth interaction module includes two residual blocks and one compression and excitation block, and the sixth interaction module includes two convolutional layers and a compression and excitation block between the two convolutional layers; the fusion module is specifically used to: Inputting the first coding feature into a first fifth interaction module for processing to obtain a seventh excitation feature; Inputting the second coding feature into a second fifth interaction module for processing to obtain an eighth excitation feature; Inputting the third coding feature into a third fifth interaction module for processing to obtain a ninth excitation feature; Downsampling the seventh excitation feature and the eighth excitation feature respectively to obtain two downsampled features; splicing the two down-sampled features and the ninth excitation feature, and inputting the spliced ​​features into the sixth interaction module to obtain a third interaction feature; The third interactive feature is concatenated with each coding feature to obtain the third fusion feature.

44. The device according to any one of claims 40 to 43, wherein: The decoder includes a first decoding stage, a second decoding stage and a third decoding stage; the decoding module is specifically used for: Inputting the first fusion feature into the first decoding stage for decoding to obtain a first decoding feature; Decoding the second fusion feature and the first decoding feature in the second decoding stage to obtain a second decoding feature; The third fusion feature and the second decoding feature are decoded in the third decoding stage to obtain a third decoding feature.

45. The device according to claim 44, wherein The decoding module is specifically used for: Performing convolution processing on the second decoding feature to obtain a convolution feature; Reconstructing the convolution feature through a reconstruction module to obtain a reconstructed feature; Splicing the third fusion feature and the reconstruction feature to obtain the splicing feature; The splicing feature is input into the third decoding stage for decoding to obtain a third decoding feature output by the third decoding stage.

46. ​​The device according to claim 44 or 45, wherein: Each decoding stage includes a preset number of spatial attention modules connected in series, and the preset number is greater than 2.

47. The device according to claim 46, wherein The reconstruction module includes a plurality of multi-scale residual group modules connected in series, wherein the multi-scale residual group module includes a first residual group module, a second residual group module, a third residual group module and a convolutional layer; the processing process of the multi-scale residual group module on the input feature includes the following steps: Inputting the input feature into the first residual group module to perform feature extraction processing based on the channel attention mechanism to obtain a first branch feature; Performing a first downsampling process on the input feature to obtain a first downsampling feature; Inputting the first down-sampled feature into the second residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a first up-sampling processing on the feature extraction result to obtain a second branch feature; Performing a second downsampling process on the input feature to obtain a second downsampling feature, wherein the first downsampling feature and the second downsampling feature have different sizes; Inputting the second down-sampled feature into the third residual group module to perform feature extraction processing based on the channel attention mechanism, and performing a second up-sampling processing on the feature extraction result to obtain a third branch feature; The input feature, the first branch feature, the second branch feature and the third branch feature are concatenated, and the convolution layer is used to perform convolution processing on the concatenated features to obtain a processing result.

48. The device according to any one of claims 38 to 47, wherein: The encoding module is specifically used for: Performing downsampling and convolution processing on the sample image to obtain preprocessing features; The preprocessing feature is input into the first encoding stage for encoding to obtain the first encoding feature.

49. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 24.

50. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-24.

51. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 24.

Images