Image Processing Method, Electronic Device, and Storage Medium

Abstract

Embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium. The method includes: obtaining an image to be processed; using an image restoration network to restore the image to be processed to obtain a restored image. The image restoration network includes: an encoder configured to perform convolutional processing on the image to be processed to obtain encoded features of the image to be processed; at least one long-distance information capture module, where the long-distance information capture module is configured to: perform self-attention operations within a first window in a first feature map to obtain a first feature; perform self-attention operations within a second window in a second feature map to obtain a second feature; generate an output result of the long-distance information capture module based on the first feature and the second feature; a decoder configured to decode the output result of the last long-distance information capture module to obtain decoded features of the image to be processed, and based on the decoded features, obtain the restored image.

Description

Technical Field

[0001] This application relates to the field of neural networks, and specifically to an image processing method, an electronic device, and a storage medium. Background Art

[0002] Currently, using a convolutional neural network to repair a to-be-processed image, such as image super-resolution, is a widely used image repair method. When using a convolutional neural network to repair a to-be-processed image, only features representing information of a single type, that is, features of pixel information suitable for being captured by convolution in the to-be-processed image, are used to obtain the repaired image. The repair effect of the repaired image needs to be improved. Summary of the Invention

[0003] Embodiments of this application provide an image processing method, a device, an electronic device, and a storage medium.

[0004] Embodiments of this application provide an image processing method, including:

[0005] Obtain a to-be-processed image;

[0006] Use an image repair network to repair the to-be-processed image to obtain a repaired image, where the image repair network includes:

[0007] An encoder, configured to perform convolutional processing on the to-be-processed image to obtain encoded features of the to-be-processed image;

[0008] At least one long-distance information capture module, where the long-distance information capture module is configured to: divide the input data of the long-distance information capture module into a first feature map sequence and a second feature map sequence of the long-distance information capture module, where the input data of the first long-distance information capture module is the encoded features, and the input data of other long-distance information capture modules except the first long-distance information capture module is the output result of the previous long-distance information capture module of the other long-distance information capture module; perform self-attention operations within each first window in the first feature maps in the first feature map sequence to obtain corresponding first features of the long-distance information capture module; perform self-attention operations within each second window in the second feature maps in the second feature map sequence to obtain corresponding second features of the long-distance information capture module; generate the output result of the long-distance information capture module based on the first features and the second features;

[0009] A decoder, configured to decode the output result of the last long-distance information capture module to obtain decoded features of the to-be-processed image, and obtain the repaired image based on the decoded features.

[0010] An embodiment of the present application provides an image processing apparatus, including:

[0011] An acquisition unit configured to acquire an image to be processed;

[0012] A processing unit configured to use an image inpainting network to inpaint the image to be processed to obtain an inpainted image. The image inpainting network includes: an encoder configured to perform convolutional processing on the image to be processed to obtain encoded features of the image to be processed; at least one long-distance information capture module. The long-distance information capture module is configured to: divide the input data of the long-distance information capture module into a first feature map sequence and a second feature map sequence of the long-distance information capture module. Among them, the input data of the first long-distance information capture module is the encoded features, and the input data of other long-distance information capture modules except the first long-distance information capture module is the output result of the previous long-distance information capture module of the other long-distance information capture modules; perform self-attention operations within each first window of the first feature maps in the first feature map sequence to obtain first features corresponding to the long-distance information capture module; perform self-attention operations within each second window of the second feature maps in the second feature map sequence to obtain second features corresponding to the long-distance information capture module; generate an output result of the long-distance information capture module based on the first features and the second features; a decoder configured to decode the output result of the last long-distance information capture module to obtain decoded features of the image to be processed, and based on the decoded features, obtain the inpainted image.

[0013] An embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the above image processing method.

[0014] An embodiment of the present application provides a computer-readable storage medium, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, the above image processing method is implemented.

[0015] An embodiment of the present application provides a computer program product, including computer program / instructions. When the computer program / instructions are executed by a processor, the above image processing method is implemented.

[0016] The image processing method provided by the embodiment of the present application combines convolutional processing with self-attention operations suitable for extracting long-range information. The corresponding first feature and the corresponding second feature obtained through the long-range information capture module can both represent the corresponding pixel information captured through convolution and the corresponding long-range information. Compared with obtaining the restored image using a feature representing a single type of information, that is, the feature of the pixel information suitable for being captured through convolution of the image to be processed, the types of information represented by the features participating in obtaining the restored image are richer. Therefore, the restoration effect of the restored image is improved. Description of the Drawings

[0017] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0018] Figure 1 It shows a flowchart of the image processing method provided by the embodiment of the present application;

[0019] Figure 2 It shows a schematic flowchart of using an image restoration network to restore the image to be processed;

[0020] Figure 3 It shows a schematic diagram of the effect of the window in the feature map;

[0021] Figure 4 It shows a schematic diagram of the structure of the residual module;

[0022] Figure 5 It shows a block diagram of the structure of the image processing device provided by the embodiment of the present application. Detailed Embodiments

[0023] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention and not to limit the invention. Additionally, it should be noted that for the sake of description, only parts related to the relevant invention are shown in the drawings.

[0024] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0025] Figure 1 It shows a flowchart of the image processing method provided by the embodiment of the present application, and the method includes:

[0026] Step 101, obtain the image to be processed.

[0027] In the present application, the image to be processed is an image that needs to be restored.

[0028] In some embodiments, the restoration of the image to be processed may be one of the following: image super-resolution, image denoising, and image de-raining.

[0029] Step 102: Use an image restoration network to restore the image to be processed, and obtain the restored image.

[0030] In this application, the image restoration network includes: an encoder, at least one long-distance information capture module, and a decoder.

[0031] In this application, the data input into the long-distance information capture module can be referred to as the input data of the long-distance information capture module. The data output by the long-distance information capture module after the long-distance information capture module processes the input data of the long-distance information capture module can be referred to as the output result of the long-distance information capture module.

[0032] In this application, the encoder is configured to perform convolutional processing on the image to be processed to obtain the encoded features of the image to be processed.

[0033] The encoder can be a residual module for feature extraction, and the structure of the residual module for feature extraction can be the same as the residual module in an existing neural network with a residual module, such as the residual module in a ResNet network. The encoder can also be the structure that completes the entire feature extraction process or a part of the structure in an existing convolutional neural network.

[0034] In this application, when the encoder performs convolutional processing on the image to be processed to obtain the encoded features of the image to be processed, the image to be processed is input into the encoder, and the encoder outputs the encoded features of the image to be processed.

[0035] In this application, if the restorations of multiple restoration types are all implemented through the image restoration network, that is, the image restoration network can restore the image to be processed in one of the multiple restoration types, then the encoder can include units corresponding to each of the multiple restoration types for extracting features through convolution. The units corresponding to the restoration type for extracting features through convolution can be residual modules, and the units corresponding to the restoration type for extracting features through convolution can also be the structure that completes the entire feature extraction process or a part of the structure in an existing convolutional neural network. For each of the multiple restoration types, the units corresponding to the restoration type for extracting features through convolution are used to extract features suitable for the restoration of this restoration type.

[0036] When performing the restoration of one restoration type among multiple restoration types on the image to be processed, after inputting the image to be processed into the image restoration network, the unit that extracts features through convolution corresponding to the one restoration type in the encoder receives the image to be processed, the unit that extracts features through convolution corresponding to the one restoration type performs convolution processing on the image to be processed, and the unit that extracts features through convolution corresponding to the one restoration type outputs the encoded features of the image to be processed.

[0037] In this application, when the image restoration network includes multiple long-distance information capture modules, the feature long-distance information capture modules other than the first feature long-distance information capture module are referred to as other feature long-distance information capture modules.

[0038] In this application, for each long-distance information capture module, the input data of the long-distance information capture module is a series of feature maps.

[0039] For each long-distance information capture module, the long-distance information capture module is configured to: divide the input data of the long-distance information capture module into the first feature map sequence of the long-distance information capture module and the second feature map sequence of the long-distance information capture module, where the input data of the first long-distance information capture module is the encoded features of the image to be processed, and the input data of the other long-distance information capture modules other than the first long-distance information capture module is the output result of the previous long-distance information capture module of the other long-distance information capture module; perform self-attention operations within each first window of the first feature map in the first feature map sequence of the long-distance information capture module to obtain the first feature corresponding to the long-distance information capture module; perform self-attention operations within each second window of the second feature map in the second feature map sequence of the long-distance information capture module to obtain the second feature corresponding to the long-distance information capture module; generate the output result of the long-distance information capture module based on the first feature corresponding to the long-distance information capture module and the second feature corresponding to the long-distance information capture module.

[0040] For a long-distance information capture module, performing self-attention operations within each first window of the first feature map in the first feature map sequence of the long-distance information capture module to obtain the first feature corresponding to the long-distance information capture module includes: for each first feature map in the first feature map sequence of the long-distance information capture module, performing self-attention operations within each first window of the first feature map.

[0041] For a long-distance information capture module, performing self-attention operations within each second window of each second feature map in the second feature map sequence of the long-distance information capture module to obtain the second feature corresponding to the long-distance information capture module includes: for each second feature map in the second feature map sequence of the long-distance information capture module, performing self-attention operations within each second window of the second feature map.

[0042] In this application, the input data of the first long-distance information capture module is the encoded features of the image to be processed. The long-distance information capture modules other than the first long-distance information capture module are collectively referred to as other long-distance information capture modules. When the image restoration network includes multiple long-distance information capture modules, for each other long-distance information capture module, the input data of the other long-distance information capture module is the output result of the previous long-distance information capture module of the other long-distance information capture module.

[0043] For example, the image restoration network includes multiple long-distance information capture modules, including 3 long-distance information capture modules. The second and third long-distance information capture modules are both other long-distance information capture modules. The input data of the first long-distance information capture module is the encoded features of the image to be processed. The output result of the first long-distance information capture module obtained by processing the input data of the first long-distance information capture module is used as the input data of the second long-distance information capture module. The output result of the second long-distance information capture module obtained by processing the input data of the second long-distance information capture module is used as the input data of the third long-distance information capture module.

[0044] In this application, for each long-distance information capture module, the positions of the first feature maps in the first feature map sequence of the long-distance information capture module are continuous in the input data of the long-distance information capture module, and the positions of the second feature maps in the second feature map sequence of the long-distance information capture module are continuous in the input data of the long-distance information capture module.

[0045] In one implementation, for each long-distance information capture module, the number of first feature maps included in the first feature map sequence of the long-distance information capture module is the same as the number of second feature maps included in the second feature map sequence of the long-distance information capture module. The input data of the long-distance information capture module includes N feature maps. The first N / 2 feature maps in the input data of the long-distance information capture module form the first feature map sequence of the long-distance information capture module, and each of the first N / 2 feature maps in the input data of the long-distance information capture module serves as a first feature map. The last N / 2 feature maps in the input data of the long-distance information capture module form the second feature map sequence of the long-distance information capture module, and each of the last N / 2 feature maps in the input data of the long-distance information capture module serves as a second feature map.

[0046] In this application, each first window can be a rectangular window, and each second window can be a rectangular window.

[0047] In this application, the windows that appear in the first feature maps of the long-distance information capture module are uniformly referred to as first windows.

[0048] In this application, the widths of each first window can be the same, the heights of each first window can be the same, and the sizes of each first window can be the same.

[0049] For each first feature map of any long-distance information capture module, all the first windows that appear in this first feature map can be obtained by dividing the frame enclosing this first feature map. The size of the frame enclosing this first feature map is the same as the size of this first feature map. The horizontal sides of the frame enclosing this first feature map are the horizontal sides of this first feature map, and the vertical sides of the frame enclosing this first feature map are the vertical sides of this first feature map.

[0050] In this application, for a long-distance information capture module, the first feature map sequence of the long-distance information capture module includes: each first feature map of the long-distance information capture module. For each first feature map of the long-distance information capture module, multiple first windows appear in this first feature map, and each first window in this first feature map encloses a part of this first feature map.

[0051] In this application, for a long-distance information capture module, the long-distance information capture module is configured to: for each first feature map of the long-distance information capture module, perform self-attention operations within each first window in this first feature map to obtain the first feature corresponding to the long-distance information capture module.

[0052] In this application, for each first feature map of a long-distance information capture module, when performing self-attention operations within each first window in the first feature map, for each first window in the first feature map, self-attention operations are performed on the part surrounded by the first window to obtain the self-attention operation result corresponding to the first window. Performing self-attention operations on the part surrounded by the first window to obtain the self-attention operation result corresponding to the first window can be achieved by: inputting the part surrounded by the first window into a multi-head self-attention (MSA) module for performing self-attention operations within the first window. The multi-head self-attention module for performing self-attention operations within the first window outputs the self-attention operation result corresponding to the first window. The self-attention operation results corresponding to each such first window form the self-attention operation result corresponding to the first feature map.

[0053] In this application, for each long-distance information capture module, after obtaining the self-attention operation result corresponding to each first feature map of the long-distance information capture module, the self-attention operation results corresponding to each first feature map in the first feature map sequence of the long-distance information capture module form the first feature corresponding to the long-distance information capture module.

[0054] In this application, the windows that appear in the second feature map of the long-distance information capture module are uniformly referred to as second windows.

[0055] In this application, the width of each second window can be the same, the height of each second window can be the same, and the size of each second window can be the same.

[0056] For each second feature map of any long-distance information capture module, all the second windows that appear in the second feature map can be obtained by dividing the frame surrounding the second feature map. The size of the frame surrounding the second feature map is the same as the size of the second feature map. The horizontal side of the frame surrounding the second feature map is the horizontal side of the second feature map, and the vertical side of the frame surrounding the second feature map is the vertical side of the second feature map.

[0057] In this application, for each long-distance information capture module, the second feature map sequence of the long-distance information capture module includes: each second feature map of the long-distance information capture module. For each second feature map of the long-distance information capture module, multiple second windows appear in the second feature map, and each second window in the second feature map surrounds a part of the second feature map.

[0058] In this application, for a long-distance information capture module, the long-distance information capture module is configured to perform self-attention operations within each second window of each second feature map of the long-distance information capture module to obtain the second feature corresponding to the long-distance information capture module.

[0059] In this application, for each second feature map of a long-distance information capture module, when performing self-attention operations within each second window of the second feature map, for each second window in the second feature map, self-attention operations are performed on the part surrounded by the second window to obtain the self-attention operation result corresponding to the second window. Performing self-attention operations on the part surrounded by the second window to obtain the self-attention operation result corresponding to the second window can be: inputting the part surrounded by the second window into the multi-head self-attention module for performing self-attention operations within the second window, and the multi-head self-attention module for performing self-attention operations within the second window outputs the self-attention operation result corresponding to the second window. The self-attention operation results corresponding to each of the second windows form the self-attention operation result corresponding to the second feature map.

[0060] In this application, for a long-distance information capture module, after obtaining the self-attention operation results corresponding to each second feature map of the long-distance information capture module, the self-attention operation results corresponding to each second feature map of the long-distance information capture module form the second feature corresponding to the long-distance information capture module.

[0061] In this application, for a long-distance information capture module, based on the first feature corresponding to the long-distance information capture module and the second feature corresponding to the long-distance information capture module, the output result of the long-distance information capture module is generated. When generating the output result of the long-distance information capture module based on the first feature corresponding to the long-distance information capture module and the second feature corresponding to the long-distance information capture module, the first feature corresponding to the long-distance information capture module and the second feature corresponding to the long-distance information capture module can be concatenated to obtain the output result of the long-distance information capture module.

[0062] In this application, the decoder is configured to decode the output result of the last long-distance information capture module to obtain the decoded feature of the image to be processed, and based on the decoded feature of the image to be processed, obtain the restored image.

[0063] For example, the image inpainting network includes multiple long-distance information capture modules, including 3 long-distance information capture modules, and the 3rd long-distance information capture module is the last long-distance information capture module. The input data of the 1st long-distance information capture module is the encoded features of the image to be processed. The output result of the 1st long-distance information capture module obtained by processing the input data of the 1st long-distance information capture module is used as the input data of the 2nd long-distance information capture module. The output result of the 2nd long-distance information capture module obtained by processing the input data of the 2nd long-distance information capture module is used as the input data of the 3rd long-distance information capture module. The 3rd long-distance information capture module processes the input data of the 3rd long-distance information capture module to obtain the output result of the 3rd long-distance information capture module. The decoder decodes the output result of the 3rd long-distance information capture module to obtain the decoded features of the image to be processed.

[0064] The decoder may include a residual module for obtaining the inpainted image. The structure of the residual module for obtaining the inpainted image may be the structure of the residual module in an existing neural network with a residual module, such as the residual module in a ResNet network.

[0065] In this application, when the decoder decodes the output result of the last long-distance information capture module to obtain the decoded features, the output result of the last long-distance information capture module is input into the decoder, and the decoder outputs the decoded features of the image to be processed. When obtaining the inpainted image based on the decoded features of the image to be processed, bilinear interpolation can be performed on the image to be processed based on the decoded features of the image to be processed to obtain the inpainted image.

[0066] In this application, if the inpainting of multiple inpainting types is all implemented through the image inpainting network, in other words, the image inpainting network can perform the inpainting of one inpainting type among multiple inpainting types on the image to be processed, then the decoder may include a residual module for obtaining the inpainted image corresponding to each inpainting type among the multiple inpainting types. For each inpainting type among the multiple inpainting types, the residual module for obtaining the inpainted image corresponding to this inpainting type outputs the decoded features suitable for the inpainting of this inpainting type.

[0067] When performing the inpainting of one inpainting type among multiple inpainting types on the image to be processed, the residual module for obtaining the inpainted image corresponding to this one inpainting type in the decoder receives the output result of the last long-distance information capture module, and the residual module for obtaining the inpainted image corresponding to this one inpainting type outputs the decoded features of the image to be processed.

[0068] Please refer toFigure 2 , which shows a schematic flow chart of using an image inpainting network to inpaint a to-be-processed image.

[0069] After inputting the to-be-processed image into the image inpainting network, the encoder performs convolutional processing on the to-be-processed image to obtain the encoded features of the to-be-processed image. The encoded features of the to-be-processed image serve as the input data of the first long-distance information capture module in the image inpainting network. The output result of the last long-distance information capture module serves as the input data of the decoder. The decoder decodes the output result of the last long-distance information capture module to obtain the decoded features of the to-be-processed image, and based on the decoded features of the to-be-processed image, obtains the inpainted image. The decoder outputs the inpainted image.

[0070] In some embodiments, for each first feature map of the first long-distance information capture module, the length of the horizontal side of the first window in the first feature map is greater than the length of the vertical side of the first window; for each second feature map of the first long-distance information capture module, the length of the horizontal side of the second window in the second feature map is less than the length of the vertical side of the second window in the second feature map.

[0071] The length of the horizontal side of the first window in the first feature map of the first long-distance information capture module is greater than the length of the vertical side of the first window, such that the first window is suitable for combination with the first long-distance information capture module to capture long-distance information in the horizontal direction. The length of the horizontal side of the second window in the second feature map of the first long-distance information capture module is less than the length of the vertical side of the second window, such that the second window is suitable for combination with the second long-distance information capture module to capture long-distance information in the vertical direction. Thus, the first long-distance information capture module can capture long-distance information in multiple directions, namely the horizontal direction and the vertical direction, to comprehensively capture long-distance information.

[0072] In some embodiments, each first window in the first feature map of the first long-distance information capture module is obtained by dividing the frame surrounding the first feature map. Each second window in the second feature map of the first long-distance information capture module is obtained by dividing the frame surrounding the second feature map. The sizes of each first window in the first feature map of the first long-distance information capture module are the same. The sizes of the first second windows in the second feature map of the first long-distance information capture module are the same. For each first window in the first feature map of each other long-distance information capture module, it is obtained by adding each first window in the first feature map of the previous long-distance information capture module of the other long-distance information capture module to the first feature map of the other long-distance information capture module and translating each added first window in the first feature map of the other long-distance information capture module. For each second window in the second feature map of each other long-distance information capture module, it is obtained by adding each second window in the second feature map of the previous long-distance information capture module of the other long-distance information capture module to the second feature map of the other long-distance information capture module and translating each added second window in the second feature map of the other long-distance information capture module.

[0073] In this application, for each first feature map of the first long-distance information capture module, the frame surrounding the first feature map is divided to obtain each first window in the first feature map. The sizes of each first window in the first feature map are the same. The size of the frame surrounding the first feature map is the same as the size of the first feature map. The horizontal side of the frame surrounding the first feature map is the horizontal side of the first feature map. The vertical side of the frame surrounding the first feature map is the vertical side of the first feature map.

[0074] In this application, for each second feature map of the first long-distance information capture module, the frame surrounding the second feature map is divided to obtain each second window in the second feature map. The sizes of each second window in the second feature map are the same. The size of the frame surrounding the second feature map is the same as the size of the second feature map. The horizontal side of the frame surrounding the second feature map is the horizontal side of the second feature map. The vertical side of the frame surrounding the second feature map is the vertical side of the second feature map.

[0075] In this application, for each other long-distance information capture module, in order to obtain the first window in each first feature map of the other long-distance information capture module, for each first feature map of the other long-distance information capture module, each first window in a first feature map of the previous long-distance information capture module of the other long-distance information capture module can be added to the first feature map, and the first feature map can be any first feature map of the previous long-distance information capture module of the other long-distance information capture module.

[0076] For each first feature map of any other long-distance information capture module, after adding each first window in the first feature map of the previous long-distance information capture module of the other long-distance information capture module to the first feature map, translate each first window added to the first feature map respectively. The translation of the first window added to the first feature map can be: first, move the first window added to the first feature map in the horizontal direction, and the horizontal movement can be a leftward movement or a rightward movement. Then, move the first window added to the first feature map in the vertical direction, and the vertical movement can be a downward movement or an upward movement. The translation of the first window added to the first feature map can also be: first, move the first window added to the first feature map in the vertical direction, and then move the first window added to the first feature map in the horizontal direction. Translating the first window added to the first feature map can form a new first window.

[0077] For each first feature map of any other long-distance information capture module, translate each first window added to the first feature map respectively. All the first windows located in the first feature map formed by translation and each first window added to the first feature map constitute all the first windows in the first feature map.

[0078] In this application, for any first window added to the first feature map, the amount of movement of the horizontal movement of the first window added to the first feature map is related to the width of the first window, and the amount of movement of the vertical movement of the first window added to the first feature map is related to the height of the first window.

[0079] In one implementation, for any first window added to the first feature map, the amount of movement of the horizontal movement of the first window added to the first feature map is 1 / 2 of the width of the first window added to the first feature map, and the amount of movement of the vertical movement of the first window added to the first feature map is 1 / 2 of the height of the first window added to the first feature map.

[0080] In this application, for each other long-distance information capture module, in order to obtain the second window in each second feature map of the other long-distance information capture module, for each second feature map of the other long-distance information capture module, each second window in a second feature map of the previous long-distance information capture module of the other long-distance information capture module may be added to the second feature map, and the second feature map may be any second feature map of the previous long-distance information capture module of the other long-distance information capture module.

[0081] For each second feature map of any other long-distance information capture module, after adding each second window in the second feature map of the previous long-distance information capture module of the other long-distance information capture module to the second feature map, each second window added to the second feature map is respectively translated. The translation of the second window added to the second feature map may be: first, the second window added to the second feature map is moved horizontally, and the horizontal movement may be a leftward movement or a rightward movement. Then, the first window added to the second feature map is moved vertically, and the vertical movement may be a downward movement or an upward movement. The translation of the second window added to the second feature map may also be: first, the second window added to the second feature map is moved vertically, and then, the second window added to the second feature map is moved horizontally. Translating the second window added to the second feature map may form a new second window.

[0082] For each second feature map of any other long-distance information capture module, each second window added to the second feature map is respectively translated. All the second windows located in the second feature map formed by translation and each second window added to the second feature map constitute all the second windows in the second feature map.

[0083] In this application, for any second window added to the second feature map, the movement amount of the horizontal movement of the second window added to the second feature map is related to the width of the second window, and the movement amount of the vertical movement of the second window added to the second feature map is related to the height of the second window.

[0084] In one implementation, for any second window added to the second feature map, the movement amount of the horizontal movement of the second window added to the second feature map is 1 / 2 of the width of the second window added to the second feature map, and the movement amount of the vertical movement of the second window added to the second feature map is 1 / 2 of the height of the second window added to the second feature map.

[0085] Please refer to Figure 3 , which shows a schematic diagram of the effect of the window in the feature map.

[0086] In Figure 3 , a first feature map 301 of a first long-distance information capture module, a first feature map 302 of a second long-distance information capture module, a second feature map 303 of the first long-distance information capture module, and a second feature map 304 of the second long-distance information capture module are exemplarily shown. In Figure 3 , the first windows in the first feature map 301 such as the first window 3011 and the first window 3012 are exemplarily shown. The second windows in the second feature map 302 such as the second window 3021, the second window 3022, and the second window 3023 are exemplarily shown. The first windows in the first feature map 303 such as the first window 3031 and the first window 3032 are exemplarily shown. The second windows in the second feature map 304 such as the second window 3041, the second window 3042, and the second window 3043 are exemplarily shown. Each first window in the first feature map 301 can be added to the first feature map 302, and each first window added to the first feature map 302 can be translated respectively. By translating each first window added to the first feature map, new first windows can be formed. All the first windows located in the first feature map 302 formed by translation and each first window added to the first feature map 302 constitute all the first windows in the first feature map 302. Each second window in the second feature map 303 can be added to the second feature map 304, and each second window added to the second feature map 304 can be translated respectively. By translating each second window added to the second feature map, new second windows can be formed. All the second windows located in the second feature map formed by translation and each second window added to the second feature map 304 constitute all the second windows in the second feature map 304.

[0087] In some embodiments, for any long-distance information capture module, generating the output result of the long-distance information capture module based on the first feature corresponding to the long-distance information capture module and the second feature corresponding to the long-distance information capture module includes: splicing the first feature corresponding to the long-distance information capture module and the second feature corresponding to the long-distance information capture module into the splicing feature corresponding to the long-distance information capture module; performing convolution processing on the splicing feature corresponding to the long-distance information capture module to obtain the convolved splicing feature corresponding to the long-distance information capture module; and generating the output result of the long-distance information capture module based on the convolved splicing feature corresponding to the long-distance information capture module.

[0088] Perform convolution processing on the splicing features corresponding to the long-distance information capture module, and the convolution-processed splicing features corresponding to the long-distance information capture module can be expressed as:

[0089] (S)CL-MSA(X) = Proj[X1’, X2’]

[0090] (S)CL-MSA(X) represents the convolution-processed splicing features corresponding to the long-distance information capture module, and Proj[X1’, X2’] represents performing convolution processing on the splicing features corresponding to the long-distance information capture module.

[0091] For any long-distance information capture module, when performing convolution processing on the splicing features corresponding to the long-distance information capture module, the splicing features corresponding to the long-distance information capture module can be input into a convolutional layer for performing convolution processing on the splicing features, and the convolutional layer for performing convolution processing on the splicing features outputs the convolution-processed splicing features corresponding to the long-distance information capture module.

[0092] When generating the output result of the long-distance information capture module based on the convolution-processed splicing features corresponding to the long-distance information capture module, the convolution-processed splicing features corresponding to the long-distance information capture module can be used as the output result of the long-distance information capture module.

[0093] In some embodiments, for any long-distance information capture module, generating the output result of the long-distance information capture module based on the convolution-processed splicing features corresponding to the long-distance information capture module includes: performing linear and activation processing on the convolution-processed splicing features corresponding to the long-distance information capture module to obtain the linearly and activation-processed features corresponding to the long-distance information capture module; performing anti-blocking effect processing on the linearly and activation-processed features to obtain the anti-blocking effect-processed features corresponding to the long-distance information capture module, where performing anti-blocking effect processing on the linearly and activation-processed features includes: using convolution kernels with scales larger than the size threshold to perform convolution on each feature map of the linearly and activation-processed features; generating the output result of the long-distance information capture module based on the anti-blocking effect-processed features corresponding to the long-distance information capture module.

[0094] Perform linear and activation processing on the concatenated features after convolution corresponding to the long-distance information capture module, and the features after linear and activation processing corresponding to the long-distance information capture module can be obtained as follows: Input the concatenated features after convolution corresponding to the long-distance information capture module into a fully connected layer, and the fully connected layer performs linear processing on the concatenated features after convolution corresponding to the long-distance information capture module to obtain the output result of the fully connected layer. Input the output result of the fully connected layer into an activation function layer, and the activation function layer performs activation processing on the output result of the fully connected layer. The activation function layer outputs the features after linear and activation processing corresponding to the long-distance information capture module.

[0095] Perform linear and activation processing on the concatenated features after convolution corresponding to the long-distance information capture module, and the features after linear and activation processing corresponding to the long-distance information capture module are represented as:

[0096] X’ = Act(Linear(X))

[0097] X represents the concatenated features after convolution corresponding to the long-distance information capture module, Linear() represents linear processing using a fully connected layer, Act() represents activation processing using an activation function layer, and X’ represents the features after linear and activation processing corresponding to the long-distance information capture module.

[0098] In this application, for any long-distance information capture module, after obtaining the features after linear and activation processing corresponding to the long-distance information capture module, anti-blocking effect processing can be performed on the features after linear and activation processing corresponding to the long-distance information capture module to obtain the features after anti-blocking effect processing corresponding to the long-distance information capture module.

[0099] Performing anti-blocking effect processing on the features after linear and activation processing corresponding to the long-distance information capture module includes: using convolution kernels with a scale larger than the size threshold to perform convolution on each feature map of the features after linear and activation processing corresponding to the long-distance information capture module.

[0100] In this application, considering that the self-attention operation is performed in units of windows and the windows may be discontinuous, anti-blocking effect processing is used to eliminate the adverse effects on the accuracy of the first feature and the second feature extracted through the self-attention operation due to the possible discontinuity between windows.

[0101] The size threshold can be a conventional convolution kernel size of 3x3. In one implementation, the convolution kernel with a scale larger than the size threshold can be a 5x5 convolution kernel.

[0102] For each feature map in the features after linear and activation processing corresponding to the long-distance information capture module, a convolutional kernel with a scale larger than the size threshold is used to perform convolution on the feature map to obtain the convolution result corresponding to the feature map, and the convolution results corresponding to each of the feature maps form the features after anti-blocking effect processing corresponding to the long-distance information capture module.

[0103] When generating the output result of the long-distance information capture module based on the features after anti-blocking effect processing corresponding to the long-distance information capture module, the features after anti-blocking effect processing can be input into the activation function layer to obtain the output result output by the activation function layer, and the output result output by the activation function layer is input into the fully connected layer, and the output result output by the fully connected layer is used as the output result of the long-distance information capture module.

[0104] Performing anti-blocking effect processing on the features after linear and activation processing corresponding to the long-distance information capture module to obtain the features after anti-blocking effect processing corresponding to the long-distance information capture module, and generating the output result of the long-distance information capture module based on the features after anti-blocking effect processing is expressed as:

[0105] X” = Linear(Act(Anti - Block(X’)))

[0106] X” represents the output result of the long-distance information capture module, X’ represents the features after linear and activation processing corresponding to the long-distance information capture module, Anti - Block() represents anti-blocking effect processing, Linear() represents linear processing using the fully connected layer, and Act() represents activation processing using the activation function layer.

[0107] In some embodiments, the encoder includes: a first residual module and a second residual module. The input data of the first residual module is determined based on the image to be processed, the input data of the second residual module is determined based on the output result of the first residual module, and the output result of the second residual module is the encoded feature of the image to be processed. The decoder includes: a third residual module and a fourth residual module. The input data of the third residual module is determined based on the output result of the last long-distance information capture module, the input data of the fourth residual module is determined based on the output result of the third residual module, and the output result of the fourth residual module is the decoded feature of the image to be processed.

[0108] In this application, the structure of the first residual module is the same as that of the second residual module. When the encoder performs convolutional processing on the image to be processed to obtain the encoded features of the image to be processed, the image to be processed can be used as the input data of the first residual module. The image to be processed is input into the first residual module, and the first residual module outputs the output result of the first residual module. The output result of the first residual module can be used as the input data of the second residual module. The output result of the first residual module is input into the second residual module, and the second residual module outputs the output result of the second residual module. The output result of the second residual module is the encoded features of the image to be processed.

[0109] In this application, the structure of the third residual module is the same as that of the fourth residual module. The output result of the last long-distance information capture module can be used as the third residual module. The output result of the last long-distance information capture module is input into the third residual module, and the third residual module outputs the output result of the third residual module. The output result of the third residual module can be used as the input data of the fourth residual module. The output result of the third residual module is input into the fourth residual module, and the fourth residual module outputs the output result of the fourth residual module. The output result of the fourth residual module is the decoded features of the image to be processed.

[0110] In some embodiments, the resolution of the image to be processed is higher than the resolution threshold. The input data of the first residual module is obtained by downsampling the image to be processed by a preset multiple. The input data of the second residual module is obtained by downsampling the output result of the first residual module by a preset multiple. The input data of the third residual module is obtained by upsampling the output result of the last long-distance information capture module by a preset multiple. The input data of the fourth residual module is obtained by upsampling the output result of the third residual module by a preset multiple.

[0111] In this application, when the resolution of the image to be processed is higher than the resolution threshold, that is, when the image to be processed is a high-resolution image, the image to be processed can be downsampled by a preset multiple to obtain the input data of the first residual module. The output result of the first residual module can be downsampled by a preset multiple to obtain the input data of the second residual module. The output result of the last long-distance information capture module can be upsampled by a preset multiple to obtain the input data of the third residual module. The output result of the third residual module can be upsampled by a preset multiple to obtain the input data of the fourth residual module.

[0112] In one implementation, the downsampling of the preset multiple is 2-fold downsampling, and the upsampling of the preset multiple is 2-fold upsampling.

[0113] Please refer to Figure 4 , which shows a schematic diagram of the structure of the residual module.

[0114] In Figure 4 it, the structures of the residual modules in the encoder and the residual modules in the decoder are exemplarily shown. The residual module in the encoder includes two 3x3 convolutional layers and one 1x1 convolutional layer. The residual module in the decoder includes two 3x3 convolutional layers and one 1x1 convolutional layer.

[0115] Please refer to Figure 5 which shows a structural block diagram of an image processing device provided by an embodiment of the present application. The device includes: an acquisition unit 501 and a processing unit 502.

[0116] The acquisition unit 501 is configured to acquire an image to be processed;

[0117] The processing unit 502 is configured to use an image inpainting network to inpaint the image to be processed to obtain an inpainted image. The image inpainting network includes: an encoder configured to perform convolutional processing on the image to be processed to obtain encoded features of the image to be processed; at least one long-distance information capture module configured to divide the input data of the long-distance information capture module into a first feature map sequence and a second feature map sequence of the long-distance information capture module, where the input data of the first long-distance information capture module is the encoded features, and the input data of other long-distance information capture modules except the first long-distance information capture module is the output result of the previous long-distance information capture module of the other long-distance information capture module; perform self-attention operations within each first window of the first feature maps in the first feature map sequence to obtain the first features corresponding to the long-distance information capture module; perform self-attention operations within each second window of the second feature maps in the second feature map sequence to obtain the second features corresponding to the long-distance information capture module; generate the output result of the long-distance information capture module based on the first features and the second features; generate the output result of the long-distance information capture module based on the first features and the second features; and a decoder configured to decode the output result of the last long-distance information capture module to obtain the decoded features of the image to be processed and, based on the decoded features, obtain the inpainted image.

[0118] In some embodiments, generating the output result of the long-distance information capture module based on the first features and the second features includes: concatenating the first features and the second features into a concatenated feature corresponding to the long-distance information capture module; performing convolutional processing on the concatenated feature to obtain a convolution-concatenated feature corresponding to the long-distance information capture module; and generating the output result of the long-distance information capture module based on the convolution-concatenated feature.

[0119] In some embodiments, generating the output result of the long-distance information capture module based on the concatenated features after convolution includes: performing linear and activation processing on the concatenated features after convolution to obtain the features after linear and activation processing corresponding to the long-distance information capture module; performing anti-blocking effect processing on the features after linear and activation processing to obtain the features after anti-blocking effect processing corresponding to the long-distance information capture module, where the anti-blocking effect processing includes: respectively performing convolution on each feature map of the features after linear and activation processing by using a convolution kernel with a scale greater than a size threshold; generating the output result of the long-distance information capture module based on the features after anti-blocking effect processing.

[0120] In some embodiments, each first window in the first feature map of the first long-distance information capture module is obtained by dividing the frame enclosing the first feature map, and each second window in the second feature map of the first long-distance information capture module is obtained by dividing the frame enclosing the second feature map. The sizes of each first window are the same, and the sizes of the first second windows are the same; each first window in the first feature map of the other long-distance information capture modules is obtained by adding each first window in the first feature map of the previous long-distance information capture module of the other long-distance information capture modules to the first feature map of the other long-distance information capture modules and translating each first window added to the first feature map of the other long-distance information capture modules, and each second window in the second feature map of the other long-distance information capture modules is obtained by adding each second window in the second feature map of the previous long-distance information capture module of the other long-distance information capture modules to the second feature map of the other long-distance information capture modules and translating each second window added to the second feature map of the other long-distance information capture modules.

[0121] In some embodiments, the length of the horizontal side of the first window in the first feature map of the first long-distance information capture module is greater than the length of the vertical side of the first window, and the length of the horizontal side of the second window in the second feature map of the first long-distance information capture module is less than the length of the vertical side of the second window.

[0122] In some embodiments, the encoder includes: a first residual module and a second residual module. The input data of the first residual module is determined based on the image to be processed. The input data of the second residual module is determined based on the output result of the first residual module. The output result of the second residual module is the encoded feature of the image to be processed. The decoder includes: a third residual module and a fourth residual module. The input data of the third residual module is determined based on the output result of the last long-distance information capture module. The input data of the fourth residual module is determined based on the output result of the third residual module. The output result of the fourth residual module is the decoded feature of the image to be processed.

[0123] In some embodiments, the resolution of the image to be processed is higher than a resolution threshold. The input data of the first residual module is obtained by downsampling the image to be processed by a preset multiple. The input data of the second residual module is obtained by downsampling the output result of the first residual module by a preset multiple. The input data of the third residual module is obtained by upsampling the output result of the last long-distance information capture module by a preset multiple. The input data of the fourth residual module is obtained by upsampling the output result of the third residual module by a preset multiple.

[0124] In some embodiments, the restoration is one of the following: image super-resolution, image denoising, and image de-raining.

[0125] An embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the above image processing method.

[0126] An embodiment of the present application provides a computer-readable storage medium, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, the above image processing method is implemented.

[0127] An embodiment of the present application provides a computer program product, including computer program / instructions. When the computer program / instructions are executed by a processor, the above image processing method is implemented.

[0128] It should be noted that a computer-readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium that includes or stores a program, which can be used by or in conjunction with a message execution system, apparatus, or device. And in the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with a message execution system, apparatus, or device. The program code included on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, a program segment, or a portion of code that includes one or more executable messages for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in a block may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer messages.

[0130] The above description is only a preferred embodiment of this request and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical embodiments formed by the specific combination of the above technical features, but should also cover other technical embodiments formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, technical embodiments formed by mutually replacing the above features with technical features (but not limited to) having similar functions disclosed in this application.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is repaired using an image repair network to obtain a repaired image, wherein the image repair network includes: An encoder is configured to perform convolution processing on the image to be processed to obtain encoding features of the image to be processed; At least one long-distance information capture module, the long-distance information capture module being configured to: divide the input data of the long-distance information capture module into a first feature map sequence of the long-distance information capture module and a second feature map sequence of the long-distance information capture module, wherein the input data of the first long-distance information capture module is the encoded feature, and the input data of other long-distance information capture modules except the first long-distance information capture module is the output result of the previous long-distance information capture module of the other long-distance information capture modules; perform a self-attention operation in each first window in the first feature map in the first feature map sequence to obtain a first feature corresponding to the long-distance information capture module; perform a self-attention operation in each second window in the second feature map in the second feature map sequence to obtain a second feature corresponding to the long-distance information capture module; and generate an output result of the long-distance information capture module based on the first feature and the second feature; The decoder is configured to decode the output result of the last long-distance information capture module to obtain the decoding features of the image to be processed, and obtain the restored image based on the decoding features.

2. The method according to claim 1, characterized in that Generating an output result of the long-distance information acquisition module based on the first feature and the second feature includes: splicing the first feature and the second feature into a splicing feature corresponding to the long-distance information capture module; Performing convolution processing on the splicing features to obtain the convolved splicing features corresponding to the long-distance information capture module; Based on the convolutional splicing features, the output result of the long-distance information capture module is generated.

3. The method according to claim 2, characterized in that Generating the output result of the long-distance information capture module based on the convolutional splicing features includes: Performing linear sum activation processing on the convolved concatenated features to obtain linear sum activation processed features corresponding to the long-distance information capture module; performing anti-blocking processing on the linear and activation processed features to obtain anti-blocking processed features corresponding to the long-distance information capture module, wherein the anti-blocking processing comprises: convolving each feature map of the linear and activation processed features with a convolution kernel having a scale greater than a size threshold; Based on the features after the anti-blocking effect processing, an output result of the long-distance information capture module is generated.

4. The method according to claim 1, wherein Each first window in the first feature map of the first long-distance information capture module is obtained by dividing the box surrounding the first feature map, and each second window in the second feature map of the first long-distance information capture module is obtained by dividing the box surrounding the second feature map, and the size of each first window is the same, and the size of each second window is the same; each first window in the first feature map of the other long-distance information capture module is obtained by adding each first window in the first feature map of the previous long-distance information capture module of the other long-distance information capture module to the first feature map of the other long-distance information capture module and translating each first window added to the first feature map of the other long-distance information capture module, and each second window in the second feature map of the other long-distance information capture module is obtained by adding each second window in the second feature map of the previous long-distance information capture module of the other long-distance information capture module to the second feature map of the other long-distance information capture module and translating each second window added to the second feature map of the other long-distance information capture module.

5. The method according to claim 1, characterized in that The length of the horizontal side of the first window in the first feature map of the first long-distance information capture module is greater than the length of the vertical side of the first window, and the length of the horizontal side of the second window in the second feature map of the first long-distance information capture module is less than the length of the vertical side of the second window.

6. The method according to claim 1, characterized in that The encoder includes: a first residual module and a second residual module, the input data of the first residual module is determined based on the image to be processed, the input data of the second residual module is determined based on the output result of the first residual module, and the output result of the second residual module is the encoding feature of the image to be processed. The decoder includes: a third residual module and a fourth residual module, the input data of the third residual module is determined based on the output result of the last long-distance information capture module, the input data of the fourth residual module is determined based on the output result of the third residual module, and the output result of the fourth residual module is the decoding feature of the image to be processed.

7. The method according to claim 6, characterized in that The resolution of the image to be processed is higher than the resolution threshold, the input data of the first residual module is obtained by downsampling the image to be processed by a preset multiple, the input data of the second residual module is obtained by downsampling the output result of the first residual module by a preset multiple, the input data of the third residual module is obtained by upsampling the output result of the last long-distance information capture module by a preset multiple, and the input data of the fourth residual module is obtained by upsampling the output result of the third residual module by a preset multiple.

8. The method according to any one of claims 1 to 7, characterized in that: The restoration is one of the following: image super-resolution, image denoising, and image deraining.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images