Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment. The method comprises the following steps: after an initial image is obtained, processing a multi-scale feature map obtained by encoding the initial image by using different filtering weights to obtain a target residual image; and performing target processing on the initial image and the target residual image to obtain a target output image.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] When capturing images through a camera, the light incident on the sensor by the object may be attenuated and diffracted, which may cause quality problems such as glare, noise and color shift in the image captured by the camera. Among them, the noise and color shift problems can be alleviated or solved by the powerful image processing capabilities of the image signal processing pipeline (ISP); however, the image glare problem has not yet been effectively solved. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device and electronic device to at least solve the above technical problems existing in the prior art.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The embodiment of the present application provides an image processing method, comprising: after obtaining an initial image, using different filtering weights to process a multi-scale feature map obtained by encoding the initial image to obtain a target residual image;

[0006] Target processing is performed on the initial image and the target residual image to obtain a target output image.

[0007] In some embodiments, the step of processing the multi-scale feature map obtained by encoding the initial image using different filtering weights to obtain a target residual image includes:

[0008] Encoding the initial image into a first feature map set, wherein the first feature map set includes feature maps of different scales;

[0009] Converting the first feature map set into a second feature map set using corresponding filter weights, wherein the second feature map set includes feature maps of different scales;

[0010] The feature maps in the second feature map set are decoded in sequence to obtain the target residual image.

[0011] In some embodiments, converting the first set of feature maps into a second set of feature maps using corresponding filter weights comprises:

[0012] Performing convolution processing on a first feature map included in the first feature map set using a first filtering weight to obtain a fourth feature map in the second feature map set;

[0013] Performing convolution processing on the second feature map included in the first feature map set using a second filtering weight to obtain a fifth feature map in the second feature map set;

[0014] Performing convolution processing on the third feature map included in the first feature map set using a third filtering weight to obtain a sixth feature map in the second feature map set;

[0015] Among them, the first feature map, the second feature map and the third feature map correspond to different scale spaces, and the fourth feature map, the fifth feature map and the sixth feature map correspond to different scale spaces.

[0016] In some embodiments, the decoding process is performed on the feature maps in the second feature map set in sequence to obtain the target residual image, including:

[0017] Decoding the sixth feature map to obtain a seventh feature map;

[0018] Accumulating and decoding the seventh feature map and the fifth feature map to obtain an eighth feature map, wherein the eighth feature map has the same scale space as the fourth feature map;

[0019] The eighth feature map and the fourth feature map are accumulated and then residual processing is performed to obtain the target residual image.

[0020] In some embodiments, the method further comprises:

[0021] Processing the initial image to obtain a target condition vector, wherein the target condition vector can represent degradation feature information of the initial image;

[0022] Extracting a third feature map set of the initial image using the target condition vector and the initial image;

[0023] Convolution and residual processing are performed on each feature map in the third feature map set to obtain filter weights in different scale spaces.

[0024] In some embodiments, the processing the initial image to obtain the target condition vector includes:

[0025] extracting degradation features of the initial image;

[0026] The target condition vector is determined based on the degradation characteristic.

[0027] In some embodiments, extracting a third feature map set of the initial image using the target condition vector and the initial image includes:

[0028] Performing convolution and spatial feature conversion processing on the initial image using a first target condition vector to extract a first scale-space feature map, wherein the first target condition vector has the same shape as the first scale-space feature map;

[0029] Using a second target condition vector to perform convolution and spatial feature conversion processing on the first scale space feature map to extract a second scale space feature map, wherein the second target condition vector has the same shape as the second scale space feature map;

[0030] Using a third target condition vector to perform convolution and spatial feature conversion processing on the second scale space feature map to extract a third scale space feature map, wherein the third target condition vector has the same shape as the third scale space feature map;

[0031] The first target condition vector, the second target condition vector and the third target condition vector are obtained based on converting the target condition vector.

[0032] In some embodiments, extracting a third feature map set of the initial image using the target condition vector and the initial image includes:

[0033] Converting the target conditional vector into a ninth feature map having the same shape as the input feature map of the different scale spaces of the initial image;

[0034] The different ninth feature maps are merged with the input feature maps of the corresponding scale space into the tenth feature map;

[0035] Perform convolution processing on different tenth feature maps to obtain corresponding feature tensors;

[0036] The feature tensor is used to perform spatial feature conversion processing on the input feature map of the corresponding scale space to obtain a third feature map set including multi-scale spatial feature maps.

[0037] The present application provides a data processing device, including:

[0038] A determination module, configured to, after obtaining an initial image, process a multi-scale feature map obtained by encoding the initial image using different filtering weights to obtain a target residual image;

[0039] The processing module is used to perform target processing on the initial image and the target residual image to obtain a target output image.

[0040] In some embodiments, the determining module is used to encode the initial image into a first feature map set, wherein the first feature map set includes feature maps of different scales;

[0041] Converting the first feature map set into a second feature map set using corresponding filter weights, wherein the second feature map set includes feature maps of different scales;

[0042] The feature maps in the second feature map set are decoded in sequence to obtain the target residual image.

[0043] In some embodiments, the determining module is used to perform convolution processing on the first feature map included in the first feature map set using the first filtering weight to obtain a fourth feature map in the second feature map set;

[0044] Performing convolution processing on the second feature map included in the first feature map set using a second filtering weight to obtain a fifth feature map in the second feature map set;

[0045] Performing convolution processing on the third feature map included in the first feature map set using a third filtering weight to obtain a sixth feature map in the second feature map set;

[0046] Among them, the first feature map, the second feature map and the third feature map correspond to different scale spaces, and the fourth feature map, the fifth feature map and the sixth feature map correspond to different scale spaces.

[0047] In some embodiments, the determining module is used to decode the sixth feature map to obtain a seventh feature map, and the seventh feature map has the same scale space as the fifth feature map;

[0048] Accumulating the seventh feature map and the fifth feature map and performing decoding processing to obtain an eighth feature map, wherein the eighth feature map has the same scale space as the fourth feature map;

[0049] The eighth feature map and the fourth feature map are accumulated and then residual processing is performed to obtain the target residual image.

[0050] In some embodiments, the processing module is used to process the initial image to obtain a target condition vector, wherein the target condition vector can represent degradation feature information of the initial image;

[0051] Extracting a third feature map set of the initial image using the target condition vector and the initial image;

[0052] Convolution and residual processing are performed on each feature map in the third feature map set to obtain filter weights in different scale spaces.

[0053] In some embodiments, the processing module is used to extract degradation features of the initial image;

[0054] The target condition vector is determined based on the degradation characteristic.

[0055] In some embodiments, the processing module is used to perform convolution and spatial feature conversion processing on the initial image using a first target condition vector to extract a first scale-space feature map, and the first target condition vector has the same shape as the first scale-space feature map;

[0056] Using a second target condition vector to perform convolution and spatial feature conversion processing on the first scale space feature map to extract a second scale space feature map, wherein the second target condition vector has the same shape as the second scale space feature map;

[0057] Using a third target condition vector to perform convolution and spatial feature conversion processing on the second scale space feature map to extract a third scale space feature map, wherein the third target condition vector has the same shape as the third scale space feature map;

[0058] The first target condition vector, the second target condition vector and the third target condition vector are obtained based on converting the target condition vector.

[0059] In some embodiments, the processing module is used to convert the target condition vector into a ninth feature map having the same shape as the input feature map of the different scale spaces of the initial image;

[0060] The different ninth feature maps are merged with the input feature maps of the corresponding scale space into the tenth feature map;

[0061] Perform convolution processing on different tenth feature maps to obtain corresponding feature tensors;

[0062] The feature tensor is used to perform spatial feature conversion processing on the input feature map of the corresponding scale space to obtain a third feature map set including multi-scale spatial feature maps.

[0063] An embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0064] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned image processing method.

[0065] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0066] The image processing method provided in the embodiment of the present application, after obtaining the initial image, uses different filtering weights to process the multi-scale feature map obtained by encoding the initial image to obtain a target residual image; and performs target processing on the initial image and the target residual image to obtain a target output image. The embodiment of the present application only needs to input the initial image, and does not need to obtain the point spread function (PSF) information corresponding to the acquisition device of the initial image, etc., in order to determine the target output image corresponding to the initial image. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a schematic diagram of an optional processing flow of the image processing method provided in an embodiment of the present application;

[0068] Figure 2 It is a schematic diagram of an optional processing flow for processing a multi-scale feature map obtained by encoding the initial image using different filtering weights to obtain a target residual image provided by an embodiment of the present application;

[0069] Figure 3 It is a schematic diagram of a specific implementation process of encoding an initial image into a first feature map set provided by an embodiment of the present application;

[0070] Figure 4 This is a schematic diagram of an optional processing flow for determining filtering weights provided in an embodiment of the present application;

[0071] Figure 5 It is a schematic diagram of a specific implementation process of determining a target condition vector provided in an embodiment of the present application;

[0072] Figure 6 It is a schematic diagram of a processing process for extracting a third feature map set of an initial image provided by an embodiment of the present application;

[0073] Figure 7 is a schematic diagram of a processing process for determining a first scale space feature map in a third feature map set provided by an embodiment of the present application;

[0074] Figure 8 is a schematic diagram of a process of determining the first scale-space feature map based on the feature tensor and the target condition vector provided in an embodiment of the present application;

[0075] Fig. 9 It is a schematic diagram of a specific process of sequentially decoding the feature maps in the second feature map set to obtain the target residual image provided by an embodiment of the present application;

[0076] Fig.10 It is a schematic diagram of an image processing process of an image to be processed acquired by an under-screen camera provided in an embodiment of the present application;

[0077] Fig.11 is an architecture diagram of an image processing method provided in an embodiment of the present application;

[0078] Fig.12 The architecture of the image processing method in the related technology provided by the embodiment of the present application Figure 1 ;

[0079] Fig.13 The architecture of the image processing method in the related technology provided by the embodiment of the present application Figure 2 ;

[0080] Fig.14 is a schematic diagram of the composition structure of an image processing device provided in an embodiment of the present application;

[0081] Fig.15 It is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0083] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0084] In the following description, the terms "first\second\third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here. In the following description, the term "multiple" refers to at least two.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0086] An optional processing flow diagram of the image processing method provided in the embodiment of the present application is as follows: Figure 1 As shown, at least the following steps are included:

[0087] Step S101, after obtaining an initial image, a multi-scale feature map obtained by encoding the initial image is processed using different filtering weights to obtain a target residual image.

[0088] In some embodiments, the initial image may be an original image captured by an image acquisition module, or an image stored locally in the electronic device, or an image transmitted by other electronic devices; the embodiments of the present application do not limit the method for acquiring the initial image.

[0089] In some embodiments, a multi-scale feature map obtained by encoding the initial image is processed using different filter weights to obtain an optional processing flow of a target residual image, such as Figure 2 As shown, at least the following steps are included:

[0090] Step S201: Encode the initial image into a first feature map set, wherein the first feature map set includes feature maps of different scales.

[0091] In some embodiments, the first feature map set may include feature maps of multiple different scales. Taking the first feature map set including the first feature map, the second feature map and the feature map of three different scales of the third feature map as an example, the specific implementation process of encoding the initial image into the first feature map set is as follows: Figure 3 As shown, it may include: performing convolution processing on the initial image, and then performing two consecutive residual processing on the initial image after the convolution processing to obtain a first feature map. Among them, performing two consecutive residual processing on the initial image after the convolution processing may be: performing a first residual processing on the initial image after the convolution processing to obtain a residual processing result; and then performing a second residual processing on the residual processing result to obtain the first feature map.

[0092] The first feature map is flattened and stretched, and then the feature map obtained by the flattening and stretching process is subjected to two residual processes to obtain a second feature map. The flattening and stretching process of the first feature map may be flattening and stretching the first feature map with fewer channels into a feature map with larger channels; for example, flattening and stretching the first feature map with 32 channels into a feature map with 64 channels. The flattening and stretching process of the first feature map may be implemented by a convolutional layer with a step size of 2.

[0093] The second feature map is flattened and stretched, and the feature map obtained by the flattening and stretching process is subjected to two residual processes to obtain a third feature map. The flattening and stretching process of the second feature map may be flattening and stretching the second feature map of fewer channels into a feature map of larger channels; for example, flattening and stretching the second feature map of 64 channels into a feature map of 128 channels. The first feature map can be represented by E1, the second feature map can be represented by E2, and the third feature map can be represented by E3. The flattening and stretching process of the second feature map can be implemented by a convolutional layer with a step size of 2.

[0094] Step S202: Convert the first feature map set into a second feature map set using corresponding filtering weights, wherein the second feature map set includes feature maps of different scales.

[0095] In some embodiments, the second feature map set may include feature maps of multiple different scales. Taking the second feature map set including feature maps of three different scales, namely, the fourth feature map, the fifth feature map and the sixth scale feature map, as an example, the specific implementation process of converting the first feature map set into the second feature map set using the corresponding filter weights may include: performing convolution processing on the first feature map included in the first feature map set using the first filter weight to obtain the fourth feature map in the second feature map set; performing convolution processing on the second feature map included in the first feature map set using the second filter weight to obtain the fifth feature map in the second feature map set; performing convolution processing on the third feature map included in the first feature map set using the third filter weight to obtain the sixth feature map in the second feature map set. The convolution processing on the first scale feature, the second feature map and the third feature map may be a dynamic convolution processing. The fourth feature map is represented by R1, the fifth feature map is represented by R2, and the sixth feature map is represented by R3. The scale spaces corresponding to the first feature map, the second feature map and the third feature map are different, and the scale spaces corresponding to the fourth feature map, the fifth feature map and the sixth feature map are different.

[0096] In the embodiment of the present application, the target condition vector is used to guide the generation of a second feature map set including feature maps of different scales, so as to realize feature extraction of different scales of the initial image.

[0097] In the embodiment of the present application, before executing step S101, the method further includes: determining a filtering weight.

[0098] In some embodiments, an optional process for determining filter weights is as follows: Figure 4 As shown, at least the following steps are included:

[0099] Step S301 : Processing the initial image to obtain a target condition vector, wherein the target condition vector can represent degradation feature information of the initial image.

[0100] In some embodiments, during the formation, transmission and recording of the initial image, the quality of the image may be degraded due to various reasons, such as image blur, distortion, noise, etc. The image with degraded quality may be called a degraded image. Each degraded image has a corresponding degradation kernel, and the process of determining the degradation kernel of the image is the same as in the prior art, which will not be described in detail here.

[0101] In the embodiment of the present application, when determining the target condition vector, the degradation kernel corresponding to the initial image may be determined first, and the degradation kernel corresponding to the initial image may be encoded into the target condition vector. The condition vector may be a one-dimensional array containing multiple channels.

[0102] During the specific implementation, the specific implementation process of the target condition vector is determined, such as Figure 5 As shown, the initial image is input to four cascaded convolutional layers, and the output results of the four cascaded convolutional layers are then processed by the global average pooling operator in the pooling layer to obtain the target conditional vector.

[0103] Step S302: extracting a third feature map set of the initial image using the target condition vector and the initial image.

[0104] In some embodiments, the process of extracting the third feature map set of the initial image is as follows: Figure 6 As shown, first, the target condition vector is converted into a first target condition vector, a second target condition vector and a third target condition vector; the first target condition vector is used to perform convolution and spatial feature conversion processing on the initial image to extract a first scale space feature map; the second target condition vector is used to perform convolution and spatial feature conversion processing on the first scale space feature map to extract a second scale space feature map; the third target condition vector is used to perform convolution and spatial feature conversion processing on the second scale space feature map to extract a third scale space feature map. The third feature map set may include at least the first scale space feature map, the second scale space feature map and the third scale space feature map. The first scale space feature map is represented by H1, the second scale space feature map is represented by H2, and the third scale space feature map is represented by H3.

[0105] Among them, the first target condition vector has the same shape as the first scale-space feature map, the second target condition vector has the same shape as the second scale-space feature map, and the third target condition vector has the same shape as the third scale-space feature map.

[0106] In the embodiment of the present application, the first target condition vector and the first scale space feature map may refer to: the length of the feature map corresponding to the first target condition vector is the same as the length of the first scale space feature map, the width of the feature map corresponding to the first target condition vector is the same as the width of the first scale space feature map, and the number of channels of the feature map corresponding to the first target condition vector may be the same as or different from the number of channels of the first scale space feature map. The second target condition vector and the second scale space feature map may refer to: the length of the feature map corresponding to the second target condition vector is the same as the length of the second scale space feature map, the width of the feature map corresponding to the second target condition vector is the same as the width of the second scale space feature map, and the number of channels of the feature map corresponding to the second target condition vector may be the same as or different from the number of channels of the second scale space feature map. The third target condition vector and the third scale space feature map may refer to: the length of the feature map corresponding to the third target condition vector is the same as the length of the third scale space feature map, the width of the feature map corresponding to the third target condition vector is the same as the width of the third scale space feature map, and the number of channels of the feature map corresponding to the third target condition vector may be the same as or different from the number of channels of the third scale space feature map.

[0107] In some embodiments, the specific implementation process of extracting the third feature map set of the initial image using the target condition vector and the initial image includes: converting the target condition vector into a ninth feature map having the same shape as the input feature map of different scale spaces of the initial image; fusing the different ninth feature maps with the input feature maps of the corresponding scale space into a tenth feature map; performing convolution processing on the different tenth feature maps to obtain corresponding feature tensors; and performing spatial feature conversion processing on the input feature map of the corresponding scale space using the feature tensor to obtain a third feature map set including multi-scale spatial feature maps.

[0108] Among them, different ninth feature maps may include: a ninth feature map corresponding to the first target condition vector, a ninth feature map corresponding to the second target condition vector, and a ninth feature map corresponding to the third target condition vector.

[0109] Correspondingly, the specific process of fusing different ninth feature maps with the input feature maps of the corresponding scale space into the tenth feature map includes: 1) fusing the ninth feature map corresponding to the first target conditional vector with the initial feature map to obtain the first tenth feature map corresponding to the scale of the initial feature map; performing convolution processing on the first tenth feature map to obtain the corresponding feature tensor, and using the feature tensor to perform spatial feature conversion processing on the initial feature map to obtain the first scale space feature map. 2) fusing the ninth feature map corresponding to the second target conditional vector with the first scale space feature map after tiled stretching processing to obtain the corresponding second tenth feature map; performing convolution processing on the second tenth feature map to obtain the corresponding feature tensor, and using the feature tensor to perform spatial feature conversion processing on the first scale space feature map after tiled stretching processing to obtain the second scale space feature map. 3) Fusing the ninth feature map corresponding to the third target conditional vector with the second-scale spatial feature map that has been tiled and stretched to obtain the corresponding third and tenth feature maps; performing convolution processing on the third and tenth feature maps to obtain corresponding feature tensors, and using the feature tensor to perform spatial feature conversion processing on the second-scale spatial feature map that has been tiled and stretched to obtain a third-scale spatial feature map.

[0110] The first scale space feature map, the second scale space feature map and the third scale space feature map constitute a third feature map set. In the embodiment of the present application, the tiling and stretching processing of the first scale space feature map and the second scale space feature map can be achieved by a convolution layer with a step size of 2.

[0111] In a specific implementation, the process of determining the processing of the first scale space feature map in the third feature map set can be as follows: Figure 7 As shown, the target condition vector is stretched into a first target condition vector with the same shape as the initial feature map of the initial image, and the first target condition vector is converted into a first ninth feature map corresponding to the initial feature map; the first ninth feature map is fused with the initial feature map to obtain a first tenth feature map (it can also be called connecting the feature map obtained by tiling and stretching the target condition vector with the initial feature map to obtain the first tenth feature map); the first tenth feature map is convolved to obtain a feature tensor; the first tenth feature map is spatially converted using the feature tensor to obtain the first-scale spatial feature map.

[0112] The process of performing convolution processing on the first tenth feature map to obtain the feature tensor can be as follows: Figure 7 The two independent sets of consecutive convolutional layers shown are implemented. The feature tensor can be represented as (γ, β).

[0113] The process of determining the first scale-space feature map based on the feature tensor and the first target condition vector may be as follows: Figure 8 shown. Figure 8 The first spatial feature transformation module in is used to implement Figure 7 The processing process of the first scale spatial feature map in the third feature map set is determined in the embodiment; the feature tensor input by the first spatial feature conversion module is processed by the first convolution layer, and the convolution result and the first target condition vector are input to the second spatial feature conversion module. After the second spatial feature conversion module performs the same operation as the first spatial feature conversion module, the output result is processed by the second convolution layer to output the first scale spatial feature map.

[0114] The above is explained by taking the acquisition of the first scale space feature map as an example. In the specific implementation, the process of acquiring the second scale space feature map and the process of acquiring the third scale space feature map are similar to the process of acquiring the first space scale feature map. The difference is that when acquiring the second scale space feature map, the target condition vector is stretched to a second target condition vector with the same shape as the first scale space feature map; when acquiring the third scale space feature map, the target condition vector is stretched to a third target condition vector with the same shape as the second scale space feature map. As an example, the first scale space feature map can be represented by H1, which is a feature map of 32 channels; the second scale space feature map can be represented by H2, which is a feature map of 64 channels; the third scale space feature map can be represented by H3, which is a feature map of 128 channels.

[0115] Step S303: performing convolution and residual processing on each feature map in the third feature map set to obtain filtering weights in different scale spaces.

[0116] In some embodiments, each feature map in the third feature map set may be subjected to a first convolution process, two residual processes, and a second convolution process in sequence to obtain a first filter weight F1, a second filter weight F2, and a third filter weight F3 corresponding to each feature map. The first convolution process may be performed using a 3*3 convolution layer, and the second convolution process may be performed using a 1*1 convolution layer.

[0117] In the embodiment of the present application, the filtering weights of different scale spaces can cope with the distortion of local changes in the image space and improve the accuracy of feature extraction from the image.

[0118] Step S203, decoding the feature maps in the second feature map set in sequence to obtain the target residual image.

[0119] In some embodiments, the specific process of sequentially decoding the feature maps in the second feature map set to obtain the target residual image is as follows: Fig. 9As shown, it may include at least:

[0120] Step S401, decoding the sixth feature map in the second feature map set to obtain a seventh feature map.

[0121] In some embodiments, the sixth feature map can be convolved, and then the result of the convolution can be reorganized using a pixel reorganization operator to decode the sixth feature map and obtain a seventh feature map; wherein the seventh feature map has the same scale space as the fifth feature map.

[0122] Step S402, the seventh feature map and the fifth feature map are accumulated and then decoded to obtain an eighth feature map.

[0123] In some embodiments, an adder may be used to perform accumulation processing on the seventh and the fifth feature maps according to pixels to obtain an accumulated feature map, and then the accumulated feature map is decoded to obtain an eighth feature map.

[0124] In some embodiments, convolution processing may be performed on the feature map after the accumulation processing, and then pixel reorganization processing may be performed on the result obtained by the convolution processing using a pixel reorganization operator to obtain an eighth feature map.

[0125] Step S403, performing residual processing after accumulating the eighth feature map and the fourth feature map to obtain a target residual image.

[0126] In some embodiments, an adder may be used to perform accumulation processing on the eighth feature map and the fourth feature map according to pixels to obtain an accumulated feature map, and then residual processing is performed on the accumulated feature map to obtain a target residual image.

[0127] In some embodiments, two consecutive residual processings may be performed on the feature map after the accumulation processing. As an example, the feature map after the accumulation processing is subjected to the first residual processing to obtain a residual processing result; and the residual processing result is subjected to the second residual processing to obtain a target residual image.

[0128] Step S102, performing target processing on the initial image and the target residual image to obtain a target output image.

[0129] In some embodiments, the initial image and the target residual image may be superimposed and / or fused to obtain a target output image.

[0130] The image processing method provided in the embodiment of the present application only needs to input the initial image, and does not need to obtain the PSF information corresponding to the acquisition device of the initial image, etc., so as to eliminate the glare problem in the original image and obtain a clear image corresponding to the original image.

[0131] One application scenario of the image processing method provided in an embodiment of the present application is to eliminate the glare problem of images captured by the under-screen camera of a smart electronic device. The under-screen camera can capture images under the screen of a smart electronic device to provide a user experience of viewing images in full screen on a smart electronic device. The under-screen camera can be applied to many scenarios, including video conferencing. In the under-screen camera system, the camera is placed under a translucent organic light-emitting diode (OLED) screen and is closely connected to the OLED screen. The OLED screen pixel array microstructure allows light to pass through the gaps between screen pixels, but this causes the light incident from the object to the sensor to attenuate and diffract, resulting in glare problems in the images captured by the camera.

[0132] The image processing process of the image to be processed collected by the under-screen camera, such as Fig.10 As shown, the image to be processed collected by the under-screen camera is input into the conditional estimation branch to obtain the target conditional vector output by the conditional estimation score module. The target conditional vector and the image to be processed are input into the filter generation branch to obtain the first scale spatial feature map H1, the second scale spatial feature map H2 and the third scale spatial feature map H3 in the third feature map set. Among them, the structure of the spatial feature conversion residual module in the filter generation branch can be Figure 8 The structure shown in Figure 1. The spatial feature conversion residual module contains a skip connection and a module composed of two spatial feature conversion modules and two convolutional layers alternately cascaded. The spatial feature conversion residual module guides the calculation of local features of the image to be processed by combining the target condition vector, and has the ability to refine and transform intermediate feature maps.

[0133] The first scale space feature map H1, the second scale space feature map H2 and the third scale space feature map H3 in the third feature map set are respectively input into the corresponding filter generation module to obtain the first filter weight F1, the second filter weight F2 and the third filter weight F3 of the three different scale spaces; the filter generation module is used to implement the processing process of the above step S1013. The image to be processed is input into the restoration branch to obtain the first feature map E1, the second feature map E2 and the third feature map E3 in the first feature map set. The first feature map E1 is convolved with the first filter weight F1 to obtain the fourth feature map R1; the second feature map E2 is convolved with the second filter weight F2 to obtain the fifth feature map R2, and the third feature map E3 is convolved with the third filter weight F3 to obtain the sixth feature map R3.

[0134] Use the repair module in the repair branch to execute Figure 5The operation flow shown determines the target residual image based on the fourth feature map R1, the fifth feature map R2 and the sixth feature map R3.

[0135] Finally, the target residual image is fused and / or superimposed with the image to be processed to obtain the target output image.

[0136] According to the above description of the image processing method provided in the embodiment of the present application, it can be determined that the architecture diagram of the image processing method provided in the embodiment of the present application is as follows: Fig.11 As shown, after the input image is processed by the conditional estimation branch, a target condition vector is obtained, the filter generation branch generates filter weights based on the input image and the target condition vector, and the repair branch obtains the output image based on the filter weights and the input image. Fig.12 The architecture of the image processing method in the related art shown Figure 1 and Fig.13 The architecture of the image processing method in the related art shown Figure 2 In contrast, a clear output image can be obtained without specifying the point spread function.

[0137] The present application also provides an image processing device, the composition structure diagram of the image processing device is as follows: Fig.14 As shown, including:

[0138] A determination module 501 is used to process a multi-scale feature map obtained by encoding the initial image using different filtering weights after obtaining the initial image, so as to obtain a target residual image;

[0139] The processing module 502 is used to perform target processing on the initial image and the target residual image to obtain a target output image.

[0140] In some embodiments, the determining module 501 is used to encode the initial image into a first feature map set, wherein the first feature map set includes feature maps of different scales;

[0141] Converting the first feature map set into a second feature map set using corresponding filter weights, wherein the second feature map set includes feature maps of different scales;

[0142] The feature maps in the second feature map set are decoded in sequence to obtain the target residual image.

[0143] In some embodiments, the determining module 501 is used to perform convolution processing on the first feature map included in the first feature map set using the first filtering weight to obtain a fourth feature map in the second feature map set;

[0144] Performing convolution processing on the second feature map included in the first feature map set using a second filtering weight to obtain a fifth feature map in the second feature map set;

[0145] A third feature map included in the first feature map set is convolved using a third filtering weight to obtain a sixth feature map in the second feature map set.

[0146] In some embodiments, the determining module 501 is used to decode the sixth feature map to obtain a decoded sixth feature map;

[0147] Accumulate the decoded sixth feature map and the fifth feature map to obtain an accumulated fifth feature map;

[0148] Decoding the fifth feature map after the accumulation process to obtain a decoded fifth feature map;

[0149] Accumulate the decoded fifth feature map and the fourth feature map to obtain an accumulated fourth feature map;

[0150] Perform residual processing on the fourth feature map after the accumulation processing to obtain the target residual image.

[0151] In some embodiments, the processing module 502 is used to process the initial image to obtain a target condition vector, where the target condition vector can represent degradation feature information of the initial image;

[0152] Extracting a third feature map set of the initial image using the target condition vector and the initial image;

[0153] Convolution and residual processing are performed on each feature map in the third feature map set to obtain filter weights in different scale spaces.

[0154] In some embodiments, the processing module 502 is used to extract degradation features of the initial image;

[0155] The target condition vector is determined based on the degradation characteristic.

[0156] In some embodiments, the processing module 502 is used to perform convolution and spatial feature conversion processing on the initial image using the first target condition vector to extract a first scale-space feature map, and the first target condition vector has the same shape as the first scale-space feature map;

[0157] Using a second target condition vector to perform convolution and spatial feature conversion processing on the first scale space feature map to extract a second scale space feature map, wherein the second target condition vector has the same shape as the second scale space feature map;

[0158] Using a third target condition vector to perform convolution and spatial feature conversion processing on the second scale space feature map to extract a third scale space feature map, wherein the third target condition vector has the same shape as the third scale space feature map;

[0159] The first target condition vector, the second target condition vector and the third target condition vector are obtained based on converting the target condition vector.

[0160] In some embodiments, the processing module 502 is used to convert the target condition vector into a ninth feature map having the same shape as the input feature map of the different scale spaces of the initial image;

[0161] The different ninth feature maps are merged with the input feature maps of the corresponding scale space into the tenth feature map;

[0162] Perform convolution processing on different tenth feature maps to obtain corresponding feature tensors;

[0163] The feature tensor is used to perform spatial feature conversion processing on the input feature map of the corresponding scale space to obtain a third feature map set including multi-scale spatial feature maps.

[0164] The application embodiment provides an electronic device, including: at least one processor; and

[0165] a memory communicatively connected to the at least one processor; wherein,

[0166] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned image processing method.

[0167] Fig.15 700 is a schematic diagram of the hardware structure of an electronic device of an embodiment of the present application. The electronic device 700 includes: at least one processor 701, a memory 702 and at least one network interface 704. The various components in the electronic device 700 are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 705.

[0168] It can be understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), ferromagnetic random access memory, flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory 702 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

[0169] The memory 702 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 700. Examples of such data include: any computer program for operating on the electronic device 700, such as the instruction set 7022. Instructions for implementing the image processing method in the embodiment of the present application may be included in the instruction set 7022.

[0170] The image processing method disclosed in the above embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the image processing method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above image processing method in combination with its hardware.

[0171] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, MPUs, or other electronic components to execute the aforementioned image processing method.

[0172] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0173] The embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the electronic device management method described in the embodiment of the present application.

[0174] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the image processing method provided by the embodiment of the present application.

[0175] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0176] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0177] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0178] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0179] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, include: After obtaining the initial image, a multi-scale feature map obtained by encoding the initial image is processed using different filtering weights to obtain a target residual image; Target processing is performed on the initial image and the target residual image to obtain a target output image.

2. The method according to claim 1, in, The method of processing the multi-scale feature map obtained by encoding the initial image using different filtering weights to obtain a target residual image includes: Encoding the initial image into a first feature map set, wherein the first feature map set includes feature maps of different scales; Converting the first feature map set into a second feature map set using corresponding filter weights, wherein the second feature map set includes feature maps of different scales; The feature maps in the second feature map set are decoded in sequence to obtain the target residual image.

3. The method according to claim 2, in, The converting the first feature map set into a second feature map set by using corresponding filtering weights includes: Performing convolution processing on a first feature map included in the first feature map set using a first filtering weight to obtain a fourth feature map in the second feature map set; Performing convolution processing on the second feature map included in the first feature map set using a second filtering weight to obtain a fifth feature map in the second feature map set; Performing convolution processing on the third feature map included in the first feature map set using a third filtering weight to obtain a sixth feature map in the second feature map set; Among them, the first feature map, the second feature map and the third feature map correspond to different scale spaces, and the fourth feature map, the fifth feature map and the sixth feature map correspond to different scale spaces.

4. The method according to claim 3, in, The decoding process is performed on the feature maps in the second feature map set in sequence to obtain the target residual image, including: Decoding the sixth feature map to obtain a seventh feature map, where the seventh feature map has the same scale space as the fifth feature map; Accumulating the seventh feature map and the fifth feature map and performing decoding processing to obtain an eighth feature map, wherein the eighth feature map has the same scale space as the fourth feature map; The eighth feature map and the fourth feature map are accumulated and then residual processing is performed to obtain the target residual image.

5. The method according to any one of claims 1 to 4, further comprising: include: Processing the initial image to obtain a target condition vector, wherein the target condition vector can represent degradation feature information of the initial image; Extracting a third feature map set of the initial image using the target condition vector and the initial image; Convolution and residual processing are performed on each feature map in the third feature map set to obtain filter weights in different scale spaces.

6. The method according to claim 5, in, The processing of the initial image to obtain a target condition vector includes: extracting degradation features of the initial image; The target condition vector is determined based on the degradation characteristic.

7. The method according to claim 5, in, The step of extracting a third feature map set of the initial image by using the target condition vector and the initial image includes: Performing convolution and spatial feature conversion processing on the initial image using a first target condition vector to extract a first scale-space feature map, wherein the first target condition vector has the same shape as the first scale-space feature map; Using a second target condition vector to perform convolution and spatial feature conversion processing on the first scale space feature map to extract a second scale space feature map, wherein the second target condition vector has the same shape as the second scale space feature map; Using a third target condition vector to perform convolution and spatial feature conversion processing on the second scale space feature map to extract a third scale space feature map, wherein the third target condition vector has the same shape as the third scale space feature map; The first target condition vector, the second target condition vector and the third target condition vector are obtained based on converting the target condition vector.

8. The method according to claim 5 or 7, in, The step of extracting a third feature map set of the initial image by using the target condition vector and the initial image includes: Converting the target conditional vector into a ninth feature map having the same shape as the input feature map of the different scale spaces of the initial image; The different ninth feature maps are merged with the input feature maps of the corresponding scale space into the tenth feature map; Perform convolution processing on different tenth feature maps to obtain corresponding feature tensors; The feature tensor is used to perform spatial feature conversion processing on the input feature map of the corresponding scale space to obtain a third feature map set including multi-scale spatial feature maps.

9. An image processing device, include: A determination module, configured to, after obtaining an initial image, process a multi-scale feature map obtained by encoding the initial image using different filtering weights to obtain a target residual image; The processing module is used to perform target processing on the initial image and the target residual image to obtain a target output image.

10. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores an instruction set that can be executed by the at least one processor, and the instruction set is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images