Image processing method and device thereof, electronic device and storage medium

Abstract

The application discloses an image processing method and device, an electronic device and a storage medium. The image processing method comprises the following steps: acquiring an input image and a reference image; performing feature extraction on the input image to obtain target input features; performing feature extraction on the reference image to obtain target reference features and reference weight information; fusing the target input features and the target reference features to obtain fused features; and processing the fused features according to the reference weight information to obtain a target image. The image processing method and device, the electronic device and the storage medium can perform feature extraction on the input image and the reference image, further adopt color correction to obtain reference weight information, fuse the obtained target input features and target reference information to obtain fused features, and process the fused features according to the reference weight information to obtain a target image, so that the coloring of the image is more accurate and natural.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method and device, an electronic device and a storage medium. BACKGROUND

[0002] At present, most coloring algorithms have some unnatural coloring situations. More and more coloring algorithms consider adding reference images to the network to guide the coloring process of the image to be colored, but there are still some inaccurate coloring situations. SUMMARY

[0003] Therefore, the present application aims to at least partially solve one of the problems in the related art. To this end, the present application aims to provide an image processing method and device, an electronic device and a storage medium.

[0004] The present application provides an image processing method. The image processing method comprises: acquiring an input image and a reference image; performing feature extraction on the input image to obtain target input features; performing feature extraction on the reference image to obtain target reference features and reference weight information; fusing the target input features and the target reference features to obtain fused features; and processing the fused features according to the reference weight information to obtain a target image.

[0005] In this way, the image processing method and device of the present application perform feature extraction on the input image and the reference image and further obtain reference weight information by color correction, fuse the obtained target input features and target reference information to obtain fused features, and process the fused features according to the reference weight information to obtain a target image, which can make the colored image more accurate and natural.

[0006] In some embodiments, the fusing the target input features and the target reference features to obtain fused features comprises: sequentially performing concatenation processing and convolution processing on the target input features and the target reference features to obtain first concatenated features; sequentially performing attention mechanism processing and convolution processing on the first concatenated features to obtain second concatenated features; and processing the first concatenated features and the second concatenated features to obtain the fused features.

[0007] In this way, the present application fuses the target input features and the target reference features through concatenation processing, attention mechanism processing and convolution processing to obtain fused features, which can better fuse the reference image and the input image and make the colored input image more accurate and natural.

[0008] In some embodiments, the feature extraction on the input image obtains a target input feature, including: sequentially performing convolution processing and attention mechanism processing on the input image to obtain a first intermediate input feature; and performing N times of looped sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate input feature to obtain the target input feature, where N is an integer and N>0.

[0009] In this way, the target input feature is obtained by sequentially performing convolution processing, attention mechanism processing and attention mechanism processing on the input image, which can reduce the negative effects caused by different textures of the reference image and the input image.

[0010] In some embodiments, the downsampling processing includes bicubic interpolation downsampling processing or maximum pooling processing.

[0011] In this way, the input image is downsampling processed by using the bicubic interpolation downsampling method, i.e., the input image is reduced, and a thumbnail corresponding to the input image can be generated.

[0012] In some embodiments, the N times of looped sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate input feature to obtain the target input feature includes: i times of looped sequentially performing downsampling processing, convolution processing and attention mechanism processing on the i-th intermediate input feature to obtain an (i+1)-th intermediate input feature, where i is an integer and 0

[0013] In this way, the final target input feature is obtained by performing N times of looped downsampling processing, convolution processing and attention mechanism processing on the intermediate input feature, which can ensure that the texture of the target input feature is clearer.

[0014] In some embodiments, the feature extraction on the reference image obtains a target reference feature and reference weight information, including: sequentially performing convolution processing and attention mechanism processing on the reference image to obtain a first intermediate reference feature; and performing N times of looped sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate reference feature to obtain the target reference feature, where N is an integer and N>0.

[0015] In this way, the first intermediate reference feature is obtained by sequentially performing convolution processing and attention mechanism processing on the reference image, and the target reference feature is obtained by performing N times of looped sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate reference feature, which can ensure that the texture of the target reference feature is clearer and can reduce the negative effects caused by different textures of the reference image and the input image.

[0016] In some embodiments, the down-sampling processing, the convolution processing and the attention mechanism processing are sequentially performed on the jth intermediate reference feature to obtain a (j+1)th intermediate reference feature, j is an integer, and 0

[0017] Thus, the intermediate reference features are subjected to the cyclic processing of the N times of down-sampling processing, convolution processing and attention mechanism processing to obtain the final target reference feature, which can ensure that the texture of the target reference feature is clearer and the negative effects caused by the different textures of the reference image and the input image are reduced.

[0018] In some embodiments, the feature extraction on the reference image to obtain the target reference feature and the reference weight information further comprises: sequentially performing global pooling processing and convolution processing on the kth intermediate reference feature to obtain the kth reference weight information, j is an integer, and 0

[0019] Thus, the global pooling processing is performed on the kth intermediate reference feature, and the convolution processing is sequentially performed on the kth intermediate reference feature, which can reduce the dimension of the feature map of the reference image, and the data of the subsequently output target reference feature can be multiplied.

[0020] In some embodiments, the processing of the fusion feature according to the reference weight information to obtain the target image comprises: sequentially performing convolution processing and attention mechanism processing on the fusion feature to obtain a 1st intermediate output feature; sequentially performing up-sampling processing, convolution processing and attention mechanism processing on m intermediate output features according to the mth weight information to obtain a (m+1)th intermediate output feature, m is an integer, and 0

[0021] Thus, the intermediate output feature is obtained by twice convolution processing of the fusion feature according to the reference weight information, which further corrects the coloring guidance of the fusion feature, so that the obtained target image is more accurate and natural.

[0022] In some embodiments, the sampling factor of the up-sampling processing and the sampling factor of the down-sampling processing are adapted to each other.

[0023] Thus, the sampling factor of the up-sampling processing and the sampling factor of the down-sampling processing are adapted to each other, so as to ensure the size consistency of the input image and the output target image in the image processing method.

[0024] In some embodiments, the up-sampling processing includes bicubic interpolation up-sampling processing or pixel reorganization processing.

[0025] Thus, the up-sampling processing is performed by the bicubic interpolation up-sampling processing mode or the pixel reorganization processing (Pixel-Shuffle) mode, so as to lay a foundation for the output target image.

[0026] In some embodiments, the attention mechanism processing includes spatial attention mechanism processing, which is realized by the following conditional expression:

[0027]

[0028] wherein, F b represents the feature output by the bth spatial attention mechanism processing, F b-1 represents the feature input for the bth spatial attention mechanism processing, b b-1 wherein, W b n represents the nth convolution parameter in the bth spatial attention mechanism processing, n = 1, 2, 3, 4, H b represents the height of the bth spatial attention mechanism processing, W b n represents the width of the bth spatial attention mechanism processing, and F b represents the feature output by the bth spatial attention mechanism processing. UP

[0029] Thus, the image processing method and the device thereof introduce the attention mechanism, so as to give different weights to different information, and lay a foundation for processing and fusing the features to obtain the target image according to the reference weight information.

[0030] The application further provides an image processing device. The image processing device includes an input feature extraction module, a reference feature extraction module, a correction module, a fusion module and an output module. The input feature extraction module is used to acquire an input image, and perform feature extraction on the input image to obtain a target input feature; the reference feature extraction module is used to acquire a reference image, and perform feature extraction on the reference image to obtain a target reference feature; the correction module is used to determine reference weight information according to the reference image; the fusion module is used to fuse the input feature and the reference feature to obtain a fusion feature; and the output module is used to process the fusion feature according to the reference weight information to obtain a target image.

[0031] ​​​The image processing apparatus of the present application extracts features from input images and reference images, and further obtains reference weight information by color correction, fuses target input features and target reference information to obtain fused features, and processes the fused features according to the reference weight information to obtain a target image, so that the colored image is more accurate and natural.

[0032] The present application also provides an electronic device. The electronic device comprises the image processing apparatus described in the above embodiments, and the image processing apparatus is used to implement the image processing method described in any one of the above embodiments.

[0033] The electronic device of the present application extracts features from input images and reference images by using the above image processing method and apparatus, and further obtains reference weight information by color correction, fuses target input features and target reference information to obtain fused features, and processes the fused features according to the reference weight information to obtain a target image, so that the colored image is more accurate and natural.

[0034] The present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the image processing method described in any one of the above embodiments is implemented.

[0035] The computer readable storage medium of the present application extracts features from input images and reference images by using the above image processing method and apparatus, and further obtains reference weight information by color correction, fuses target input features and target reference information to obtain fused features, and processes the fused features according to the reference weight information to obtain a target image, so that the colored image is more accurate and natural.

[0036] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0037] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0038] Figure 1 is a flowchart of an image processing method of some embodiments of the present application;

[0039] Figure 2 is a structural schematic diagram of an image processing apparatus of some embodiments of the present application;

[0040] Figure 3 is a flowchart of an image processing method of some embodiments of the present application;

[0041] Figure 4 is a structural schematic diagram of a fusion module in an image processing apparatus of some embodiments of the present application;

[0042] Figure 5 is a flowchart of an image processing method of some embodiments of the present application;

[0043] Figure 6 is a structural schematic diagram of the inside of an input feature extraction module or a reference feature extraction module of some embodiments of the present application;

[0044] Figure 7 is a flowchart of an image processing method of some embodiments of the present application;

[0045] Figure 8 is a flowchart of an image processing method of some embodiments of the present application;

[0046] Figure 9 is a flowchart of an image processing method of some embodiments of the present application;

[0047] Figure 10 is a flowchart of an image processing method of some embodiments of the present application;

[0048] Figure 11 is a structural schematic diagram of a rectification module in an image processing apparatus of some embodiments of the present application;

[0049] Figure 12 is a flowchart of an image processing method of some embodiments of the present application;

[0050] Figure 13 is a structural schematic diagram of the inside of an output module in an image processing apparatus of some embodiments of the present application;

[0051] Figure 14 is a structural schematic diagram of the inside of a spatial attention mechanism in an image processing apparatus of some embodiments of the present application. DETAILED DESCRIPTION

[0052] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout the drawings denote the same or similar elements or elements having the same or similar functions. The embodiments described below by reference to the drawings are exemplary only, and are used only for the purpose of explanation of the present application, and are not to be understood as limiting the present application.

[0053] In the description of the application, it needs to be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can be explicitly or implicitly included one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically defined.

[0054] In the description of the application, it needs to be explained that, unless otherwise specifically defined and limited, the terms "mounting", "connecting", "connecting" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; It can be mechanically connected, or it can be electrically connected or can communicate with each other; It can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0055] The following disclosure provides many different embodiments or examples for implementing different structures of the application. In order to simplify the disclosure of the application, the components and arrangements of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the application. In addition, the application can repeatedly refer to numbers and / or letters in different examples, and such repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or arrangements discussed.

[0056] The embodiments of the application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary, only for explaining the application, and cannot be understood as limiting the application.

[0057] Most of the current coloring methods use automatic coloring schemes, and automatic coloring schemes often appear inaccurate coloring in some real photo coloring. Among them, some schemes are automatic coloring schemes using reference images, and the automatic coloring scheme using reference images is specifically to directly infer and automatically color according to the color of the reference image under the guidance of the reference image. This automatic coloring scheme using reference images still cannot avoid situations such as color boundary overflow.

[0058] In view of this, please refer to Figure 1 The application provides an image processing method. The image processing method comprises:

[0059] 01: obtaining an input image and a reference image;

[0060] 03: performing feature extraction on the input image to obtain target input features;

[0061] 05: performing feature extraction on the reference image to obtain target reference features and reference weight information;

[0062] 07: fusing the target input features and the target reference features to obtain fused features;

[0063] 09: processing the fused features according to the reference weight information to obtain a target image.

[0064] Referring to Figure 2 The application further provides an image processing device 100. The image processing device 100 comprises an input feature extraction module 110, a reference feature extraction module 120, a correction module 130, a fusion module 140 and an output module 150. The input feature extraction module 110 is configured to acquire an input image and perform feature extraction on the input image to obtain target input features. The reference feature extraction module 120 is configured to acquire a reference image and perform feature extraction on the reference image to obtain target reference features. The correction module 130 is configured to determine reference weight information according to the reference image. The fusion module 140 is configured to fuse the target input features and the target reference features to obtain fused features. The output module 150 is configured to process the fused features according to the reference weight information to obtain a target image.

[0065] Specifically, referring to Figure 2 First, an input image and a reference image are acquired. The input image comprises a full black-and-white image or a partial black-and-white image, i.e., the input image is a to-be-colored image. That is to say, the application can realize coloring processing of a full black-and-white image or a partial black-and-white image. The reference image is a full-color image corresponding to the input image.

[0066] Next, feature extraction is performed on the input image to obtain target input features. That is, the input image is input to the input feature extraction module 110 for feature extraction to obtain target input features. The target input features may, for example, be a certain black-and-white image region or a plurality of black-and-white image regions in the input image.

[0067] Feature extraction is performed on the reference image to obtain target reference features and reference weight information. That is, while performing feature extraction on the input image, the reference image is also input to the input feature extraction module 120 for feature extraction to obtain target reference features, and the reference image is simultaneously input to the correction module 130 to determine the reference weight information. The target reference features are reference images corresponding to the target input features. The reference image may, for example, be a target color image required by a user corresponding to the input image.

[0068] It can be understood that the application can assign different weights to different reference information to obtain reference weight information. For example, more local information can be assigned a greater weight for image details, and less smooth areas can be assigned a smaller weight, so that the target image coloring obtained according to the reference weight information is more natural.

[0069] In addition, the input feature extraction module 110 and the reference feature extraction module 120 of the application use a hierarchical downsampling combined with a spatial attention mechanism (SAM) to extract features from the input image and the reference image, which can reduce the negative effects caused by the different textures of the reference image and the input image.

[0070] Finally, the input feature and the reference feature are fused to obtain the fusion feature, and the fusion feature is processed according to the reference weight information to obtain the target image. As shown in Figure 2 After obtaining the input feature and the reference feature, the input feature and the reference feature can be input into the fusion module 140 together to obtain the fusion feature, and then the fusion feature is input into the output module 150, and the output module 150 processes the fusion feature according to the reference weight information to obtain the target image.

[0071] Thus, the image processing method and device of the application extract features from the input image and the reference image and further obtain the reference weight information by color correction, fuse the target input feature and the target reference information to obtain the fusion feature, and process the fusion feature according to the reference weight information to obtain the target image, which can make the coloring of the image more accurate and natural.

[0072] For how to obtain the fusion feature, please refer to Figure 3 , step 07 includes:

[0073] 071: sequentially coupling and convolving the target input feature and the target reference feature to obtain a first coupled feature;

[0074] 072: sequentially performing attention mechanism processing and convolution processing on the first coupled feature to obtain a second coupled feature;

[0075] 073: processing the first coupled feature and the second coupled feature to obtain the fusion feature.

[0076] Please refer to Figure 2, the fusion module 140 is used for sequentially coupling and convolution processing of the target input feature and the target reference feature to obtain a first coupling feature; sequentially performing attention mechanism processing and convolution processing on the first coupling feature to obtain a second coupling feature; and processing the fusion feature according to the first coupling feature and the second coupling feature. The internal structure diagram of the fusion module 140 is shown in Figure 4

[0077] Please refer to Figure 4 , the present application sequentially processes the target input feature and the target reference feature by concatenation (Concat) and convolution (Conv) to obtain a first coupling feature; then, the first coupling feature is sequentially processed by attention mechanism (SAM) and convolution (Conv) to obtain a second coupling feature. Finally, the fusion feature can be processed according to the first coupling feature and the second coupling feature.

[0078] That is, in order to better fuse the reference image and the input image, after the target reference feature extracted from the reference image feature and the target input feature extracted from the input image feature are concatenated (Concat) and processed, the image processing method of the present application further processes the target reference feature and the target input feature by convolution (Conv) - spatial attention (SAM) - convolution (Conv) structure to obtain the fusion feature, that is, a skip connection method is used to process the target reference feature and the target input feature to obtain the fusion feature.

[0079] In this way, the present application processes the target input feature and the target reference feature by concatenation, attention mechanism and convolution to obtain the fusion feature, which can better fuse the reference image and the input image, and make the input image more accurate and natural after coloring.

[0080] For how to obtain the target input feature according to the input image, specifically, please refer to Figure 5 , step 03 includes:

[0081] 031: sequentially performing convolution processing and attention mechanism processing on the input image to obtain a first intermediate input feature;

[0082] 032: N times of loop implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate input feature to obtain the target input feature, N is an integer and N>0.

[0083] Please combine Figure 2 ​, the input feature extraction module 110 is configured to sequentially perform convolution processing and attention mechanism processing on the input image to obtain a first intermediate input feature; and perform N times of loop implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate input feature to obtain a target input feature, where N is an integer and N>0. An internal structure diagram of the input feature extraction module 110 is shown in Figure 6 It should be noted that Figure 6 The structure shown is after twice downsampling (i.e., 4 times downsampling), and other downsampling structures can be used according to actual requirements of image processing by a user, for example, a structure of once downsampling or three times downsampling can be used.

[0084] Referring to Figure 6 , the input image is sequentially subjected to convolution processing and attention mechanism processing to obtain a first intermediate input feature, and N times of loop implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing on the first intermediate input feature is performed to obtain a target input feature, where N is an integer and N>0.

[0085] In this way, the target input feature is obtained by sequentially performing convolution processing, attention mechanism processing and attention mechanism processing on the input image, so that negative effects caused by different textures of the reference image and the input image can be reduced.

[0086] The downsampling processing includes bicubic interpolation downsampling processing or maximum pooling processing (Max-Pooling). The input image can be subjected to bicubic interpolation downsampling processing. The maximum pooling can obtain local information and can better preserve features on the texture. If the specific position of an object in a picture is not observed and only whether the object appears in the picture is concerned, the effect of the maximum pooling processing is better.

[0087] In this way, the input image is subjected to downsampling processing by using the bicubic interpolation downsampling manner, i.e., the input image is subjected to reduction processing, so that a thumbnail corresponding to the input image can be generated.

[0088] Referring to Figure 7 , the step 032 includes:

[0089] 0321: performing i times of loop implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing on the i-th intermediate input feature to obtain an (i+1)-th intermediate input feature, where i is an integer and 0

[0090] 0322: taking the (N+1)-th intermediate input feature as the target input feature.

[0091] Please combine Figure 2 and Figure 6 The input feature extraction module 110 is configured to sequentially perform downsampling processing, convolution processing and attention mechanism processing on the i-th intermediate input feature in the i-th cycle to obtain an (i+1)-th intermediate input feature, i is an integer, and 0

[0092] That is, the present application needs to perform N times of cyclic processing of downsampling processing, convolution processing and attention mechanism processing on the i-th intermediate input feature to obtain an (N+1)-th intermediate input feature, which can be used as the final target input feature.

[0093] In this way, the present application performs cyclic processing of N times of downsampling processing, convolution processing and attention mechanism processing on the intermediate input feature to obtain the final target input feature, which can ensure that the texture of the target input feature is clearer.

[0094] Similarly, while performing cyclic processing of downsampling processing, convolution processing and attention mechanism processing on the input image to obtain the target input feature in the bicubic interpolation downsampling manner, in order to reduce the negative impact caused by the different textures of the reference image and the input image, the present application also needs to perform cyclic processing of downsampling processing, convolution processing and attention mechanism processing on the reference image to obtain a target reference feature.

[0095] For how to obtain the target reference feature from the reference image, specifically, please refer to Figure 8 Step 05 includes:

[0096] 051: sequentially performing convolution processing and attention mechanism processing on the reference image to obtain a 1st intermediate reference feature;

[0097] 052: performing cyclic processing of downsampling processing, convolution processing and attention mechanism processing on the 1st intermediate reference feature for N times to obtain a target reference feature, N is an integer, and N>0.

[0098] Please combine Figure 2 The reference feature extraction module 120 is configured to sequentially perform convolution processing and attention mechanism processing on the reference image to obtain a 1st intermediate reference feature; and perform cyclic processing of downsampling processing, convolution processing and attention mechanism processing on the 1st intermediate reference feature for N times to obtain a target reference feature, N is an integer, and N>0.

[0099] The internal structure of the reference feature extraction module 120 is the same as that of the input feature extraction module 110. That is, the process of extracting features from the reference image is the same as that of extracting features from the input image. In this way, it can be ensured that the texture of the target reference feature obtained by processing the reference image is the same as that of the target input feature obtained by processing the input image.

[0100] In this way, the reference image is also sequentially subjected to convolution processing and attention mechanism processing to obtain the first intermediate reference feature, and then subjected to N times of cyclic implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing to obtain the target reference feature. This can ensure that the texture of the target reference feature is clearer, and can reduce the negative effects caused by the different textures of the reference image and the input image.

[0101] In this way, the reference image is also sequentially subjected to convolution processing and attention mechanism processing to obtain the first intermediate reference feature, and then subjected to N times of cyclic implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing to obtain the target reference feature. This can ensure that the texture of the target reference feature is clearer, and can reduce the negative effects caused by the different textures of the reference image and the input image.

[0101] In this way, the reference image is also sequentially subjected to convolution processing and attention mechanism processing to obtain the first intermediate reference feature, and then subjected to N times of cyclic implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing to obtain the target reference feature. This can ensure that the texture of the target reference feature is clearer, and can reduce the negative effects caused by the different textures of the reference image and the input image.

[0102] Further, referring to Figure 9 , step 052 comprises:

[0103] 0521: performing, on the jth intermediate reference feature, cyclic implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing to obtain the (j+1)th intermediate reference feature, j being an integer and 0

[0104] 0522: taking the (N+1)th intermediate reference feature as the target reference feature.

[0105] Please combine Figure 2 and Figure 6 , the reference feature extraction module 120 is further configured to perform, on the jth intermediate reference feature, cyclic implementation of sequentially performing downsampling processing, convolution processing and attention mechanism processing to obtain the (j+1)th intermediate reference feature, j being an integer and 0

[0106] That is, the process of sequentially performing downsampling processing, convolution processing and attention mechanism processing on the intermediate reference features generated in the process of extracting features from the reference image is the same as the process of sequentially performing downsampling processing, convolution processing and attention mechanism processing on the intermediate input features generated in the process of extracting features from the input image. This can ensure that the texture of the target reference feature obtained by processing the reference image is the same as that of the target input feature obtained by processing the input image.

[0107] Thus, the application performs the cyclic processing of N times downsampling processing, convolution processing and attention mechanism processing on the intermediate reference features to obtain the final target reference feature, which can ensure that the texture of the target reference feature is clearer and reduce the negative impact caused by the different textures of the reference image and the input image.

[0108] In addition, for how to obtain the reference weight information, specifically, please refer to Figure 10 , step 05 further comprises:

[0109] 053: sequentially performing global pooling processing and convolution processing on the kth intermediate reference feature to obtain the kth reference weight information, j is an integer, and 0

[0110] Please refer to Figure 2 , the correction module 130 is configured to sequentially perform global pooling processing and convolution processing on the kth intermediate reference feature to obtain the kth reference weight information, j is an integer, and 0 Figure 11

[0111] Wherein, the purpose of the global pooling (GlobalPooling) processing on the kth intermediate reference feature is to reduce the dimension of the feature map of the reference image, from 3 dimensions to 1 dimension, and output 1 response operation for each feature map.

[0112] The purpose of sequentially performing convolution processing on the kth intermediate reference feature is to control the number of intermediate reference features, so that the data between the subsequent intermediate reference features can be multiplied.

[0113] Wherein, the convolution processing can utilize 1x1 convolution to realize the dimension reduction of the feature map of the reference image. It can be understood that different sizes of convolution kernel mean different sizes of receptive field.

[0114] Thus, the application can reduce the dimension of the feature map of the reference image by performing global pooling (GlobalPooling) processing on the kth intermediate reference feature and sequentially performing convolution processing on the kth intermediate reference feature, so that the data between the subsequent output target reference features can be multiplied.

[0115] For how to process the intermediate output feature to output the target image according to the reference weight information, specifically, please refer to Figure 12 , step 09 comprises:

[0116] 091: sequentially performing convolution processing and attention mechanism processing on the fusion feature to obtain the first intermediate output feature;

[0117] ​092: sequentially performing upsampling processing, convolution processing and attention mechanism processing on the m intermediate output features according to the mth weight information to obtain an (m+1)th intermediate output feature, m is an integer, and 0

[0118] 093: sequentially performing twice convolution processing on the (N+1)th intermediate output feature according to the (N+1)th weight information to obtain the target image.

[0119] Please refer to Figure 2 The output module 150 is configured to sequentially perform convolution processing and attention mechanism processing on the fused features to obtain a 1st intermediate output feature; sequentially perform upsampling processing, convolution processing and attention mechanism processing on the m intermediate output features according to the mth weight information to obtain an (m+1)th intermediate output feature, m is an integer, and 0 Figure 13

[0120] Please refer to Figure 13 The output module 150 of the present application first sequentially performs convolution processing and attention mechanism processing on the fused features to obtain a 1st intermediate output feature, and then sequentially performs upsampling processing (Upsampling), convolution processing and attention mechanism processing on the m intermediate output features according to the mth weight information to obtain an (m+1)th intermediate output feature, m is an integer, and 0

[0121] The output module 150 performs upsampling processing on the fused features, specifically, enlarges the previously reduced input image, so that the output target image can be displayed on a higher resolution display device. The upsampling processing method of the present application includes but is not limited to pixel reorganization processing (Pixel-Shuffle) and bicubic interpolation upsampling processing method.

[0122] The output module 150 and the input feature extraction module 110 and the reference feature extraction module 120 have two different places. The first obvious difference is that the output module 150 has two more convolutions in the last output position. The second difference is that the upsampling method used by the output module 150 and the upsampling method used by the input feature extraction module 110 and the reference feature extraction module 120 can be different, for example, the output module 150 can use the pixel reorganization processing (Pixel-Shuffle) upsampling method, and the input feature extraction module 110 and the reference feature extraction module 120 use the bicubic interpolation upsampling processing upsampling method.

[0123] ​It can be understood that, in order to ensure the size consistency of the input image and the output target image in the image processing method of the present application, the number of upsampling in the output module 150 is the same as the number of downsampling in the input feature extraction module 110 and the reference feature extraction module 120. That is, the sampling factor of the upsampling processing in the output module 150 is adapted to the sampling factor of the downsampling processing in the input feature extraction module 110 and the reference feature extraction module 120.

[0124] Then, please combine Figure 2 After the output module 150 processes the fused features to obtain the intermediate output features, the intermediate output features can also be sequentially subjected to two convolution processes according to the reference weight information determined by the correction module 130 according to the reference image, so as to obtain the target image. That is, the output module 150 of the present application can sequentially subject the N+1 intermediate output features to two convolution processes according to the N+1 weight information to obtain the target image.

[0125] It can be understood that, since the feature extraction module includes an attention mechanism, different weights are given to different information, then the two are fused, and the coloring process is further guided and corrected by the color correction module, so that the final output result can be obtained, and the target image after coloring the image is more accurate and natural.

[0126] Thus, the present application processes the fused features to obtain the intermediate output features according to the reference weight information to obtain the target image by two convolution processes, and further guides and corrects the coloring of the fused feature image, so that the obtained target image is more accurate and natural.

[0127] The above-mentioned attention mechanism can adopt a spatial attention mechanism, the structure of which is as Figure 14 shown. The spatial attention mechanism processing can be realized by the following conditional expression:

[0128]

[0129] wherein F b represents the feature output by the bth spatial attention mechanism processing, F b-1 represents the feature input for the bth spatial attention mechanism processing, represents the nth convolution parameter in the bth spatial attention mechanism processing, n = 1, 2, 3, 4, H UP represents the up-sampling operator in the spatial attention mechanism processing.

[0130] Thus, the image processing method and device of the present application lay a foundation for processing the fused features to obtain the target image according to the reference weight information by introducing the attention mechanism to give different weights to different information.

[0131] In addition, the application further provides an electronic device. The electronic device comprises the image processing apparatus 100 described above, and the image processing apparatus 100 is used to realize the image processing method described in any one of the above embodiments. The electronic device comprises a computer, a pad, a mobile phone and the like.

[0132] The electronic device provided by the application extracts features from the input image and the reference image by using the image processing method and the image processing apparatus described above, and further obtains reference weight information by using color correction. The target input feature and the target reference information are fused to obtain fused features, and the fused features are processed according to the reference weight information to obtain a target image, so that the coloring of the image is more accurate and natural.

[0133] The application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the material information management method described in the above embodiments is realized.

[0134] For example, the computer program is executed by the processor to realize the following steps of the material information management method:

[0135] 01: obtaining an input image and a reference image;

[0136] 03: extracting features from the input image to obtain a target input feature;

[0137] 05: extracting features from the reference image to obtain a target reference feature and reference weight information;

[0138] 07: fusing the target input feature and the target reference feature to obtain fused features;

[0139] 09: processing the fused features according to the reference weight information to obtain a target image.

[0140] It can be understood that the computer program comprises computer program code. The computer program code can be in the form of source code, object code, an executable file or some intermediate form. The computer readable storage medium can comprise any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM) and a software distribution medium, etc.

[0141] The computer readable storage medium of the present application applies the image processing method and the device thereof to feature extraction on the input image and the reference image, and further obtains reference weight information by color correction, fuses the target input feature and the target reference information to obtain a fused feature, and processes the fused feature according to the reference weight information to obtain a target image, so that the coloring of the image is more accurate and natural.

[0142] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized by, include: Obtain an input image and a reference image; wherein, the input image is an image to be colored, and the input image includes a completely black and white image or a partially black and white image, and the reference image is a full-color image corresponding to the input image; The target input features are obtained by performing feature extraction on the input image; Feature extraction is performed on the reference image to obtain target reference features and reference weight information; The target input features and the target reference features are fused to obtain the fused features; The target image is obtained by processing the fusion features based on the reference weight information; The step of extracting features from the reference image to obtain target reference features and reference weight information includes: The reference image is sequentially processed by convolution and attention mechanisms to obtain the first intermediate reference feature; The target reference feature is obtained by performing downsampling, convolution, and attention mechanisms sequentially on the first intermediate reference feature N times. The process of performing downsampling, convolution, and attention mechanisms on the first intermediate reference feature N times sequentially to obtain the target reference feature includes: The j-th intermediate reference feature is processed sequentially by downsampling, convolution, and attention mechanisms in the j-th iteration to obtain the (j+1)-th intermediate reference feature, where j is an integer and 0 < j ≤ N; The (N+1)th intermediate reference feature is used as the target reference feature; The step of extracting features from the reference image to obtain target reference features and reference weight information also includes: The k-th intermediate reference feature is subjected to global pooling and convolution processing in sequence to obtain the k-th reference weight information, where k is an integer and 0 < k ≤ N+1; The step of processing the fused features based on the reference weight information to obtain the target image includes: The fused features are sequentially processed by convolution and attention mechanisms to obtain the first intermediate output feature; Based on the m-th weight information, the m intermediate output features are sequentially upsampled, convolved, and processed by attention mechanism to obtain the (m+1)-th intermediate output feature, where m is an integer and 0 < m ≤ N; The target image is obtained by performing two convolution processes on the (N+1)th intermediate output feature based on the (N+1)th weight information.

2. The image processing method of claim 1, wherein, The process of fusing the target input features and the target reference features to obtain the fused features includes: The target input features and the target reference features are sequentially concatenated and convolved to obtain the first concatenated features; The first connection feature is processed sequentially by attention mechanism and convolution to obtain the second connection feature; The fusion feature is obtained by processing the first connection feature and the second connection feature.

3. The image processing method of claim 1, wherein, The step of extracting target input features from the input image includes: The first intermediate input feature is obtained by sequentially performing convolution and attention processing on the input image; The target input feature is obtained by performing downsampling, convolution, and attention mechanisms sequentially on the first intermediate input feature N times, where N is an integer and N > 0.

4. The image processing method of claim 3, wherein, The downsampling process includes bicubic interpolation downsampling or max pooling.

5. The image processing method of claim 3, wherein, The process of performing downsampling, convolution, and attention mechanisms on the first intermediate input feature N times sequentially to obtain the target input feature includes: The i-th intermediate input feature is processed by downsampling, convolution and attention mechanisms in the i-th iteration to obtain the (i+1)-th intermediate input feature, where i is an integer and 0 < i ≤ N; The (N+1)th intermediate input feature is used as the target input feature.

6. The image processing method of claim 1, wherein, The sampling factors for the upsampling process and the sampling factors for the downsampling process are adapted to each other.

7. The image processing method of claim 1, wherein, The upsampling process includes bicubic interpolation upsampling or pixel recombination.

8. The image processing method of any of claims 1-5, wherein, The attention mechanism processing includes spatial attention mechanism processing, which is implemented through the following conditional expression: wherein, represents the feature output by the bth spatial attention mechanism processing, represents the feature input to the bth spatial attention mechanism processing, represents the n-th convolutional parameter in the bth spatial attention mechanism processing, n = 1, 2, 3, 4, represents the up-sampling operator in the spatial attention mechanism processing.

9. An image processing apparatus characterized by comprising: The image processing apparatus is used to implement the image processing method according to any one of claims 1 to 8, and the image processing apparatus includes: An input feature extraction module is used to acquire an input image and extract features from the input image to obtain target input features; A reference feature extraction module is used to acquire a reference image and extract features from the reference image to obtain target reference features; The correction module is used to determine reference weight information based on the reference image; The fusion module is used to fuse the input features and the reference features to obtain fused features; and The output module is used to process the fused features according to the reference weight information to obtain the target image.

10. An electronic device, characterized in that, The image processing apparatus includes the image processing apparatus of claim 9, which is used to implement the image processing method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the image processing method as described in any one of claims 1 to 8.

Images