Image processing method and device, and electronic equipment

By calculating filter weight values ​​in image processing and applying them to the image to be processed, the problem of harsh color effects when applying filters is solved, and a natural filter effect transition is achieved.

CN113658066BActive Publication Date: 2024-11-15VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110912037.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-09
Publication Date
2024-11-15
Estimated Expiration
2041-08-09

AI Technical Summary

Technical Problem

In existing technologies, when a filter is extracted from one image to another, the color effect is harsh and unnatural, producing a mask-like effect.

Method used

By acquiring the first filter image and the image to be processed, the target filter is determined, and based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, the filter weight value of each pixel is calculated. These weight values ​​are then applied to the image to be processed to generate the second filter image.

Benefits of technology

It achieves a natural transition in filter effects, avoids harsh color effects, and produces a well-balanced and natural effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113658066B_ABST
    Figure CN113658066B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method, device, and electronic device, which belongs to the field of artificial intelligence technology. The method includes: obtaining a first filter image and an image to be processed, wherein the first filter image includes an image after filter processing using a target filter; determining the target filter based on the first filter image and the image to be processed; determining the filter weight value corresponding to each pixel in the image to be processed based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter; applying the target filter to each pixel in the image to be processed with the filter weight value corresponding to each pixel, and obtaining a second filter image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing method and device, and electronic equipment. Background Art

[0002] In current photography and photo album processing, filters have become an essential image processing function. Essentially, filters adjust the color of an image to achieve the purpose of changing the image style.

[0003] In order to enable users to apply the filter effects of other images to the target image, filter migration technology came into being. That is, by extracting the filter on the picture and applying it to the new photo, the corresponding filter can be obtained without downloading the corresponding software.

[0004] However, a filter usually corresponds to a single-color pure color image. When the extracted filter is applied to other images, although a special color effect will be produced, it will also produce an effect similar to a mask, making the color effect on the image very stiff and unnatural. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and device, and an electronic device, which can solve the problem in the prior art that the color effect is stiff and unnatural when the filter extracted from the filter image is applied to other images.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the image processing method comprising:

[0007] Acquire a first filtered image and an image to be processed, wherein the first filtered image includes an image after filter processing using a target filter;

[0008] Determining the target filter based on the first filtered image and the image to be processed;

[0009] Determining filter weight values ​​corresponding to respective pixels in the image to be processed based on differences between the colors of the respective pixels in the image to be processed and the colors indicated by the target filter;

[0010] The target filter is applied to each pixel in the image to be processed with the filter weight value corresponding to each pixel, so as to obtain a second filtered image.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0012] A first acquisition module, used to acquire a first filtered image and an image to be processed, wherein the first filtered image includes an image after filter processing using a target filter;

[0013] A first determination module, configured to determine the target filter based on the first filtered image and the image to be processed;

[0014] A second determination module, configured to determine a filter weight value corresponding to each pixel in the image to be processed based on a difference between a color of each pixel in the image to be processed and a color indicated by the target filter;

[0015] The filter module is used to apply the target filter to each pixel in the image to be processed with the filter weight value corresponding to each pixel, so as to obtain a second filtered image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0019] In an embodiment of the present application, a target filter used when performing filter processing to obtain a first filter image is determined based on a first filter image after filter processing and an image to be processed that has not been filtered. By taking the image to be processed into consideration, interference caused by the content color of the first filter image itself when extracting the target filter based only on the first filter image can be avoided. Then, based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, the filter weight values ​​corresponding to each pixel in the image to be processed are determined, and the target filter is applied to each pixel of the image to be processed in combination with the filter weight values ​​corresponding to each pixel to obtain a second filter image after filter processing. Since the filter weight values ​​of the target filter applied to pixels of different colors are different, the color effect caused by the target filter in the entire second filter image is well-arranged and very natural, and there will be no mask-like effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of the steps of the image processing method provided in an embodiment of the present application;

[0021] Figure 2is a schematic diagram of the actual application of the image processing method provided in the embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the process of processing an image by a deep learning network model provided in an embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the training process of the deep learning network model provided in the embodiment of the present application;

[0024] Figure 5 is a structural block diagram of an image processing device provided in an embodiment of the present application;

[0025] Figure 6 This is one of the hardware structure diagrams of the electronic device provided in the embodiment of the present application;

[0026] Figure 7 This is the second schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0030] like Figure 1 As shown, an image processing method provided by an embodiment of the present application, the image processing method includes:

[0031] Step 101: Obtain a first filtered image and an image to be processed.

[0032] In this step, the first filter image includes an image after filtering with a target filter, that is, the first filter image can be understood as an image that produces a color effect after a filter is applied to any image, wherein the target filter is the applied filter. Applying a filter to an image can make the image produce a special color effect, and a filter can be understood as a kind of color data, for example, a pure color image of a single color. Therefore, each filter can indicate a color. The color effect here can be the effect produced by adjusting the color, texture, etc. Of course, for images containing faces, the color effect can also include the effects brought by different makeup. The image to be processed can be any image selected by the user. Specifically, the image to be processed is an image selected by the user and has not been processed by a filter. The user wants to apply a target filter to the image to achieve the same color effect as the first filter image.

[0033] Step 102: Determine a target filter based on the first filtered image and the image to be processed.

[0034] In this step, after applying a filter to an image, a filter image with a color effect will be generated. The applied filter can be extracted based on the images before and after the filter is applied. Here, the original image corresponding to the first filter image is replaced with the image to be processed, wherein the original image corresponding to the first filter image is subjected to filter processing to obtain the first filter image. Of course, the filter applied to the image can also be directly extracted from the filter image, that is, the filter color in the first filter image is extracted to obtain the target filter.

[0035] Step 103: Based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, determine the filter weight value corresponding to each pixel in the image to be processed.

[0036] In this step, the difference between colors can be understood as the difference between the color values ​​of different colors in the same color space. The distance between two colors in the same color space can be used as a measure of the difference between the two colors. The larger the distance, the greater the difference between the colors. Similarly, the smaller the distance, the smaller the difference. If the distances are the same, it means that the two colors are the same and have no difference.

[0037] The filter weight value of each pixel is associated with the difference between the color of the pixel and the color indicated by the target filter, so the filter weight values ​​corresponding to pixels of different colors are different. It can be understood that the image to be processed is composed of a large number of pixels, and the colors of each pixel can be the same or different. The filter weight values ​​corresponding to pixels of the same color are the same, and the filter weight values ​​corresponding to pixels of different colors are different.

[0038] Step 104: applying the target filter to each pixel in the image to be processed with the filter weight value corresponding to each pixel, to obtain a second filtered image.

[0039] In this step, the target filter is applied to each pixel in the image to be processed with the filter weight value corresponding to each pixel, that is, the image to be processed is subjected to filter processing by the target filter and the filter weight value. Specifically, for the target pixel of the image to be processed, the target filter is applied to the target pixel with the filter weight value corresponding to the target pixel, wherein the target pixel is all the pixels of the image to be processed, that is, each pixel in the image to be processed needs to be subjected to filter processing. It can be understood that the filter weight value is a specific value. When the target filter is applied to a pixel in the image to be processed with a certain filter weight value, the color value of the target filter is multiplied by the filter weight value to obtain a new color value, and then the new color value is applied to a pixel in the image to be processed. For example, if the color value of the target filter is 100 and the filter weight value of the target pixel on the image to be processed is 0.5, the process of applying the target filter to the target pixel includes: obtaining a new color value 50 by multiplying the color value 100 by the filter weight value 0.5, and applying the color value 50 to the target pixel. Since the filter weight values ​​corresponding to pixels of different colors in the image to be processed are different, the color values ​​applied to pixels of different colors are different, and the color effects on pixels of different colors in the second filter image will not be consistent, but will be different. For example, if the color indicated by the target filter is blue, when it is applied to the pixels of the blue sky part with a larger filter weight value in the image to be processed, the blue sky will be bluer, and when it is applied to the pixels of the building part with a smaller filter weight value in the image to be processed, the building part will only have a faint and inconspicuous blue, so that the overall color effect of the second filter image after the filter processing will be patchwork.

[0040] In an embodiment of the present application, a target filter used when performing filter processing to obtain a first filter image is determined based on a first filter image after filter processing and an image to be processed that has not been filtered. By taking the image to be processed into consideration, interference caused by the content color of the first filter image itself when extracting the target filter based only on the first filter image can be avoided. Then, based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, the filter weight values ​​corresponding to each pixel in the image to be processed are determined, and the target filter is applied to each pixel of the image to be processed in combination with the filter weight values ​​corresponding to each pixel to obtain a second filter image after filter processing. Since the filter weight values ​​of the target filter applied to pixels of different colors are different, the color effect caused by the target filter in the entire second filter image is well-arranged and very natural, and there will be no mask-like effect.

[0041] Optionally, determining a target filter based on the first filtered image and the image to be processed includes:

[0042] The first filtered image and the image to be processed are input into a preset target network model.

[0043] In this step, the preset target network model is a pre-trained network model. Here, an initial model can be trained based on a deep learning network to obtain the target network model. The first filter image and the image to be processed are model inputs of the target network model.

[0044] The first color feature of the first filter image and the second color feature of the image to be processed are obtained through the target network model.

[0045] In this step, the first color feature is a feature of data associated with its color in the first filter image, and the second color feature is a feature of data associated with its color in the image to be processed. Here, the color feature can be used as a staged output of the target network model rather than the final model output.

[0046] The feature difference between the first color feature and the second color feature is obtained through the target network model, and the feature difference is determined as the target filter.

[0047] In this step, the feature difference value obtained by subtracting the first color feature from the second color feature can represent the target filter, and the color corresponding to the feature difference value can be used as the color indicated by the target filter. It can be understood that the feature difference value can represent the color difference between the image to be processed and the first filter image, so that by processing the image to be processed through the color difference, an image with no color difference with the first filter image can be obtained. Therefore, the feature difference value can be used as the target filter of the first filter image.

[0048] In an embodiment of the present application, a target filter is extracted using a pre-trained target network model, and the first filter image and the image to be processed are used as model inputs of the target network model, so that the target filter can be obtained quickly and accurately.

[0049] Optionally, obtaining a first color feature of the first filter image and a second color feature of the image to be processed through a target network model includes:

[0050] Through the image feature extraction module of the target network model, a first image feature vector of the first filter image and a second image feature vector of the image to be processed are obtained.

[0051] In this step, in order to facilitate image processing, the image can be first converted into a mathematical expression, and the first filter image and the image to be processed are represented by the mathematical expression. Among them, the first image feature vector is a mathematical expression of the first filter image, which can represent the characteristics of each dimension of the first filter image. The second image feature vector is a mathematical expression of the image to be processed, which can represent the characteristics of each dimension of the image to be processed. Among them, the dimensions of the image can include brightness dimension, color dimension, etc.

[0052] Specifically, the image feature extraction module can be implemented by a series of stacked mathematical operations to obtain the image feature vector of the image. For example, the first image feature vector and the second image feature vector are calculated by using the first formula.

[0053] First formula: outFeature = w n *(w n-1 *(...(w1*x+b1))+b n-1 )+b n , where x is the input and outFeature is the output. When x is the first filter image, outFeature is the first image feature vector. When x is the image to be processed, outFeature is the second image feature vector. * indicates convolution operation. w1~w n is n convolution kernels, b1~b n is n bias values, where n is a positive integer.

[0054] The first image feature vector and the second image feature vector are respectively input into the color feature extraction module of the target network model to obtain the first color feature of the first filter image and the second color feature of the image to be processed.

[0055] In this step, the color feature extraction module can also use a series of stacked mathematical operations to obtain the color features of the image feature vector. For example, the second formula is used to calculate the first color feature and the second color feature.

[0056] Second formula: outColor = cw n *(cw n-1 *(...(cw1*outFeature+cb1))+cb n-1 )+cb n , where outFeature is the input and outColor is the output. When outFeature is the first image feature vector, outColor is the first color feature. When outFeature is the second image feature vector, outColor is the second color feature. * is the convolution operation. cw1~cw n is n convolution kernels, cb1~cb n is n bias values, where n is a positive integer.

[0057] In the embodiment of the present application, a staged processing method is adopted. First, an image feature vector is obtained, and then based on the image feature vector, color features related to color are screened therefrom. The whole process is convenient, simple and easy to implement.

[0058] Optionally, determining the filter weight values ​​corresponding to the pixels in the image to be processed based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter includes:

[0059] The first filtered image and the image to be processed are input into a preset target network model.

[0060] In this step, the preset target network model is a pre-trained network model. Here, an initial model can be trained based on a deep learning network to obtain the target network model. The first filter image and the image to be processed are model inputs of the target network model.

[0061] Obtaining a first image feature vector of the first filter image and a second image feature vector of the image to be processed through an image feature extraction module of the target network model;

[0062] In this step, in order to facilitate image processing, the image can be first converted into a mathematical expression, and the first filter image and the image to be processed are represented by the mathematical expression. Among them, the first image feature vector is a mathematical expression of the first filter image, which can represent the characteristics of each dimension of the first filter image. The second image feature vector is a mathematical expression of the image to be processed, which can represent the characteristics of each dimension of the image to be processed. Among them, the dimensions of the image can include brightness dimension, color dimension, etc.

[0063] Specifically, the image feature extraction module can be implemented by a series of stacked mathematical operations to obtain the image feature vector of the image. For example, the first image feature vector and the second image feature vector are calculated by using the first formula in the above application embodiment, which will not be repeated here.

[0064] The vector difference between the first image feature vector and the second image feature vector is obtained through the target network model.

[0065] In this step, when the first image feature vector and the second image feature vector have been obtained, the two image feature vectors are subtracted to obtain a vector difference.

[0066] The absolute value of the vector difference is input into the weight branch module of the target network model to obtain the filter weight value corresponding to each pixel in the image to be processed, wherein the closer the color of the target pixel is to the color indicated by the target filter, the larger the filter weight value corresponding to the target pixel, and the target pixel includes any pixel in the image to be processed.

[0067] In this step, the weight branch module can be implemented by a series of stacked mathematical operations to obtain the filter weight value. For example, the filter weight values ​​of different parts of the image to be processed can be calculated by the third formula. The third formula can be: WeightImg = ww n*(ww n-1 *(...(ww1*|outFeature1-outFeature2|+wb1))+wb n-1 )+wb n , where |outFeature1-outFeature2| is the absolute value of the vector difference, WeightImg is the output, indicating the filter weight value corresponding to different pixels of the image to be processed, * is the convolution operation, ww1~ww n is n convolution kernels, wb1~wb n is n bias values, where n is a positive integer. It can be understood that WeightImg can also be regarded as a global weight map of the image to be processed, which includes the filter weight value corresponding to each pixel of the image to be processed.

[0068] In the embodiment of the present application, the closer the color of the target pixel in the image to be processed is to the color indicated by the target filter, the greater the filter weight value of the target pixel, and thus the more obvious the color effect when the target filter acts on the target pixel.

[0069] Optionally, before acquiring the first filtered image and the image to be processed, the image processing method further includes:

[0070] Obtain an initial network model and sample data; wherein the initial network model includes: an image feature extraction module, a color feature extraction module, a weight branch module and a content extraction module, and the sample data includes: an original image, a filter result image, and a filter color image, and the filter result image is an image obtained after adding a filter color image to the original image.

[0071] In this step, the initial network model can be regarded as an untrained target network model. Different modules in the initial network model are used to perform different functions, such as the image feature extraction module for extracting the image feature vector of the image, the color feature extraction module for extracting color features from the image feature vector, the weight branch module for extracting the weight value of each pixel in the original image, and the content extraction module for extracting the image content from the image feature vector through decoupled learning.

[0072] Inputting the filter result image and the original image into the image feature extraction module respectively to obtain a first training image feature vector and a second training image feature vector;

[0073] In this step, each image can be in RGB color mode, where RGB color mode is a color standard in the industry. Various colors are obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. Of course, it can also be in Lab color model, where Lab is a device-independent color model and a color model based on physiological characteristics. The Lab color model consists of three elements, one of which is brightness (L), and a and b are two color channels. The colors included in a are from dark green (low brightness value) to gray (medium brightness value) to bright pink (high brightness value); b is from bright blue (low brightness value) to gray (medium brightness value) to yellow (high brightness value). Here, in order to facilitate the decoupling of color and brightness, when each image is in RGB color mode, it is converted to Lab color model, which is what we often call converting the RGB space of the image to Lab space. When extracting the feature vector of the training image, it can be extracted based on the above first formula.

[0074] Inputting the first training image feature vector and the second training image feature vector into a color feature extraction module respectively to obtain a first training color feature and a second training color feature;

[0075] In this step, color features are extracted from the training image feature vector, which can be extracted based on the second formula above.

[0076] A difference value obtained by subtracting the first training color feature from the second training color feature is determined as the predicted filter color.

[0077] In this step, the predicted filter color is the color indicated by the filter applied when the original image is subjected to filter processing to obtain the filter result image.

[0078] The target absolute value is input into the weight branch module to obtain the training filter weight value corresponding to each pixel in the original image, wherein the target absolute value is the absolute value of the difference obtained by subtracting the second training image feature vector from the first training image feature vector.

[0079] In this step, the training filter weight values ​​corresponding to each pixel in the original image can be extracted based on the third formula.

[0080] Inputting the first training image feature vector into a content extraction module to obtain a training content graph related to the image content of the filter result graph;

[0081] In this step, the image content of the filter result image, that is, the training content image, can be extracted based on the fourth formula: ImageContent = iw n *(iw n-1*(...(iw1*outFeature+ib1))+ib n-1 )+ib n , where outFeature is the input, representing the first training image feature vector of the filter result image, ImageContent is the output, representing the training content image, * is the convolution operation, iw1~iw n is n convolution kernels, ib1~ib n is n bias values, where n is a positive integer.

[0082] Determine a model loss based on the predicted filter color, the trained filter weight value, the trained content map, and the filter color map, wherein the model loss includes an image content loss, an image color loss, and a cycle consistency loss;

[0083] In this step, the image content loss is used to measure the content difference; specifically, image content loss = |ImgContent-ImgSource|, where ImgContent represents the training content image and ImgSource represents the original image. The difference in filter color is measured by subtracting the value output by the color branch (predicted filter color) from the marked color (filter color map) obtained in advance. Specifically, color loss = |ColorGroudtruth-ColorPredicted], where ColorGroudtruth represents the filter color map; ColorPredicted represents the predicted filter color; the training filter weight values ​​corresponding to different pixels output by the weight branch module are combined with the color branch output value (predicted filter color) and the original image. Then add the original image value to the color and multiply it by the training filter weight value to get the result image, and calculate the L1 loss of the obtained result image and the input filter result image to measure the accuracy of reconstruction; specifically, cycle consistency loss = |ImgTarget-ImgSource*ColorPredicted*WeightImg|, where ImgTarget represents the filter result image; ImgSource represents the original image; ColorPredicted represents the predicted filter color; WeightImg represents the training filter weight value.

[0084] Update the model parameters and continue training based on the new sample data until the model loss converges. The initial network model after training is determined as the target network model.

[0085] In this step, the content loss, color loss, and cycle consistency loss are solved, and the partial derivatives of the convolution kernels in the above formulas are calculated. Then, each convolution kernel is updated, where the new convolution kernel is equal to the old convolution kernel plus the partial derivative of the convolution kernel in the previous training process. In this way, the training ends after the model loss converges, and the model parameters are saved. The model parameters are the convolution kernels in the above formulas. Here, different sample data need to be used for model training, and the model parameters are updated every time training is performed.

[0086] In an embodiment of the present application, model training is performed based on the original image, filter result image, and filter color image in the sample data, the model parameters are updated based on the results of each training, and the training is stopped after the model loss converges to obtain a trained target network model.

[0087] Optionally, inputting the first filtered image and the image to be processed into a preset target network model includes:

[0088] When the first filter image and the image to be processed are in RGB color mode, the first filter image and the image to be processed are converted into Lab color mode respectively.

[0089] In the embodiment of the present application, the color space of the image is converted to facilitate the subsequent extraction of color features.

[0090] like Figure 2 The figure shows a practical application diagram of the image processing method provided by the embodiment of the present application, which includes:

[0091] Step 201: Obtain a user map and a target filter map to which the user wants to migrate filters, wherein the user map is the image to be processed in the above-mentioned application embodiment, and the target filter map is the first filter map in the above-mentioned application embodiment.

[0092] Step 202: Input the acquired image into the deep learning network model.

[0093] Step 203: Obtain the result image after the user image migration filter, that is, the second filter image in the above application embodiment.

[0094] in, Figure 3The figure shows a schematic diagram of the image processing process of the deep learning network model. The target filter image and the user image are respectively input into the image feature extraction module to obtain their respective image feature vectors, and their respective image feature vectors are respectively input into the corresponding color feature extraction modules to obtain their respective color features. Then, the color features obtained by the original image processing branch are subtracted from the color features obtained by the filter image processing branch to obtain the predicted color. At the same time, the absolute value after subtracting the two image feature vectors is input into the weight branch module of the original image processing branch to obtain the global weight map. The predicted color, the global weight map and the user image can be used to obtain the result image after the user image migration filter. Result map = ImgSource*ColorPredicted*WeightImg, where ImgSource represents the user image; ColorPredicted represents the predicted color; WeightImg represents the global weight map. The process of obtaining the image feature vector, color feature and global weight map in the embodiment of the present application can refer to the first formula, the second formula and the third formula in the above-mentioned application embodiment, which will not be repeated here. It is worth noting that Figure 3 The convolution kernels are shared between the two image feature extraction modules, and the convolution kernels are shared between the two color feature extraction modules.

[0095] It is understandable that the process of training a deep learning network module (target network model) is similar to Figure 3 Similar, such as Figure 4 As shown in Figure 1, specifically, the sample data used in the training process includes: the original image, the filter result image, and the filter color map. The filter result image is the image obtained after adding the filter color map to the original image. The original image and the filter result image are input into the model respectively to obtain the predicted color and the global weight map. The process of obtaining the predicted color and the global weight map is the same as Figure 3 The process of inputting the user graph and the target filter graph respectively to obtain the predicted color and global weight graph is similar and will not be repeated here. It is worth noting that during the training process, the training content graph corresponding to the filter result graph can also be obtained through the image feature extraction module and content extraction module of the model. Figure 4The convolution kernel is shared between the two image feature extraction modules, and the convolution kernel is shared between the two color feature extraction modules. After obtaining various data, the model loss is calculated based on the obtained data, and then the model parameters are updated. Here, the model loss includes content loss, color loss, and cycle consistency loss, among which the content loss measures the difference in image content; specifically, image content loss = |ImgContent-ImgSource|, where ImgContent represents the training content map, and ImgSource represents the original image. Color loss measures the difference in filter color, color loss = |ColorGroudtruth-ColorPredicted|, where ColorGroudtruth represents the actual color, that is, the filter color map; ColorPredicted represents the predicted color; the training filter weight values ​​corresponding to different pixels output by the weight branch module are used, combined with the color branch output value (predicted color) and the original image. Then add the original image value to the color and multiply it by the training filter weight value to get the result image. The obtained result image and the input filter result image are used to calculate the L1 loss to measure the reconstruction accuracy. Specifically, the cycle consistency loss = |ImgTarget-ImgSource*ColorPredicted*WeightImg|, where ImgTarget represents the filter result image; ImgSource represents the original image; ColorPredicted represents the predicted color; and WeightImg represents the training filter weight value. Through continuous training, the model parameters can be continuously updated until the model loss converges and the training is stopped.

[0096] The embodiment of the present application can estimate a more accurate filter color to avoid interference from the image content. In addition, by outputting an additional global weight map, filters with different weights are used at different positions of the image, making the result image more natural and avoiding the mask effect.

[0097] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0098] like Figure 5 As shown, the embodiment of the present application also provides an image processing device, which includes:

[0099] A first acquisition module 51 is used to acquire a first filtered image and an image to be processed, wherein the first filtered image includes an image processed by a target filter;

[0100] A first determination module 52, configured to determine a target filter based on the first filter image and the image to be processed;

[0101] A second determination module 53, for determining the filter weight value corresponding to each pixel in the image to be processed based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter;

[0102] The filter module 54 is used to apply the target filter to each pixel in the image to be processed with the filter weight value corresponding to each pixel, so as to obtain a second filtered image.

[0103] Optionally, the first determining module 52 includes:

[0104] A first input unit, used to input the first filtered image and the image to be processed into a preset target network model;

[0105] A first model unit, used for acquiring a first color feature of a first filter image and a second color feature of an image to be processed through a target network model;

[0106] The second model unit is used to obtain a feature difference between the first color feature and the second color feature through a target network model, and determine the feature difference as a target filter.

[0107] Optionally, the first model unit includes:

[0108] A first model subunit is used to obtain a first image feature vector of a first filter image and a second image feature vector of an image to be processed through an image feature extraction module of a target network model;

[0109] The second model subunit is used to input the first image feature vector and the second image feature vector into the color feature extraction module of the target network model respectively, so as to obtain the first color feature of the first filter image and the second color feature of the image to be processed.

[0110] Optionally, the second determining module 53 includes:

[0111] A second input unit, used for inputting the first filtered image and the image to be processed into a preset target network model;

[0112] A third model unit is used to obtain a first image feature vector of the first filter image and a second image feature vector of the image to be processed through an image feature extraction module of the target network model;

[0113] A fourth model unit, used for obtaining a vector difference between the first image feature vector and the second image feature vector through a target network model;

[0114] The fifth model unit is used to input the absolute value of the vector difference into the weight branch module of the target network model to obtain the filter weight value corresponding to each pixel in the image to be processed, wherein the closer the color of the target pixel is to the color indicated by the target filter, the larger the filter weight value corresponding to the target pixel, and the target pixel includes any pixel in the image to be processed.

[0115] Optionally, the image processing device further includes:

[0116] The second acquisition module is used to acquire an initial network model and sample data; wherein the initial network model includes: an image feature extraction module, a color feature extraction module, a weight branch module and a content extraction module; the sample data includes: an original image, a filter result image, and a filter color image; the filter result image is an image obtained by adding the filter color image to the original image;

[0117] A first training module, used to input the filter result image and the original image into an image feature extraction module respectively, to obtain a first training image feature vector and a second training image feature vector;

[0118] A second training module, used to input the first training image feature vector and the second training image feature vector into a color feature extraction module to obtain a first training color feature and a second training color feature;

[0119] A third training module is used to determine a difference obtained by subtracting the first training color feature from the second training color feature as a predicted filter color;

[0120] A fourth training module is used to input the target absolute value into the weight branch module to obtain the training filter weight value corresponding to each pixel in the original image, wherein the target absolute value is the absolute value of the difference between the first training image feature vector and the second training image feature vector;

[0121] A fifth training module, configured to input the first training image feature vector into a content extraction module to obtain a training content graph related to the image content of the filter result graph;

[0122] a sixth training module, configured to determine a model loss based on the predicted filter color, the training filter weight value, the training content map, and the filter color map, wherein the model loss includes an image content loss, an image color loss, and a cycle consistency loss;

[0123] The seventh training module is used to update the model parameters and continue training based on new sample data until the model loss converges, and the initial network model after the training is determined as the target network model.

[0124] Optionally, the first input unit is specifically configured to convert the first filtered image and the image to be processed into Lab color models respectively when the first filtered image and the image to be processed are in RGB color mode.

[0125] In an embodiment of the present application, a target filter used when performing filter processing to obtain a first filter image is determined based on a first filter image after filter processing and an image to be processed that has not been filtered. By taking the image to be processed into consideration, interference caused by the content color of the first filter image itself when extracting the target filter based only on the first filter image can be avoided. Then, based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, the filter weight values ​​corresponding to each pixel in the image to be processed are determined, and the target filter is applied to each pixel of the image to be processed in combination with the filter weight values ​​corresponding to each pixel to obtain a second filter image after filter processing. Since the filter weight values ​​of the target filter applied to pixels of different colors are different, the color effect caused by the target filter in the entire second filter image is well-arranged and very natural, and there will be no mask-like effect.

[0126] The image processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0127] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0128] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0129] Alternatively, if Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0130] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0131] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0132] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0133] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0134] The processor 710 is used to obtain a first filtered image and an image to be processed, wherein the first filtered image includes an image after being filtered using a target filter;

[0135] The processor 710 is further configured to determine a target filter based on the first filter image and the image to be processed;

[0136] The processor 710 is further configured to determine the filter weight value corresponding to each pixel in the image to be processed based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter;

[0137] The processor 710 is further configured to apply the target filter to each pixel in the image to be processed with the filter weight value corresponding to each pixel, so as to obtain a second filtered image.

[0138] In an embodiment of the present application, a target filter used when performing filter processing to obtain a first filter image is determined based on a first filter image after filter processing and an image to be processed that has not been filtered. By taking the image to be processed into consideration, interference caused by the content color of the first filter image itself when extracting the target filter based only on the first filter image can be avoided. Then, based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, the filter weight values ​​corresponding to each pixel in the image to be processed are determined, and the target filter is applied to each pixel of the image to be processed in combination with the filter weight values ​​corresponding to each pixel to obtain a second filter image after filter processing. Since the filter weight values ​​of the target filter applied to pixels of different colors are different, the color effect caused by the target filter in the entire second filter image is well-arranged and very natural, and there will be no mask-like effect.

[0139] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0140] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0141] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0143] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0144] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0146] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, It is characterized in that The image processing method comprises: Acquire a first filtered image and an image to be processed, wherein the first filtered image includes an image after filter processing using a target filter; Determining the target filter based on the first filtered image and the image to be processed; Based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, determine the filter weight value corresponding to each pixel in the image to be processed, wherein the filter weight value is inversely proportional to the color difference; The target filter is applied to each pixel in the image to be processed with the filter weight value corresponding to each pixel, so as to obtain a second filtered image.

2. The image processing method according to claim 1, It is characterized in that The step of determining the target filter based on the first filtered image and the image to be processed includes: Inputting the first filtered image and the image to be processed into a preset target network model; Acquire a first color feature of the first filter image and a second color feature of the image to be processed through the target network model; A feature difference between the first color feature and the second color feature is obtained through the target network model, and the feature difference is determined as the target filter.

3. The image processing method according to claim 2, It is characterized in that The step of obtaining the first color feature of the first filtered image and the second color feature of the image to be processed through the target network model includes: Obtaining, by means of an image feature extraction module of the target network model, a first image feature vector of the first filtered image and a second image feature vector of the image to be processed; The first image feature vector and the second image feature vector are respectively input into the color feature extraction module of the target network model to obtain the first color feature of the first filter image and the second color feature of the image to be processed.

4. The image processing method according to claim 1, It is characterized in that The determining, based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, the filter weight value corresponding to each pixel in the image to be processed comprises: Inputting the first filtered image and the image to be processed into a preset target network model; Obtaining, by means of an image feature extraction module of the target network model, a first image feature vector of the first filtered image and a second image feature vector of the image to be processed; Acquire a vector difference between the first image feature vector and the second image feature vector through the target network model; The absolute value of the vector difference is input into the weight branch module of the target network model to obtain the filter weight values ​​corresponding to each pixel in the image to be processed, wherein the closer the color of the target pixel is to the color indicated by the target filter, the larger the filter weight value corresponding to the target pixel, and the target pixel includes any pixel in the image to be processed.

5. The image processing method according to claim 4, It is characterized in that Before acquiring the first filtered image and the image to be processed, the image processing method further includes: Acquire an initial network model and sample data; wherein the initial network model includes: the image feature extraction module, the color feature extraction module, the weight branch module and the content extraction module; the sample data includes: an original image, a filter result image, and a filter color image; the filter result image is an image obtained by adding the filter color image to the original image; Inputting the filter result image and the original image into the image feature extraction module respectively to obtain a first training image feature vector and a second training image feature vector; Inputting the first training image feature vector and the second training image feature vector into the color feature extraction module respectively to obtain a first training color feature and a second training color feature; Determine a difference obtained by subtracting the first training color feature from the second training color feature as a predicted filter color; Inputting the target absolute value into the weight branch module to obtain the training filter weight value corresponding to each pixel in the original image, wherein the target absolute value is the absolute value of the difference between the first training image feature vector and the second training image feature vector; Inputting the first training image feature vector into the content extraction module to obtain a training content graph related to the image content of the filter result graph; Determining a model loss based on the predicted filter color, the training filter weight value, the training content map, and the filter color map, wherein the model loss includes an image content loss, an image color loss, and a cycle consistency loss; The model parameters are updated, and training is continued based on new sample data until the model loss converges, and the initial network model after the training is determined as the target network model.

6. An image processing device, It is characterized in that The image processing device comprises: A first acquisition module, used to acquire a first filtered image and an image to be processed, wherein the first filtered image includes an image after filter processing using a target filter; A first determination module, configured to determine the target filter based on the first filtered image and the image to be processed; A second determination module is used to determine the filter weight value corresponding to each pixel in the image to be processed based on the difference between the color of each pixel in the image to be processed and the color indicated by the target filter, wherein the filter weight value is inversely proportional to the color difference; The filter module is used to apply the target filter to each pixel in the image to be processed with the filter weight value corresponding to each pixel, so as to obtain a second filtered image.

7. The image processing device according to claim 6, It is characterized in that The first determining module includes: A first input unit, used for inputting the first filtered image and the image to be processed into a preset target network model; A first model unit, configured to obtain a first color feature of the first filter image and a second color feature of the image to be processed through the target network model; The second model unit is used to obtain a feature difference between the first color feature and the second color feature through the target network model, and determine the feature difference as the target filter.

8. The image processing device according to claim 7, It is characterized in that The first model unit comprises: A first model subunit is used to obtain a first image feature vector of the first filtered image and a second image feature vector of the image to be processed through an image feature extraction module of the target network model; The second model subunit is used to input the first image feature vector and the second image feature vector into the color feature extraction module of the target network model respectively, so as to obtain the first color feature of the first filter image and the second color feature of the image to be processed.

9. The image processing device according to claim 6, It is characterized in that The second determining module includes: A second input unit, used for inputting the first filtered image and the image to be processed into a preset target network model; A third model unit, configured to obtain a first image feature vector of the first filtered image and a second image feature vector of the image to be processed through an image feature extraction module of the target network model; a fourth model unit, configured to obtain a vector difference between the first image feature vector and the second image feature vector through the target network model; The fifth model unit is used to input the absolute value of the vector difference into the weight branch module of the target network model to obtain the filter weight value corresponding to each pixel in the image to be processed, wherein the closer the color of the target pixel is to the color indicated by the target filter, the larger the filter weight value corresponding to the target pixel, and the target pixel includes any pixel in the image to be processed.

10. The image processing device according to claim 9, It is characterized in that The image processing device further comprises: A second acquisition module is used to acquire an initial network model and sample data; wherein the initial network model includes: the image feature extraction module, the color feature extraction module, the weight branch module and the content extraction module; the sample data includes: an original image, a filter result image, and a filter color image; the filter result image is an image obtained by adding the filter color image to the original image; A first training module, used for inputting the filter result image and the original image into the image feature extraction module respectively, to obtain a first training image feature vector and a second training image feature vector; A second training module, used for inputting the first training image feature vector and the second training image feature vector into the color feature extraction module respectively to obtain a first training color feature and a second training color feature; A third training module, configured to determine a difference obtained by subtracting the first training color feature from the second training color feature as a predicted filter color; a fourth training module, configured to input a target absolute value into the weight branch module to obtain a training filter weight value corresponding to each pixel in the original image, wherein the target absolute value is an absolute value of a difference between the first training image feature vector and the second training image feature vector; A fifth training module, configured to input the first training image feature vector into the content extraction module to obtain a training content graph related to the image content of the filter result graph; a sixth training module, configured to determine a model loss based on the predicted filter color, the training filter weight value, the training content map, and the filter color map, wherein the model loss includes an image content loss, an image color loss, and a cycle consistency loss; The seventh training module is used to update the model parameters and continue training based on new sample data until the model loss converges, and determine the initial network model after the training as the target network model.

11. An electronic device, It is characterized in that The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 5.

12. A readable storage medium, It is characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented.