Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a saliency pixel region and a non-saliency pixel region, which are divided in a preset dimension, of an original image based on the difference, in the preset dimension, of each pixel point contained in the original image; and through different image enhancement processing strategies, respectively carrying out image enhancement processing on the salient pixel region and the non-salient pixel region to obtain a processed image. Through the method provided by the invention, the image enhancement requirements of different areas in the original image can be met, so that a better image enhancement effect is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Image enhancement technology can make an originally unclear image clear by improving the color, brightness, and contrast of the image, or it can enhance the scene expression ability of the image by emphasizing the features of the object of interest in the image, suppressing the features of the object of no interest, and expanding the differences between the features of different objects in the image, thereby facilitating subsequent high-level processing tasks such as target recognition. Summary of the invention

[0003] In view of this, the present disclosure provides an image processing method, device, electronic device and storage medium to solve the above technical problems.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for processing an image is provided, the method comprising:

[0005] Based on the difference of each pixel point contained in the original image in the preset dimension, determining the significant pixel area and the non-significant pixel area divided by the original image in the preset dimension;

[0006] By using different image enhancement processing strategies, image enhancement processing is performed on the significant pixel region and the non-significant pixel region respectively to obtain a processed image.

[0007] Optionally, the method further includes: inputting the original image into an image enhancement model; the image enhancement model includes a generation sub-model and a discrimination sub-model, wherein:

[0008] The generating sub-model is used to divide the original image into the significant pixel region and the non-significant pixel region, and perform image enhancement processing on the significant pixel region and the non-significant pixel region respectively through different image enhancement processing strategies to obtain a processed image;

[0009] The discriminant sub-model is used to assist in training the generative sub-model according to the difference between the sample processed image and the true value image in the sample significant pixel area and / or the difference between the sample processed image and the true value image in the sample non-significant pixel area when the sample original image includes the sample significant pixel area and the sample non-significant pixel area divided according to the preset dimension; wherein the sample processed image is output by the generative sub-model after processing the sample original image.

[0010] Optionally, the image enhancement model is a generative adversarial network, wherein:

[0011] The generative sub-model is a generator in the generative adversarial network;

[0012] The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

[0013] Optionally, the preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

[0014] According to a second aspect of an embodiment of the present disclosure, a method for training an image enhancement model is provided, the method comprising:

[0015] Acquire a sample original image and a true value image corresponding to the sample original image;

[0016] The sample original image is input into the image enhancement model, so that the generation sub-model in the image enhancement model divides the sample original image into a sample significant pixel region and a sample non-significant pixel region based on the difference of each pixel point contained in the sample original image in a preset dimension, and performs image enhancement processing on the sample significant pixel region and the sample non-significant pixel region respectively through different image enhancement processing strategies to obtain a sample processed image;

[0017] The image data of the sample processed image and the true value image corresponding to the sample significant pixel area and / or the sample non-significant pixel area are input into the discriminant sub-model in the image enhancement model, so that the discriminant sub-model assists in training the generation sub-model according to the difference between the sample processed image and the true value image at the sample significant pixel area and / or the difference between the sample processed image and the true value image at the sample non-significant pixel area.

[0018] Optionally, the image enhancement model is a generative adversarial network, wherein:

[0019] The generative sub-model is a generator in the generative adversarial network;

[0020] The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

[0021] Optionally, the preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

[0022] Optionally, the method further includes: updating the parameters of the generation sub-model according to the difference between the sample processed image and the true value image.

[0023] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing device, the device comprising:

[0024] A region determination module, configured to determine a significant pixel region and a non-significant pixel region divided into the original image on the preset dimension based on differences between pixels contained in the original image on the preset dimension;

[0025] The image processing module is used to perform image enhancement processing on the significant pixel area and the non-significant pixel area respectively through different image enhancement processing strategies to obtain a processed image.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a training device for an image enhancement model is provided, the device comprising:

[0027] An image acquisition module is used to acquire a sample original image and a true value image corresponding to the sample original image;

[0028] A sample image processing module, used for inputting the sample original image into an image enhancement model, so that the generation sub-model in the image enhancement model divides the sample original image into a sample significant pixel region and a sample non-significant pixel region based on the difference of each pixel point contained in the sample original image in a preset dimension, and performs image enhancement processing on the sample significant pixel region and the sample non-significant pixel region respectively through different image enhancement processing strategies to obtain a sample processed image;

[0029] an assisting training module, for inputting the image data of the processed sample image and the true value image corresponding to the sample significant pixel area and / or the sample non-significant pixel area into the discriminant sub-model in the image enhancement model, so that the discriminant sub-model assists in training the generating sub-model according to the difference between the processed sample image and the true value image at the sample significant pixel area and / or the difference between the processed sample image and the true value image at the sample non-significant pixel area.

[0030] According to a fifth aspect of this embodiment, an electronic device is provided, including:

[0031] processor;

[0032] a memory for storing a computer program executable by the processor;

[0033] The processor is configured to execute the computer program in the memory to implement the steps of the method as described in any one of the first aspect and / or the second aspect.

[0034] According to a sixth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method as described in any one of the first aspect and / or the second aspect are implemented.

[0035] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0036] According to the difference of the pixels in the original image in the preset dimension, the original image is divided into a significant pixel area and a non-significant pixel area in the preset dimension, so as to better reflect the characteristics of the image scene by dividing the areas with different characteristics. In addition, different image enhancement processing strategies are adopted for the significant pixel area and the non-significant pixel area, which is also conducive to meeting the image enhancement requirements of different areas of the original image.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] Figure 1 is a flowchart of an image processing method shown in the present disclosure according to an exemplary embodiment;

[0040] Figure 2is a structural schematic diagram of an image enhancement model according to an exemplary embodiment of the present disclosure;

[0041] Figure 3 is a flowchart of a training method for an image enhancement model according to an exemplary embodiment of the present disclosure;

[0042] Figure 4 It is a flowchart of dividing a significant pixel area and a non-significant pixel area of ​​a true value image and a sample processed image according to an exemplary embodiment of the present disclosure;

[0043] Figure 5 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure;

[0044] Figure 6 is a block diagram of a training device for an image enhancement model according to an exemplary embodiment of the present disclosure;

[0045] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0047] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0048] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0049] Image enhancement technology can make an originally unclear image clear by improving the color, brightness, and contrast of the image, or it can enhance the scene expression ability of the image by emphasizing the features of the object of interest in the image, suppressing the features of the object of no interest, and expanding the differences between the features of different objects in the image, thereby facilitating subsequent high-level processing tasks such as target recognition.

[0050] Traditional image enhancement techniques often use the same image enhancement strategy for the entire image. However, different areas in an image may have different features and content, and the same image enhancement strategy will have the same impact on all areas of the entire image and cannot be adjusted to the specific needs of each area, which will cause details in some areas to be lost or over-emphasized, thus having a negative effect on image quality.

[0051] The present disclosure provides an image processing method to solve the above problems. Next, the embodiments of the present disclosure are described in detail.

[0052] like Figure 1 As shown, Figure 1 FIG. 1 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure, comprising the following steps:

[0053] Step 101: Based on the difference of each pixel point contained in the original image in a preset dimension, determine the significant pixel area and the non-significant pixel area divided in the original image in the preset dimension.

[0054] The original image may refer to an image that has not been processed or modified in any way, and may be an original form of an image obtained from an image acquisition device (such as a camera or sensor). For example, it may include but is not limited to the following types:

[0055] Low-light images: Low-light images refer to images taken in low-light or poor conditions.

[0056] Depth image: A depth image is an image that contains the distance or depth information of each pixel.

[0057] Color cast images: Color cast of an image refers to the situation where the image appears to be colored due to white balance problems or uneven color of the light source.

[0058] It should be noted that the image processing method disclosed in the present invention is not limited to processing the above-mentioned image types.

[0059] In an illustrated embodiment, the preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

[0060] When processing different types of original images, a suitable preset dimension can be selected according to the scenario-based requirements for image enhancement of the original image or the characteristics of the original image, and based on the differences between the pixels contained in the original image on the preset dimension, the significant pixel area and the non-significant pixel area divided by the original image on the preset dimension can be determined, so that different image enhancement processing strategies can be adopted for the divided areas in subsequent steps to meet the image enhancement requirements of areas with different characteristics.

[0061] Take the aforementioned low-light image as an example. If the brightness value of the low-light image is simply adjusted to perform image enhancement processing, since there is no effective distinction between the significant pixel areas (specifically, the illuminated areas) and the non-significant pixel areas (specifically, the non-illuminated areas) in the low-light image, the noise and exposure parts in the illuminated areas of the image are also amplified, causing "light pollution" to the processed image, that is, over-enhanced areas appear in the image, forming bright spots, which make these areas inconsistent with other adjacent areas on the image.

[0062] Therefore, in this step 101, before performing image enhancement processing on the low-light image, the illuminated area and the non-illuminated area in the low-light image can be distinguished, and different image enhancement strategies can be adopted for different areas, so as to meet the image enhancement requirements of areas with different characteristics. Specifically, the brightness value of the pixel point can be selected as the preset dimension, and based on the difference in brightness of each pixel point contained in the low-light image, the illuminated area and the non-illuminated area divided by the brightness of the low-light image can be determined.

[0063] For other image types involved in the above embodiments, the selection process of the preset dimensions is the same as that of the low-light images. The appropriate preset dimensions can be selected according to the scenario requirements of the image enhancement of the original image or the characteristics of the original image. For example, for a depth image, the depth information of the pixels in the depth image can be used as the preset dimension. For another example, for an image with color cast, the pixel value of the pixels in the image can be used as the preset dimension. The above is only an example, and you can also try other preset dimensions and select the optimal preset dimension based on the actual enhancement effect of the image.

[0064] Of course, in addition to the preset dimensions shown in this embodiment, other preset dimensions may be selected as the basis for dividing the significant pixel area and the non-significant pixel area of ​​the original image, for example, texture information, contrast information, etc. of the image. The present disclosure is not limited to this.

[0065] Step 102: Perform image enhancement processing on the significant pixel region and the non-significant pixel region respectively through different image enhancement processing strategies to obtain processed images.

[0066] It should be noted that, in response to the image enhancement processing requirements of different areas of the original image, the user can customize the image enhancement processing strategy of the area to meet the image enhancement processing requirements of different areas of the original image. Of course, the image enhancement model can also be trained through deep learning to identify and process the ability of significant pixel areas and non-significant pixel areas of the original image to meet the image enhancement processing requirements of different areas of the original image. Next, the present disclosure takes the type of original image as a low-light image as an example for detailed description.

[0067] If the user customizes the image enhancement processing strategy to meet the image enhancement requirements of different areas of the original image, after dividing the low-light image into the illuminated area and the non-illuminated area, the corresponding image enhancement processing strategy can be customized according to the characteristics of the illuminated area and the non-illuminated area. For example, since the brightness of the illuminated area is too high, resulting in loss of details and color distortion, an image enhancement processing strategy such as exposure compensation can be applied to perform image enhancement processing on the illuminated area to restore the details of the illuminated area and avoid exposure; since the illumination of the non-illuminated area is weak, an image enhancement processing strategy such as a mean filter, a bilateral filter, etc. can be applied to perform image enhancement processing on the non-illuminated area.

[0068] If deep learning is used, the image enhancement model can be trained to identify and process the salient pixel areas and non-salient pixel areas of the original image to meet the needs of image enhancement processing in different areas of the original image. Figure 2 Specifically, Figure 2 It is a structural schematic diagram of an image enhancement model shown in the present disclosure according to an exemplary embodiment.

[0069] like Figure 2 As shown, the dotted line part represents the model processing process, and the solid line part represents the model training process. The original image 220 can be input into the image enhancement model 20, and the image enhancement model 20 implements the region division and the image enhancement processing as described below. The image enhancement model 20 includes a generation sub-model 230 and a discrimination sub-model 290, wherein:

[0070] The generating sub-model 230 is used to divide the original image 220 into the significant pixel region 231 and the non-significant pixel region 232, and perform image enhancement processing on the significant pixel region 231 and the non-significant pixel region 232 respectively through different image enhancement processing strategies to obtain a processed image;

[0071] The discriminant sub-model 290 is used to assist in training the generative sub-model 230 according to the difference between the sample processed image 250 and the true value image 260 in the sample significant pixel area 270 and / or the difference between the sample processed image 250 and the true value image 260 in the sample non-significant pixel area 280 when the sample original image 210 includes the sample significant pixel area 270 and the sample non-significant pixel area 280 divided according to the preset dimension; wherein the sample processed image 250 is output by the generative sub-model 230 after processing the sample original image 210.

[0072] Specifically, when the sample original image 210 includes a sample significant pixel area 270 and a sample non-significant pixel area 280 divided according to the preset dimension, the sample processed image 250 can be divided into a significant pixel area 272 of the sample processed image and a non-significant pixel area 282 of the sample processed image, and the true value image 260 can be divided into a significant pixel area 271 of the true value image and a non-significant pixel area 281 of the true value image. The salient pixel region 271 of the true value image and the salient pixel region 272 of the sample processed image are input into the first discriminant sub-model 291 of the discriminant sub-model 290, and according to the difference between the sample processed image 250 and the true value image 260 in the sample salient pixel region 270, the sub-model 230 is assisted in training so that the sub-model 230 can identify and process the salient pixel region 231 of the original image 220; and the non-significant pixel region 281 of the true value image and the non-significant pixel region 282 of the sample processed image are input into the second discriminant sub-model 292 of the discriminant sub-model 290, and according to the difference between the sample processed image 250 and the true value image 260 in the sample non-significant pixel region 280, the sub-model 230 is assisted in training so that the sub-model 230 can identify and process the non-significant pixel region 232 of the original image 220.

[0073] Of course, the generation sub-model 230 may be assisted in training only according to the difference between the sample processed image 250 and the true value image 260 in the sample significant pixel region 270, so that the generation sub-model 230 can identify and process the significant pixel region 231; or the generation sub-model 230 may be assisted in training only according to the difference between the sample processed image 250 and the true value image 260 in the sample non-significant pixel region 280, so that the generation sub-model 230 can identify and process the non-significant pixel region 232.

[0074] In an illustrated embodiment, a generative adversarial network can be directly used as Figure 2 The image enhancement model 20, where:

[0075] The generation sub-model 230 is a generator in the generative adversarial network;

[0076] The discriminant sub-model includes a first discriminant sub-model 291 and a second discriminant sub-model 292; the first discriminant sub-model 291 is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model 230 according to the difference between the sample processed image 250 and the true value image 260 in the sample significant pixel area 270; the second discriminant sub-model 292 is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model 230 according to the difference between the sample processed image 250 and the true value image 260 in the sample non-significant pixel area 280.

[0077] Since the Generative Adversarial Network is a widely studied and verified network architecture, it has achieved remarkable success in various tasks such as image generation, image editing, and generative models. Therefore, adopting the existing Generative Adversarial Network network architecture can save development time and resources, and has been proven to be effective in many cases, with better performance and stability.

[0078] Of course, technicians in this field can also choose to customize Figure 2 The network architecture of the generation sub-model 230 and the discrimination sub-model 290 of the image enhancement model 20 is used to implement the logic of the image enhancement model 20.

[0079] Specifically, the neural network architecture adopted by the generative submodel 230 and the discriminative submodel 290 of the above-mentioned image enhancement model 20 can be customized. For example, the network architecture of the generative submodel 230 and the discriminative submodel 290 can adopt the network architecture of a traditional convolutional neural network, which is composed of an input layer, a hidden layer composed of a plurality of convolutional layers and an activation layer, and an output layer. The generative submodel 230 and the discriminative submodel 290 can be alternately trained based on the customized loss function of the generative submodel 230 and the loss function of the discriminative submodel 290, so that the discriminative submodel 290 can assist in training the generative submodel 230 to recognize and process the significant pixel area 231 and the non-significant pixel area 232 of the original image 220.

[0080] In this embodiment, according to the difference between the pixels in the original image in the preset dimension, the original image is divided into a significant pixel area and a non-significant pixel area in the preset dimension, so as to better reflect the characteristics of the image scene by dividing the areas with different characteristics. In addition, different image enhancement processing strategies are adopted for the significant pixel area and the non-significant pixel area, which is also conducive to meeting the image enhancement requirements of different areas of the original image.

[0081] Next, the training process of the image enhancement model provided by the present disclosure is introduced. Figure 3 As shown, Figure 3 This is a flow chart of a training method for an image enhancement model according to an exemplary embodiment of the present disclosure. The specific steps are as follows:

[0082] Step 301: Obtain a sample original image and a true value image corresponding to the sample original image.

[0083] The source of the sample original image can be a public data set, or a personalized data set collected by the user through acquisition equipment, web crawlers or other means to meet specific needs. For example, low-contrast image samples are specially collected to train the model's processing ability to process low-contrast images; or blurred image samples are specially collected to train the model's processing ability to process blurred images.

[0084] If the source of the sample original image is a public dataset, the true value image corresponds to the label image provided in the public dataset. If the source of the sample original image is an image sample collected by the user, the sample original image can be manually annotated by experts or annotators, and the standard for image enhancement can be defined. Based on the standard, the quality of the sample original image can be manually adjusted or optimized to generate a true value image corresponding to the sample original image.

[0085] Step 302: input the sample original image into an image enhancement model, so that the generation sub-model in the image enhancement model divides the sample original image into a sample significant pixel region and a sample non-significant pixel region based on the difference of each pixel point contained in the sample original image in a preset dimension, and performs image enhancement processing on the sample significant pixel region and the sample non-significant pixel region respectively through different image enhancement processing strategies to obtain a sample processed image.

[0086] In an illustrated embodiment, the preset dimension includes at least one of the following: the brightness value of the pixel, the pixel value of the pixel, and the depth information of the pixel. The user can select different preset dimensions according to the application scenario of the image or the characteristics of the image. For example, when it is necessary to perform image enhancement processing on a low-light image, the user can select the brightness value of the pixel as the preset dimension to train the image enhancement model to identify the illuminated area and the non-illuminated area of ​​the image, and adopt different image enhancement processing strategies for the characteristics of the two areas. For another example, in the field of image segmentation, the colors of the same area are often similar, while the colors of different areas are quite different. Therefore, the user can use the pixel value of the pixel as the preset dimension to train the image enhancement model to identify different types of areas of the image, and adopt different image enhancement processing strategies for the characteristics of different areas.

[0087] Of course, in addition to the preset dimensions shown in this embodiment, other preset dimensions may be selected as the basis for dividing the significant pixel area and the non-significant pixel area of ​​the original image, for example, texture information, contrast information of the image, etc. The present disclosure is not limited to this.

[0088] Step 303: input the image data of the processed sample image and the true value image corresponding to the sample significant pixel area and / or the sample non-significant pixel area into the discriminant sub-model in the image enhancement model, so that the discriminant sub-model assists in training the generation sub-model according to the difference between the processed sample image and the true value image in the sample significant pixel area and / or the difference between the processed sample image and the true value image in the sample non-significant pixel area.

[0089] In an illustrated embodiment, the image enhancement model is a generative adversarial network, wherein:

[0090] The generative sub-model is a generator in the generative adversarial network;

[0091] The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

[0092] In an illustrated embodiment, the parameters of the generative sub-model are updated according to the difference between the sample processed image and the true value image. The difference between the sample processed image and the true value image can be represented by the following loss function of the generative sub-model and the loss function of the discriminative sub-model:

[0093] (1) Generate the loss function of the sub-model, using mean square error loss Indicates, including:

[0094]

[0095] Among them, W and H represent the width and height of the original sample image, I real represents the true value image, I out Represents the sample image after processing.

[0096] (2) The loss function of the discriminant sub-model can be divided into the loss function of the first discriminant sub-model And the loss function of the second discriminant model to They are expressed by the following formulas:

[0097]

[0098]

[0099] Among them, I sig_real and I sig_out Respectively represent the salient pixel area of ​​the true value image and the salient pixel area of ​​the processed image, I nonsig_real and I nonsig_out Denote the non-significant areas of the true image and the non-significant areas of the processed image, respectively. D1(*) and D2(*) denote the probability values ​​output by the first discriminant sub-model and the second discriminant sub-model, respectively.

[0100] Next, the present disclosure introduces the steps of obtaining the significant pixel regions and the non-significant pixel regions of the true value image, and the significant pixel regions and the non-significant pixel regions of the sample processed image in the training stage of the image enhancement model.

[0101] like Figure 4 As shown, Figure 4 This is a flow chart of dividing the significant pixel area and the non-significant pixel area of ​​the true value image and the sample processed image according to an exemplary embodiment of the present disclosure. The specific steps are as follows:

[0102] Step 401: Perform saliency detection on the sample original image according to the selected preset dimension to obtain a saliency map.

[0103] When processing different types of sample original images, we can first select different preset dimensions according to the scenario requirements of image enhancement or the characteristics of the sample original images, and based on the selected preset dimensions, select the corresponding saliency detection algorithm to perform saliency detection on the sample original images to obtain a saliency map.

[0104] For example, if the type of the sample original image is a low-light image, then based on the characteristics of the sample low-light image, the brightness value of the pixel is selected as the preset dimension, and saliency detection is performed on the sample low-light image to obtain a saliency map. Next, taking a low-light image as an example, how to use the saliency detection algorithm to perform saliency detection on a sample low-light image to obtain a saliency map is introduced:

[0105] First, the brightness value of each pixel of the sample low-light image is obtained, and the saliency value of each pixel is calculated using the following formula to generate a saliency map:

[0106]

[0107] Among them, I inRepresents a matrix composed of the brightness values ​​of the pixels of the sample low-light image, k is the brightness value of the currently calculated pixel, and i is the brightness value of all other pixels on the sample low-light image.

[0108] Secondly, by calculating the difference between the brightness value of pixel k and the brightness values ​​of all other pixel i on the image, the saliency value S(k) of pixel k is obtained. The calculation steps of the saliency values ​​of other pixels are the same as above, and finally a saliency map is obtained.

[0109] For other types of sample original images, a saliency detection algorithm corresponding to the preset dimension may be selected according to the selected preset dimension, and the selected saliency detection algorithm may be used to perform saliency detection on the sample original image to obtain a saliency map.

[0110] For example, if the type of the sample original image is a depth image, the depth information of the pixels in the depth image can be used as a preset dimension to perform saliency detection on the sample depth image. By calculating the depth difference of the pixels in the sample depth image, the areas with larger depth are marked as significant pixel areas to obtain a saliency map.

[0111] For another example, if the type of the sample original image is a color-biased image, the pixel values ​​of the pixels in the color-biased image can be used as preset dimensions to perform saliency detection on the color-biased image. The significant pixel areas are determined by calculating the color differences between different areas in the color-biased image to obtain a saliency map.

[0112] Since the preset dimensions are different, the saliency detection algorithm corresponding to the preset dimensions also changes accordingly. Therefore, the present disclosure does not impose any limitation on the selected saliency detection algorithm.

[0113] Step 402: Binarize the saliency map to obtain a saliency mask and a non-saliency mask.

[0114] The mask is a binary image or a two-dimensional array consisting of 0 and 1, and has the same size as the sample original image. By performing element-by-element logical calculations on the mask and the sample original image, specific areas of the sample original image can be selected or excluded. For example, when the mask value is 1, it means that the pixel value at the corresponding position of the sample original image belongs to the significant pixel area; when the mask value is 0, it means that the pixel value at the corresponding position of the sample original image belongs to the non-significant pixel area.

[0115] In this step 402, each saliency value on the saliency map can be compared with a preset threshold value. When the saliency value is greater than the preset threshold value, it indicates that the pixel value at the position of the original sample image corresponding to the saliency value belongs to the saliency pixel region, and the mask value of the corresponding position of the mask is set to 1; otherwise, it indicates that the pixel value at the position of the original sample image corresponding to the saliency value belongs to the non-saliency pixel region, and the mask value of the corresponding position of the mask is set to 0. By converting each saliency value on the saliency map into a mask value in the above manner, a saliency mask is finally obtained. After the mask value of the saliency mask is inverted, a non-saliency mask can be obtained.

[0116] Step 403: Based on the saliency mask and the insignificance mask, respectively obtain the saliency pixel region and the insignificance pixel region of the true value image, and the saliency pixel region and the insignificance pixel region of the sample processed image.

[0117] Specifically, by calculating the product of the saliency mask and the true value image, the salient pixel area of ​​the true value image can be extracted; by calculating the product of the non-salient mask and the true value image, the non-salient pixel area of ​​the true value image can be extracted. Similarly, the salient pixel area and non-salient pixel area of ​​the sample processed image can be obtained.

[0118] For the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0119] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0120] Corresponding to the aforementioned application function implementation method embodiment, the present disclosure also provides an application function implementation device and a corresponding terminal embodiment.

[0121] like Figure 5 As shown, Figure 5 1 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure. The device may include:

[0122] A region determination module 502 is used to determine a significant pixel region and a non-significant pixel region divided into the original image on the preset dimension based on the difference of each pixel point contained in the original image on the preset dimension;

[0123] The image processing module 504 is used to perform image enhancement processing on the significant pixel region and the non-significant pixel region respectively through different image enhancement processing strategies to obtain processed images.

[0124] Optionally, the device further comprises an image input module, configured to input the original image into an image enhancement model; the image enhancement model comprises a generation sub-model and a discrimination sub-model, wherein:

[0125] The generating sub-model is used to divide the original image into the significant pixel region and the non-significant pixel region, and perform image enhancement processing on the significant pixel region and the non-significant pixel region respectively through different image enhancement processing strategies to obtain a processed image;

[0126] The discriminant sub-model is used to assist in training the generative sub-model according to the difference between the sample processed image and the true value image in the sample significant pixel area and / or the difference between the sample processed image and the true value image in the sample non-significant pixel area when the sample original image includes the sample significant pixel area and the sample non-significant pixel area divided according to the preset dimension; wherein the sample processed image is output by the generative sub-model after processing the sample original image.

[0127] Optionally, the image enhancement model is a generative adversarial network, wherein:

[0128] The generative sub-model is a generator in the generative adversarial network;

[0129] The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

[0130] Optionally, the preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

[0131] like Figure 6 As shown, Figure 6 1 is a block diagram of a training device for an image enhancement model according to an exemplary embodiment of the present disclosure. The device may include:

[0132] An image acquisition module 602 is used to acquire a sample original image and a true value image corresponding to the sample original image;

[0133] The sample image processing module 604 is used to input the sample original image into the image enhancement model, so that the generation sub-model in the image enhancement model divides the sample original image into a sample significant pixel region and a sample non-significant pixel region based on the difference of each pixel point contained in the sample original image in a preset dimension, and performs image enhancement processing on the sample significant pixel region and the sample non-significant pixel region respectively through different image enhancement processing strategies to obtain a sample processed image;

[0134] The assisting training module 606 is used to input the image data of the sample processed image and the true value image corresponding to the sample significant pixel area and / or the sample non-significant pixel area into the discriminant sub-model in the image enhancement model, so that the discriminant sub-model assists in training the generating sub-model according to the difference between the sample processed image and the true value image in the sample significant pixel area and / or the difference between the sample processed image and the true value image in the sample non-significant pixel area.

[0135] Optionally, the image enhancement model is a generative adversarial network, wherein:

[0136] The generative sub-model is a generator in the generative adversarial network;

[0137] The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

[0138] Optionally, the preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

[0139] Optionally, the device further comprises a model updating module, configured to update the parameters of the generated sub-model according to the difference between the sample processed image and the true value image.

[0140] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. The electronic device 700 may be a user device, and may specifically be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, a wearable device such as a smart watch, a smart glasses, a smart bracelet, a smart running shoe, etc.

[0141] like Figure 7 As shown, electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0142] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0143] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0144] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0145] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0146] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0147] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0148] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of the components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of contact between the user and the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0149] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0150] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0151] The present disclosure also provides an executable non-temporary computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the aforementioned image processing methods and / or image enhancement model training methods provided by the present disclosure are implemented. The non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0152] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0153] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for processing an image, characterized in that: The method comprises: Based on the difference of each pixel point contained in the original image in the preset dimension, determining the significant pixel area and the non-significant pixel area divided by the original image in the preset dimension; By using different image enhancement processing strategies, image enhancement processing is performed on the significant pixel region and the non-significant pixel region respectively to obtain a processed image.

2. The method according to claim 1, characterized in that Also includes: Inputting the original image into an image enhancement model; The image enhancement model includes a generation sub-model and a discrimination sub-model, wherein: The generating sub-model is used to divide the original image into the significant pixel region and the non-significant pixel region, and perform image enhancement processing on the significant pixel region and the non-significant pixel region respectively through different image enhancement processing strategies to obtain a processed image; The discriminant sub-model is used to assist in training the generative sub-model according to the difference between the sample processed image and the true value image in the sample significant pixel area and / or the difference between the sample processed image and the true value image in the sample non-significant pixel area when the sample original image includes the sample significant pixel area and the sample non-significant pixel area divided according to the preset dimension; wherein the sample processed image is output by the generative sub-model after processing the sample original image.

3. The method according to claim 2, characterized in that The image enhancement model is a generative adversarial network, where: The generative sub-model is a generator in the generative adversarial network; The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

4. The method according to claim 1, characterized in that The preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

5. A training method for an image enhancement model, characterized in that: The method comprises: Acquire a sample original image and a true value image corresponding to the sample original image; The sample original image is input into the image enhancement model, so that the generation sub-model in the image enhancement model divides the sample original image into a sample significant pixel region and a sample non-significant pixel region based on the difference of each pixel point contained in the sample original image in a preset dimension, and performs image enhancement processing on the sample significant pixel region and the sample non-significant pixel region respectively through different image enhancement processing strategies to obtain a sample processed image; The image data of the sample processed image and the true value image corresponding to the sample significant pixel area and / or the sample non-significant pixel area are input into the discriminant sub-model in the image enhancement model, so that the discriminant sub-model assists in training the generation sub-model according to the difference between the sample processed image and the true value image at the sample significant pixel area and / or the difference between the sample processed image and the true value image at the sample non-significant pixel area.

6. The method according to claim 5, characterized in that The image enhancement model is a generative adversarial network, where: The generative sub-model is a generator in the generative adversarial network; The discriminant sub-model includes a first discriminant sub-model and a second discriminant sub-model; the first discriminant sub-model is the first discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample salient pixel area; the second discriminant sub-model is the second discriminator in the generative adversarial network, and is used to assist in training the generative sub-model according to the difference between the processed sample image and the true value image in the sample non-salient pixel area.

7. The method according to claim 5, characterized in that The preset dimension includes at least one of the following: a brightness value of a pixel, a pixel value of a pixel, and depth information of a pixel.

8. The method according to claim 5, characterized in that Also includes: The parameters of the generation sub-model are updated according to the difference between the sample processed image and the true value image.

9. An image processing device, characterized in that: The device comprises: A region determination module, configured to determine a significant pixel region and a non-significant pixel region divided into the original image on the preset dimension based on differences between pixels contained in the original image on the preset dimension; The image processing module is used to perform image enhancement processing on the significant pixel area and the non-significant pixel area respectively through different image enhancement processing strategies to obtain a processed image.

10. A training device for an image enhancement model, characterized in that: The device comprises: An image acquisition module is used to acquire a sample original image and a true value image corresponding to the sample original image; A sample image processing module, used for inputting the sample original image into an image enhancement model, so that the generation sub-model in the image enhancement model divides the sample original image into a sample significant pixel region and a sample non-significant pixel region based on the difference of each pixel point contained in the sample original image in a preset dimension, and performs image enhancement processing on the sample significant pixel region and the sample non-significant pixel region respectively through different image enhancement processing strategies to obtain a sample processed image; an assisting training module, for inputting the image data of the processed sample image and the true value image corresponding to the sample significant pixel area and / or the sample non-significant pixel area into the discriminant sub-model in the image enhancement model, so that the discriminant sub-model assists in training the generating sub-model according to the difference between the processed sample image and the true value image at the sample significant pixel area and / or the difference between the processed sample image and the true value image at the sample non-significant pixel area.

11. An electronic device, characterized in that: include: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images