Image processing method, electronic device, storage medium and computer program product

Abstract

The embodiment of the present invention provides an image processing method, an electronic device, a storage medium and a computer program product. The method comprises: performing frequency domain dimension conversion on the image to be processed to obtain a first spectrum image; inputting the first spectrum image into a pre-trained generator to obtain a second spectrum image; performing inverse frequency domain dimension conversion on the second spectrum image to obtain an image after the image to be processed is adjusted according to the target contrast adjustment parameter. In the embodiment of the present invention, the generator has learned the correspondence between the images before and after contrast adjustment in advance from the frequency domain dimension, and can characterize the change law of the images before and after contrast adjustment in the frequency domain dimension under the premise of keeping the structural features unchanged. Therefore, in the embodiment of the present invention, the pre-trained generator can be used to perform contrast adjustment on the image to be processed from the frequency domain dimension to obtain a more natural and realistic adjusted image.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular to an image processing method, electronic equipment, storage medium, and computer program product. Background Art

[0002] In the field of image processing, whether it is image enhancement, image restoration, or image fusion, contrast adjustment is involved. In layman's terms, contrast refers to the contrast between the brightest white and the darkest black in an image. The larger the difference, the greater the contrast, and the smaller the difference, the smaller the contrast.

[0003] In the related art, the commonly used methods for adjusting the contrast of images are: histogram equalization algorithm CLAHE, adaptive contrast enhancement algorithm ACE, and adaptive histogram equalization algorithm AHE. However, the adjusted images obtained by using these methods for contrast adjustment are prone to the following obvious abnormalities: increased noise in dark areas, brightness inversion, graying of white walls, overexposure in bright areas, etc., which makes the adjusted images incapable of practical application and requires repeated adjustments, reducing image processing efficiency. In addition, the structural features of the adjusted images have changed compared to the images before adjustment, and the adjusted images are not natural and realistic enough.

[0004] It can be seen that there is an urgent need for a new image processing method to overcome or partially overcome the problems existing in traditional technologies. Summary of the invention

[0005] Embodiments of the present invention provide an image processing method, an electronic device, a storage medium, and a computer program product to overcome or partially overcome the problems existing in the related art.

[0006] A first aspect of an embodiment of the present invention provides an image processing method, the method comprising:

[0007] Convert the image to be processed into a frequency domain dimension to obtain a first spectrum image;

[0008] Inputting the first spectrum image into a pre-trained generator to obtain a second spectrum image, wherein the pre-trained generator has pre-learned a correspondence between a spectrum image of a sample to-be-processed image and a spectrum image of a sample processed image, wherein the sample processed image is obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter;

[0009] The second spectrum image is inversely transformed in the frequency domain dimension to obtain an image after the image to be processed is adjusted according to the target contrast adjustment parameter.

[0010] Optionally, after obtaining the image to be processed after adjusting the image according to the target contrast adjustment parameter, the method further includes:

[0011] The specified area of ​​the adjusted image is corrected by a pre-trained correction model to obtain a corrected image. The pre-trained correction model is pre-labeled with the specified area of ​​the sample image to be processed and has learned the correction operation to be performed on the specified area of ​​the sample processed image.

[0012] Optionally, the training step of the generator includes:

[0013] Obtaining a sample image pair, the sample image pair comprising: a sample to-be-processed image and a sample processed image obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter;

[0014] Performing frequency domain conversion on two images in the sample image pair respectively to obtain a sample spectrum image pair;

[0015] The sample spectrum image pairs are used as training sample sets, and the generator and the discriminator in the generative adversarial model are alternately trained to obtain a trained generator.

[0016] Optionally, the training step of the generator includes:

[0017] Obtaining a positive sample image pair, the positive sample image pair comprising: a sample to-be-processed image and a sample processed image obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter;

[0018] Performing frequency domain dimension conversion on two images in the positive sample image pair respectively to obtain a positive sample spectrum image pair;

[0019] Obtaining a negative sample image pair, the negative sample image pair comprising: a first sample to-be-processed image and a sample processed image obtained by adjusting the second sample to-be-processed image according to a target contrast adjustment parameter, the first sample to-be-processed image and the second sample to-be-processed image being two groups of images with different image contents;

[0020] Performing frequency domain conversion on two images in the negative sample image pair respectively to obtain a negative sample spectrum image pair;

[0021] The positive sample spectrum image pair and the negative sample spectrum image pair are used as training sample sets, and the generator and the discriminator in the generative adversarial model are alternately trained to obtain a trained generator.

[0022] Optionally, the discriminator included in the generative adversarial model is a first discriminator; and alternately training the generator and the discriminator in the generative adversarial model to obtain a trained generator includes:

[0023] Inputting the first sample spectrum image in each sample spectrum image pair included in the training sample set into the generator in the generative adversarial model to obtain a simulated sample spectrum image generated by the generator, wherein the first sample spectrum image is the spectrum image of the sample image to be processed;

[0024] Performing an inverse transformation of the simulated sample spectrum image in the frequency domain dimension to obtain a sample simulated image after contrast adjustment of the sample image to be processed;

[0025] Obtaining a structural feature loss value of the generator according to a difference in structural features between the sample simulated image and the sample to-be-processed image;

[0026] Obtaining a spectrum loss value of the generator according to a spectrum difference outputted after the first discriminator discriminates the simulated sample spectrum image and the second sample spectrum image, wherein the second sample spectrum image is a spectrum image of the sample processed image;

[0027] The model parameters of the generator are updated according to the structural feature loss value and the spectrum loss value.

[0028] Optionally, the generative adversarial model further includes a second discriminator; the generator and the discriminator in the generative adversarial model are alternately trained to obtain a trained generator, further comprising:

[0029] Obtaining an image feature loss value of the generator according to a difference in image features outputted after the second discriminator discriminates the sample simulated image and the sample to-be-processed image;

[0030] The method includes updating the model parameters of the generator according to the structural feature loss value and the spectrum loss value, including:

[0031] The model parameters of the generator are updated according to the structural feature loss value, the spectrum loss value and the image feature loss value.

[0032] Optionally, the training step of the correction model includes:

[0033] Obtaining a sample image group, the sample image group comprising: a sample image to be processed, a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter, and a sample image to be corrected after contrast adjustment is performed on the sample image to be processed;

[0034] Inputting the sample image to be corrected into a preset model to obtain a sample corrected image output by the preset model;

[0035] Obtaining a first loss value of the preset model according to an image feature difference between a designated area of ​​the sample to-be-processed image and an image area corresponding to the designated area in the sample corrected image;

[0036] Obtaining a second loss value of the preset model according to an image feature difference between an image region corresponding to the designated region in the sample corrected image and an image region corresponding to the designated region in the sample processed image;

[0037] Updating the preset model parameters according to the first loss value and the second loss value;

[0038] The trained preset model is determined as the revised model.

[0039] Optionally, the sample image to be corrected after contrast adjustment is performed on the sample image to be processed is generated by using a pre-trained generator according to the following steps:

[0040] Convert the sample image to be processed into a frequency domain dimension to obtain a first sample spectrum image;

[0041] Inputting the first sample spectrum image into a pre-trained generator to obtain a third sample spectrum image;

[0042] The third sample spectrum image is inversely transformed in frequency domain dimension to obtain the sample image to be corrected.

[0043] A second aspect of an embodiment of the present invention provides an image processing device, the device comprising:

[0044] A conversion module, used for converting the image to be processed into a frequency domain dimension to obtain a first spectrum image;

[0045] An input module, used for inputting the first spectrum image into a pre-trained generator to obtain a second spectrum image, wherein the pre-trained generator has pre-learned the correspondence between the spectrum image of the sample to-be-processed image and the spectrum image of the sample processed image, wherein the sample processed image is obtained by adjusting the sample to-be-processed image according to the target contrast adjustment parameter;

[0046] The inverse conversion module is used to perform an inverse conversion of the second spectrum image in the frequency domain dimension to obtain an image after the image to be processed is adjusted according to the target contrast adjustment parameter.

[0047] Optionally, the device further comprises:

[0048] The correction module is used to correct the specified area of ​​the adjusted image through a pre-trained correction model to obtain a corrected image. The pre-trained correction model is pre-labeled with the specified area of ​​the sample image to be processed and has learned the correction operation to be performed on the specified area of ​​the sample processed image.

[0049] Optionally, the training process of the generator is completed by the following modules:

[0050] A sample image pair determination module is used to obtain a sample image pair, wherein the sample image pair includes: a sample image to be processed and a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter;

[0051] A sample image conversion module, used to convert the two images in the sample image pair in frequency domain dimensions respectively to obtain a sample spectrum image pair;

[0052] The training module is used to use the sample spectrum image pairs as training sample sets to alternately train the generator and the discriminator in the generative adversarial model to obtain a trained generator.

[0053] Optionally, the training process of the generator is completed by the following modules:

[0054] A positive sample image pair determination module is used to obtain a positive sample image pair, wherein the positive sample image pair includes: a sample image to be processed and a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter;

[0055] A positive sample image conversion module, used to convert the two images in the positive sample image pair in frequency domain dimensions respectively to obtain a positive sample spectrum image pair;

[0056] a negative sample image pair determination module, configured to obtain a negative sample image pair, wherein the negative sample image pair comprises: a first sample image to be processed and a sample processed image obtained by adjusting the second sample image to be processed according to a target contrast adjustment parameter, wherein the first sample image to be processed and the second sample image to be processed are two groups of images with different image contents;

[0057] A negative sample image conversion module, used to convert the two images in the negative sample image pair in frequency domain dimensions to obtain a negative sample spectrum image pair;

[0058] The training module is used to alternately train the generator and the discriminator in the generative adversarial model using the positive sample spectrum image pair and the negative sample spectrum image pair as training sample sets to obtain a trained generator.

[0059] Optionally, the discriminator included in the generative adversarial model is a first discriminator; and the training module includes:

[0060] An input submodule, used for inputting the first sample spectrum image in each sample spectrum image pair included in the training sample set into the generator in the generative adversarial model to obtain a simulated sample spectrum image generated by the generator, wherein the first sample spectrum image is the spectrum image of the sample image to be processed;

[0061] An inverse conversion submodule is used to perform an inverse conversion of the simulated sample spectrum image in the frequency domain dimension to obtain a sample simulated image after contrast adjustment of the sample image to be processed;

[0062] A structural feature loss value determination submodule, used for obtaining a structural feature loss value of the generator according to a difference in structural features between the sample simulated image and the sample to-be-processed image;

[0063] a spectrum loss value determination submodule, configured to obtain a spectrum loss value of the generator according to a spectrum difference outputted after the first discriminator discriminates the simulated sample spectrum image and a second sample spectrum image, wherein the second sample spectrum image is a spectrum image of the sample processed image;

[0064] An updating submodule is used to update the model parameters of the generator according to the structural feature loss value and the spectrum loss value.

[0065] Optionally, the generative adversarial model further includes a second discriminator; and the training module further includes:

[0066] An image feature loss value determination submodule, used to obtain the image feature loss value of the generator according to the difference in image features output after the second discriminator discriminates the sample simulated image and the sample to-be-processed image;

[0067] The updating submodule is specifically used for:

[0068] The model parameters of the generator are updated according to the structural feature loss value, the spectrum loss value and the image feature loss value.

[0069] Optionally, the training process of the correction model is completed by the following modules:

[0070] A sample image group determination module is used to obtain a sample image group, wherein the sample image group includes: a sample image to be processed, a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter, and a sample image to be corrected after contrast adjustment is performed on the sample image to be processed;

[0071] A sample image to be corrected input module is used to input the sample image to be corrected into a preset model to obtain a sample corrected image output by the preset model;

[0072] A first loss value determination module, used for obtaining a first loss value of the preset model according to an image feature difference between a designated area of ​​the sample to-be-processed image and an image area corresponding to the designated area in the sample corrected image;

[0073] A second loss value determination module, used for obtaining a second loss value of the preset model according to an image feature difference between an image region corresponding to the designated region in the sample corrected image and an image region corresponding to the designated region in the sample processed image;

[0074] An updating module, configured to update the preset model parameters according to the first loss value and the second loss value;

[0075] The correction module determines the module, which is used to determine the trained preset model as the correction model.

[0076] Optionally, the device further comprises:

[0077] A first sample spectrum image determination module is used to convert the sample image to be processed into a frequency domain dimension to obtain a first sample spectrum image;

[0078] A third sample spectrum image determination module, configured to input the first sample spectrum image into a pre-trained generator to obtain a third sample spectrum image;

[0079] The sample image to be corrected determining module is used to perform an inverse transformation of the third sample spectrum image in the frequency domain dimension to obtain the sample image to be corrected.

[0080] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the image processing method as described in the first aspect of the present invention.

[0081] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the image processing method as described in the first aspect of the present invention is implemented.

[0082] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the image processing method described in the first aspect of the present invention when executed by a processor.

[0083] In the embodiment of the present invention, the generator has learned the correspondence between the images before and after contrast adjustment from the frequency domain dimension in advance, and can characterize the change law of the images before and after contrast adjustment in the frequency domain dimension under the premise of keeping the structural features unchanged. Therefore, in the embodiment of the present invention, the pre-trained generator can be used to perform contrast adjustment on the image to be processed from the frequency domain dimension to obtain a more natural and realistic adjusted image. Moreover, in the embodiment of the present invention, the pre-trained generator is used to perform contrast adjustment on the image to be processed from the frequency domain dimension, which can avoid obvious abnormalities such as increased noise in dark areas, brightness inversion, graying of white walls, overexposure of bright areas, etc. in the adjusted image, thereby improving the image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0085] Figure 1 is a flow chart of an image processing method according to an embodiment of the present invention;

[0086] Figure 2 is a flowchart of a training method for a generator in an image processing method according to an embodiment of the present invention;

[0087] Figure 3 is a flow chart of another image processing method according to an embodiment of the present invention;

[0088] Figure 4 is a flowchart of a training method for a correction model in an image processing method according to an embodiment of the present invention;

[0089] Figure 5 is a schematic diagram of a model training process in an image processing method according to an embodiment of the present invention;

[0090] Figure 6 It is a structural block diagram of an image processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0092] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0093] Among them, in the image processing related technology, the contrast of the image is usually adjusted from the dimension of the histogram. However, the applicant found that the adjusted image obtained by contrast adjustment based on the histogram has the problem of not being natural and realistic enough.

[0094] To this end, the applicant proposed: from the frequency domain dimension of the image, analyze the change pattern of the frequency spectrum of two images with the same structural features before and after contrast adjustment. In addition, the applicant further proposed to use the generative adversarial model to learn the change pattern of the frequency spectrum of the image after contrast enhancement compared with the frequency spectrum of the original image before contrast enhancement. Based on this, the inventor proposed a technical concept: adjust the contrast of the image from the frequency domain dimension combined with the generative adversarial model. Based on this inventive concept, the applicant specifically proposed the following technical solutions:

[0095] Reference Figure 1 , shows a flow chart of an image processing method according to an embodiment of the present invention, the method comprising the following steps:

[0096] S101, converting the image to be processed into a frequency domain dimension to obtain a first spectrum image.

[0097] In the embodiment of the present invention, the image may be converted according to any feasible method in the related art to obtain a spectrum image corresponding to the image to be processed. For example, the image to be processed may be subjected to Fourier transform to obtain its spectrum image.

[0098] S102: Input the first spectrum image into a pre-trained generator to obtain a second spectrum image.

[0099] In an embodiment of the present invention, the pre-trained generator pre-learns the correspondence between the spectrum image of the sample image to be processed and the spectrum image of the sample processed image, and the sample processed image is obtained after adjusting the sample image to be processed according to the target contrast adjustment parameter.

[0100] In the embodiment of the present invention, the generator has learned the correspondence between the images before and after contrast adjustment in the frequency domain dimension in advance, and can characterize the change law of the images before and after contrast adjustment in the frequency domain dimension. Therefore, in the embodiment of the present invention, inputting the first spectrum image into the generator can simulate the spectrum diagram of the image after contrast adjustment. For the specific generator training method, please refer to the following description.

[0101] S103, performing inverse transformation of the second spectrum image in the frequency domain dimension to obtain an image after adjusting the image to be processed according to the target contrast adjustment parameter.

[0102] In an embodiment of the present invention, the frequency domain dimension of the image to be processed is first converted to obtain a first spectrum image, and then the first spectrum image is processed using a pre-trained generator to obtain a second spectrum image. Finally, the second spectrum image is inversely converted in the frequency domain dimension to obtain an image after the image to be processed is adjusted according to the target contrast adjustment parameter.

[0103] Therefore, in the embodiment of the present invention, the pre-trained generator can be used to adjust the contrast of the image to be processed from the frequency domain dimension to obtain a more natural and realistic adjusted image. This avoids obvious abnormalities such as increased noise in dark areas, brightness inversion, graying of white walls, and overexposure of bright areas in the adjusted image, thereby improving image processing efficiency.

[0104] In an optional embodiment of the present invention, a training method for a generator is provided, such as Figure 2 As shown, the training method comprises the following steps:

[0105] S201, obtaining a sample image pair, wherein the sample image pair comprises: a sample to-be-processed image and a sample processed image obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter.

[0106] In the embodiment of the present invention, a large number of original images can be processed according to the target contrast adjustment parameter to obtain processed images, and then relatively natural and realistic processed images are screened out as sample processed images, and the original images corresponding to the sample processed images are used as sample images to be processed, thereby obtaining sample image pairs.

[0107] In the embodiment of the present invention, a large number of original images are processed according to the same target contrast adjustment parameter, and there is no need to perform contrast adjustment on each original image separately, which can reduce the workload of early data preparation and improve efficiency.

[0108] S202 , performing frequency domain dimension conversion on two images in the sample image pair respectively to obtain a sample spectrum image pair.

[0109] In the embodiment of the present invention, the method for converting the sample image into a frequency domain dimension is the same as the method for converting the image to be processed into a frequency domain in the above step S101.

[0110] S203, using the sample spectrum image pairs as training sample sets, alternately training the generator and the discriminator in the generative adversarial model to obtain a trained generator.

[0111] In an embodiment of the present invention, each image pair in the training sample set can be input into the generator and discriminator of the generative adversarial model respectively, so that the generator learns the correspondence between the spectrum image of the sample to be processed and the spectrum image of the sample processed image.

[0112] In an optional implementation, the discriminator included in the generative adversarial model is a first discriminator, and step S203 includes:

[0113] S2031, input the first sample spectrum image in each sample spectrum image pair included in the training sample set into the generator in the generative adversarial model to obtain a simulated sample spectrum image generated by the generator, wherein the first sample spectrum image is the spectrum image of the sample image to be processed.

[0114] S2032, performing an inverse transformation of the simulated sample spectrum image in the frequency domain dimension to obtain a sample simulated image after contrast adjustment is performed on the sample image to be processed.

[0115] In the embodiment of the present invention, during the training process of the generative adversarial model, the generator may process the spectrum image of the input sample to-be-processed image to obtain a simulated sample spectrum image, and then inversely transform the simulated sample spectrum image in the frequency domain dimension to obtain a sample simulated image.

[0116] In the embodiment of the present invention, the specific method of inverse conversion corresponds to the frequency domain conversion method used in the aforementioned step S202. For example, if the frequency domain conversion method used in step S202 is Fourier transform, then in this step, the simulated sample spectrum image is subjected to inverse Fourier transform to obtain a sample simulated image.

[0117] S2033: Obtain a structural feature loss value of the generator according to a difference in structural features between the sample simulated image and the sample image to be processed.

[0118] S2034, obtaining a spectrum loss value of the generator according to a spectrum difference outputted after the first discriminator discriminates the simulated sample spectrum image and the second sample spectrum image, wherein the second sample spectrum image is a spectrum image of the sample processed image.

[0119] S2035: Update the model parameters of the generator according to the structural feature loss value and the spectrum loss value.

[0120] In an embodiment of the present invention, during the training process of the generator, the difference in structural features between the sample simulated image and the sample image to be processed is taken into account, and the structural feature loss value of the generator is determined. Based on the structural feature loss value, during the process of updating the model parameters of the generator, the structural feature loss value is taken into account when calculating the total loss value, thereby ensuring that the structural features of the image generated by the trained generator and the input image are basically the same, thereby ensuring that the image after contrast adjustment is as natural and realistic as possible.

[0121] In an embodiment of the present invention, during the training process of the generator, the spectral difference output by the first discriminator after discriminating the spectral image of the simulated sample and the spectral image of the sample processed image is also considered to determine the spectral loss value of the generator, so that in the process of updating the model parameters of the generator, the spectral loss value is taken into account when calculating the total loss value, so that the simulated sample spectral image generated by the trained generator can be the same as the spectral image of the sample processed image.

[0122] In an optional implementation, the generative adversarial model further includes a second discriminator, which is used to discriminate the difference between the image generated by the generator and the sample image from the image feature dimension. In this case, step S203 further includes:

[0123] S2036, obtaining the image feature loss value of the generator according to the difference in image features outputted after the second discriminator discriminates the sample simulated image and the sample image to be processed.

[0124] Step S2036 is performed between step S2032 and step S2035.

[0125] In this case, S2035 includes: updating the model parameters of the generator according to the structural feature loss value, the spectrum loss value and the image feature loss value.

[0126] In an embodiment of the present invention, during the training process of the generator, a second discriminator is used to discriminate the authenticity of the image generated by the generator from the dimension of image features, thereby continuously training the generator, further ensuring that the image features of the image generated by the trained generator and the input image are basically the same, thereby ensuring that the image after contrast adjustment is as natural and realistic as possible.

[0127] In the embodiment of the present invention, the sample image pairs in the training sample set can also be split and disrupted, and the non-corresponding images to be processed and processed images can be combined to obtain new sample image pairs, which are added to the training sample set to obtain a new training sample set, and the generator and the discriminator in the generative adversarial model are alternately trained using the new training sample set. In this case, the embodiment of the present invention provides another generator training method, which specifically includes the following steps:

[0128] S211, obtaining a positive sample image pair, wherein the positive sample image pair includes: a sample to-be-processed image and a sample processed image obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter.

[0129] S212 performs frequency domain dimension conversion on the two images in the positive sample image pair respectively to obtain a positive sample spectrum image pair.

[0130] The above steps S211-S3212 are similar to the above steps S201-S202, except that steps S301-S302 are aimed at obtaining a positive sample image pair by combining a sample to-be-processed image and a sample processed image with the same image content. Other descriptions of the embodiments of the present invention are not repeated here.

[0131] S213, obtaining a negative sample image pair, wherein the negative sample image pair includes: a first sample image to be processed and a sample processed image obtained by adjusting the second sample image to be processed according to a target contrast adjustment parameter, wherein the first sample image to be processed and the second sample image to be processed are two groups of images with different image contents.

[0132] S214, performing frequency domain dimension conversion on the two images in the negative sample image pair respectively to obtain a negative sample spectrum image pair.

[0133] The above steps S303-S304 are similar to the above steps S201-S202, except that steps S204-S205 are aimed at obtaining a negative sample image pair by combining a first sample to-be-processed image and a second sample processed image with different image contents. Other descriptions of the embodiment of the present invention are not repeated here.

[0134] S215, using the positive sample spectrum image pair and the negative sample spectrum image pair as training sample sets, alternately training the generator and the discriminator in the generative adversarial model to obtain a trained generator.

[0135] Step S215 is similar to the above step 203 and will not be described in detail in this embodiment of the present invention.

[0136] In an embodiment of the present invention, the sample image pairs in the training sample set are split and shuffled, the negative sample image pairs obtained by combining the to-be-processed image and the processed image with the same image content, and the negative sample image pairs obtained by combining the to-be-processed image and the processed image with different image content, so as to obtain richer sample image pairs, and the generator is alternately trained using the positive sample image pairs and the negative sample image pairs, so as to enhance the stability of the trained generator.

[0137] Reference Figure 3 , shows a flow chart of another image processing method according to an embodiment of the present invention, the method comprising the following steps:

[0138] S301, converting the image to be processed into a frequency domain dimension to obtain a first spectrum image.

[0139] S302: Input the first spectrum image into a pre-trained generator to obtain a second spectrum image.

[0140] The pre-trained generator pre-learns the correspondence between the spectrum image of the sample to-be-processed image and the spectrum image of the sample processed image, wherein the sample processed image is obtained by adjusting the sample to-be-processed image according to the target contrast adjustment parameter.

[0141] S303: Perform an inverse transformation of the second spectrum image in the frequency domain dimension to obtain an image after adjusting the image to be processed according to the target contrast adjustment parameter.

[0142] The steps S301 - S303 are similar to the steps S101 - S303 and are not described again here.

[0143] S304, correcting the designated area of ​​the adjusted image using a pre-trained correction model to obtain a corrected image.

[0144] In the embodiment of the present invention, the pre-trained correction model uses the designated area of ​​the sample image to be processed as a label in advance, and learns the correction operation to be performed on the designated area of ​​the sample processed image.

[0145] Optionally, the correction model mainly corrects the highlight and dark areas of the adjusted image, which can reduce the noise caused by brightening the dark areas and the phenomenon of suppressing the highlight areas. In this case, the highlight and dark areas of the sample image to be processed are used as designated areas, and the pre-trained correction model can be pre-labeled with the designated areas to learn the correction operations for the highlight and dark areas of the sample processed image.

[0146] Specifically, the highlight and dark areas of the sample image to be processed can be manually annotated, or the highlight and dark areas of the image can be determined according to a preset brightness threshold. For example, an area with a brightness greater than 240 is considered a highlight area, and an area with a brightness less than 20 is considered a dark area. After determining the highlight and dark areas of the sample image to be processed, a corresponding Mask is obtained, and the Mask is used to determine the corresponding highlight and dark areas from the sample processed image, so that the correction model can learn the correction operation performed on the sample processed image based on the image feature differences between the highlight and dark areas of the sample image to be processed and the highlight and dark areas in the sample processed image. In an embodiment of the present invention, the image feature difference may include an image structure feature difference and an image brightness difference, wherein the image structure feature difference mainly exists in the dark area, and the image brightness difference mainly exists in the highlight area. Therefore, in an embodiment of the present invention, the correction model can correct the brightness of the highlight area of ​​the adjusted image and correct the structural features of the dark area.

[0147] In the embodiment of the present invention, the training step of the correction model may specifically include:

[0148] S401, obtaining a sample image group, wherein the sample image group includes: a sample image to be processed, a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter, and a sample image to be corrected after contrast adjustment is performed on the sample image to be processed.

[0149] In the embodiment of the present invention, the sample image to be processed and the sample processed image may be the aforementioned sample image pair. In the embodiment of the present invention, the sample image to be corrected may be obtained by performing contrast adjustment on the sample image to be processed, and the sample processed image may be obtained by correcting the sample image to be corrected. Any contrast adjustment method in the related art may be used to perform contrast adjustment on the sample image to be processed.

[0150] S402, inputting the sample image to be corrected into a preset model to obtain a sample corrected image output by the preset model.

[0151] In the embodiment of the present invention, the preset model may be any neural network model in the related art that can implement image brightness processing.

[0152] S403, obtaining a first loss value of the preset model according to an image feature difference between a designated area of ​​the sample to-be-processed image and an image area corresponding to the designated area in the sample corrected image.

[0153] S404, obtaining a second loss value of the preset model according to an image feature difference between an image region corresponding to the designated region in the sample corrected image and an image region corresponding to the designated region in the sample processed image.

[0154] S405: Update the preset model parameters according to the first loss value and the second loss value.

[0155] In an embodiment of the present invention, during the training process of the preset model, the first loss value of the preset model is determined taking into account the difference in image features between the sample corrected image and the sample image to be processed. During the updating process of the model parameters, when calculating the total loss value, the first loss value is taken into account, thereby ensuring that the image features of the image output by the preset model and the sample image to be corrected are basically the same, thereby ensuring that the corrected image is as natural and realistic as possible.

[0156] In an embodiment of the present invention, during the training process of the preset model, the second loss value of the preset model is determined taking into account the difference in image features between the sample corrected image and the sample processed image. During the updating process of the model parameters, when calculating the total loss value, the second loss value is taken into account, so that the image features of the sample corrected image obtained by the trained correction model can be the same as those of the sample processed image.

[0157] S406: Determine the trained preset model as the revised model.

[0158] In an embodiment of the present invention, the correction model obtained through training can be used to correct the highlight and dark areas of the simulated image generated and converted by the generator, thereby reducing the noise caused by brightening the dark areas and the phenomenon of blackening the highlight areas, and obtaining a more natural and realistic processed image.

[0159] In an optional implementation manner, the sample image to be corrected after contrast adjustment is performed on the sample image to be processed is generated by using a pre-trained generator according to the following steps:

[0160] S1, converting the sample image to be processed in frequency domain dimension to obtain a first sample spectrum image.

[0161] S2: Input the first sample spectrum image into a pre-trained generator to obtain a third sample spectrum image.

[0162] S3, performing an inverse transformation of the third sample spectrum image in the frequency domain dimension to obtain the sample image to be corrected.

[0163] In this case, in the embodiment of the present invention, the generator can be trained first, and after the training is completed, the model parameters of the generator can be fixed, and then the sample image to be processed and the sample processed image can be used to train the correction image. So that the correction network can learn the correction operation on the brightness of the sample processed image based on the sample image to be processed, the sample image to be corrected generated and converted by the generator, and the sample processed image. For ease of understanding, the following is combined with Figure 5 The model training method in an image processing method provided by an embodiment of the present invention is further explained:

[0164] Figure 5 middle, I ori represents the sample image to be processed, I1 represents the sample image to be corrected, I2 represents the corrected image, and GT represents the sample processed image.

[0165] Combination Figure 5 It can be seen that in the embodiment of the present invention, the sample image to be processed and the sample image after processing can be Fourier transformed respectively to obtain corresponding spectrum images S1 and S3. The generator processes the input S1 and outputs S2. The discriminator discriminates S2 output by the generator based on S3 until S2 generated by the generator is the same as S3, and the training of the generator is completed.

[0166] After completing the training of the generator, fix the model parameters of the generator, perform Fourier transform on the sample image to be processed, and the corresponding spectrum image S1 is processed by the generator to obtain S2. Perform inverse Fourier transform on S2 to obtain the sample image to be corrected I1. The sample image to be corrected is input into the correction model, and the correction model is trained with the sample image to be processed GT as the target until the trained correction model is obtained.

[0167] Based on the same inventive concept, an embodiment of the present invention provides an image processing device. Figure 6 , Figure 6 Schematic diagram of an image processing device provided by an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0168] The conversion module 601 is used to convert the image to be processed into a frequency domain dimension to obtain a first spectrum image;

[0169] An input module 602 is used to input the first spectrum image into a pre-trained generator to obtain a second spectrum image, wherein the pre-trained generator has pre-learned the correspondence between the spectrum image of the sample to-be-processed image and the spectrum image of the sample processed image, wherein the sample processed image is obtained by adjusting the sample to-be-processed image according to the target contrast adjustment parameter;

[0170] The inverse conversion module 603 is used to perform inverse conversion of the second spectrum image in the frequency domain dimension to obtain an image after the image to be processed is adjusted according to the target contrast adjustment parameter.

[0171] Optionally, the device further comprises:

[0172] The correction module is used to correct the specified area of ​​the adjusted image through a pre-trained correction model to obtain a corrected image. The pre-trained correction model is pre-labeled with the specified area of ​​the sample image to be processed and has learned the correction operation to be performed on the specified area of ​​the sample processed image.

[0173] Optionally, the training process of the generator is completed by the following modules:

[0174] A sample image pair determination module is used to obtain a sample image pair, wherein the sample image pair includes: a sample image to be processed and a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter;

[0175] A sample image conversion module, used to convert the two images in the sample image pair in frequency domain dimensions respectively to obtain a sample spectrum image pair;

[0176] The training module is used to use the sample spectrum image pairs as training sample sets to alternately train the generator and the discriminator in the generative adversarial model to obtain a trained generator.

[0177] Optionally, the training process of the generator is completed by the following modules:

[0178] A positive sample image pair determination module is used to obtain a positive sample image pair, wherein the positive sample image pair includes: a sample image to be processed and a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter;

[0179] A positive sample image conversion module, used to convert the two images in the positive sample image pair in frequency domain dimensions respectively to obtain a positive sample spectrum image pair;

[0180] a negative sample image pair determination module, configured to obtain a negative sample image pair, wherein the negative sample image pair comprises: a first sample image to be processed and a sample processed image obtained by adjusting the second sample image to be processed according to a target contrast adjustment parameter, wherein the first sample image to be processed and the second sample image to be processed are two groups of images with different image contents;

[0181] A negative sample image conversion module, used to convert the two images in the negative sample image pair in frequency domain dimensions to obtain a negative sample spectrum image pair;

[0182] The training module is used to alternately train the generator and the discriminator in the generative adversarial model using the positive sample spectrum image pair and the negative sample spectrum image pair as training sample sets to obtain a trained generator.

[0183] Optionally, the discriminator included in the generative adversarial model is a first discriminator; and the training module includes:

[0184] An input submodule, used for inputting the first sample spectrum image in each sample spectrum image pair included in the training sample set into the generator in the generative adversarial model to obtain a simulated sample spectrum image generated by the generator, wherein the first sample spectrum image is the spectrum image of the sample image to be processed;

[0185] An inverse conversion submodule is used to perform an inverse conversion of the simulated sample spectrum image in the frequency domain dimension to obtain a sample simulated image after contrast adjustment of the sample image to be processed;

[0186] A structural feature loss value determination submodule, used for obtaining a structural feature loss value of the generator according to a difference in structural features between the sample simulated image and the sample to-be-processed image;

[0187] a spectrum loss value determination submodule, configured to obtain a spectrum loss value of the generator according to a spectrum difference outputted after the first discriminator discriminates the simulated sample spectrum image and a second sample spectrum image, wherein the second sample spectrum image is a spectrum image of the sample processed image;

[0188] An updating submodule is used to update the model parameters of the generator according to the structural feature loss value and the spectrum loss value.

[0189] Optionally, the generative adversarial model further includes a second discriminator; and the training module further includes:

[0190] An image feature loss value determination submodule, used to obtain the image feature loss value of the generator according to the difference in image features output after the second discriminator discriminates the sample simulated image and the sample to-be-processed image;

[0191] The updating submodule is specifically used for:

[0192] The model parameters of the generator are updated according to the structural feature loss value, the spectrum loss value and the image feature loss value.

[0193] Optionally, the training process of the correction model is completed by the following modules:

[0194] A sample image group determination module is used to obtain a sample image group, wherein the sample image group includes: a sample image to be processed, a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter, and a sample image to be corrected after contrast adjustment is performed on the sample image to be processed;

[0195] A sample image to be corrected input module is used to input the sample image to be corrected into a preset model to obtain a sample corrected image output by the preset model;

[0196] A first loss value determination module, used for obtaining a first loss value of the preset model according to an image feature difference between a designated area of ​​the sample to-be-processed image and an image area corresponding to the designated area in the sample corrected image;

[0197] A second loss value determination module, used for obtaining a second loss value of the preset model according to an image feature difference between an image region corresponding to the designated region in the sample corrected image and an image region corresponding to the designated region in the sample processed image;

[0198] An updating module, configured to update the preset model parameters according to the first loss value and the second loss value;

[0199] The correction module determines the module, which is used to determine the trained preset model as the correction model.

[0200] Optionally, the device further comprises:

[0201] A first sample spectrum image determination module is used to convert the sample image to be processed into a frequency domain dimension to obtain a first sample spectrum image;

[0202] A third sample spectrum image determination module, configured to input the first sample spectrum image into a pre-trained generator to obtain a third sample spectrum image;

[0203] The sample image to be corrected determining module is used to perform an inverse transformation of the third sample spectrum image in the frequency domain dimension to obtain the sample image to be corrected.

[0204] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0205] An embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the image processing method described in any of the above embodiments when executed.

[0206] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the image processing method described in any of the above embodiments are implemented.

[0207] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the image processing method described in any of the above embodiments when executed by a processor.

[0208] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0209] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0210] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0211] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0213] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0214] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0215] The above is a detailed introduction to an image processing method, electronic device, storage medium and computer program product provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Convert the image to be processed into a frequency domain dimension to obtain a first spectrum image; Inputting the first spectrum image into a pre-trained generator to obtain a second spectrum image, wherein the pre-trained generator has pre-learned a correspondence between a spectrum image of a sample to-be-processed image and a spectrum image of a sample processed image, wherein the sample processed image is obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter; Performing an inverse transformation of the frequency domain dimension on the second spectrum image to obtain an image after adjusting the image to be processed according to the target contrast adjustment parameter; The training steps of the generator include: Obtaining a sample image pair, the sample image pair comprising: a sample to-be-processed image and a sample processed image obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter; Performing frequency domain conversion on two images in the sample image pair respectively to obtain a sample spectrum image pair; Using the sample spectrum image pairs as training sample sets, alternately training the generator and the discriminator in the generative adversarial model to obtain a trained generator; The discriminator included in the generative adversarial model is a first discriminator; The generator and discriminator in the generative adversarial model are trained alternately to obtain a trained generator, including: Inputting the first sample spectrum image in each sample spectrum image pair included in the training sample set into the generator in the generative adversarial model to obtain a simulated sample spectrum image generated by the generator, wherein the first sample spectrum image is the spectrum image of the sample image to be processed; Performing an inverse transformation of the simulated sample spectrum image in the frequency domain dimension to obtain a sample simulated image after contrast adjustment of the sample image to be processed; Obtaining a structural feature loss value of the generator according to a difference in structural features between the sample simulated image and the sample to-be-processed image; Obtaining a spectrum loss value of the generator according to a spectrum difference outputted after the first discriminator discriminates the simulated sample spectrum image and the second sample spectrum image, wherein the second sample spectrum image is a spectrum image of the sample processed image; According to the structural feature loss value and the spectrum loss value, the model parameters of the generator are updated; The generative adversarial model also includes a second discriminator; Among them, the generator and the discriminator in the generative adversarial model are alternately trained to obtain a trained generator, which also includes: Obtaining an image feature loss value of the generator according to a difference in image features outputted after the second discriminator discriminates the sample simulated image and the sample to-be-processed image; The method includes updating the model parameters of the generator according to the structural feature loss value and the spectrum loss value, including: The model parameters of the generator are updated according to the structural feature loss value, the spectrum loss value and the image feature loss value.

2. The method according to claim 1, characterized in that After obtaining the image to be processed after adjusting the image according to the target contrast adjustment parameter, the method further includes: The specified area of ​​the adjusted image is corrected by a pre-trained correction model to obtain a corrected image. The pre-trained correction model is pre-labeled with the specified area of ​​the sample image to be processed and has learned the correction operation to be performed on the specified area of ​​the sample processed image.

3. The method according to claim 1, characterized in that The training step of the generator also includes: Obtaining a positive sample image pair, the positive sample image pair comprising: a sample to-be-processed image and a sample processed image obtained by adjusting the sample to-be-processed image according to a target contrast adjustment parameter; Performing frequency domain dimension conversion on two images in the positive sample image pair respectively to obtain a positive sample spectrum image pair; Obtaining a negative sample image pair, the negative sample image pair comprising: a first sample to-be-processed image and a sample processed image obtained by adjusting the second sample to-be-processed image according to a target contrast adjustment parameter, the first sample to-be-processed image and the second sample to-be-processed image being two groups of images with different image contents; Performing frequency domain conversion on two images in the negative sample image pair respectively to obtain a negative sample spectrum image pair; The positive sample spectrum image pair and the negative sample spectrum image pair are used as training sample sets, and the generator and the discriminator in the generative adversarial model are alternately trained to obtain a trained generator.

4. The method according to claim 2, characterized in that: The training steps of the modified model include: Obtaining a sample image group, the sample image group comprising: a sample image to be processed, a sample processed image obtained by adjusting the sample image to be processed according to a target contrast adjustment parameter, and a sample image to be corrected after contrast adjustment is performed on the sample image to be processed; Inputting the sample image to be corrected into a preset model to obtain a sample corrected image output by the preset model; Obtaining a first loss value of the preset model according to an image feature difference between a designated area of ​​the sample to-be-processed image and an image area corresponding to the designated area in the sample corrected image; Obtaining a second loss value of the preset model according to an image feature difference between an image region corresponding to the designated region in the sample corrected image and an image region corresponding to the designated region in the sample processed image; Updating the preset model parameters according to the first loss value and the second loss value; The trained preset model is determined as the revised model.

5. The method according to claim 4, characterized in that The sample image to be corrected after contrast adjustment of the sample image to be processed is generated by using a pre-trained generator according to the following steps: Convert the sample image to be processed into a frequency domain dimension to obtain a first sample spectrum image; Inputting the first sample spectrum image into a pre-trained generator to obtain a third sample spectrum image; The third sample spectrum image is inversely transformed in frequency domain dimension to obtain the sample image to be corrected.

6. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the image processing method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images