Image processing method and device, equipment, storage medium and computer program product

Abstract

The invention relates to an image processing method and apparatus, a device, a storage medium and a computer program product. The method comprises the steps of obtaining a to-be-processed image; processing the to-be-processed image based on a preset vignetting removal model to obtain a vignetting-removed image; wherein the vignetting removal model is obtained by training a neural network model by using a training sample data set, and the training sample data set comprises multiple groups of paired non-vignetting sample images and first vignetting sample images; the first vignetting sample images in the same group are generated after the non-vignetting sample images are processed by using a preset vignetting mask. Through the method, the effect of removing the image vignetting can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of electronic devices, more and more things can be done on them, such as taking or viewing images. However, factors such as lighting conditions, lens design, and the use of filters and lens hoods can all cause vignetting, affecting image quality. Currently, methods for removing vignetting and improving image quality are relatively inaccurate, produce poor results, and are slow. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides an image processing method, apparatus, device, storage medium, and computer program product.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0005] Obtain the image to be processed;

[0006] The image to be processed is processed based on a preset vignetting removal model to obtain an image after vignetting removal; wherein, the vignetting removal model is obtained by training a neural network model using a training sample dataset, the training sample dataset includes multiple sets of paired vignetting-free sample images and a first vignetting sample image, the first vignetting sample image in the same set is generated by processing the vignetting-free sample image using a preset vignetting mask.

[0007] In some embodiments, the method further includes:

[0008] Obtain vignetting-free sample images from the training sample dataset;

[0009] Obtain multiple mask images obtained by taking pictures of a preset scene based on different shooting parameters;

[0010] For each sample image without dark corners, a weighted fusion is performed on any number of mask images from the multiple mask images a preset number of times to obtain the target mask image corresponding to the preset number of times;

[0011] For each vignetting-free sample image, a masking process is performed on the vignetting-free sample image using each target mask image associated with the vignetting-free sample image to obtain a first vignetting sample image corresponding to the vignetting-free sample image; wherein, the number of the first vignetting sample images corresponding to the vignetting-free sample image is the same as the preset number of times;

[0012] The neural network model is trained using vignetting-free sample images from the training sample dataset and a first vignetting sample image associated with the vignetting-free sample images to obtain the vignetting removal model.

[0013] In some embodiments, the neural network model includes a feature extraction layer, a first feature processing layer, and a second feature processing layer; training the neural network model using vignetting-free sample images from the training sample dataset and first vignetting sample images associated with the vignetting-free sample images to obtain the vignetting removal model includes:

[0014] For each first vignetting sample image, the features of the first vignetting sample image are extracted in the feature extraction layer to obtain the features of the first vignetting sample image.

[0015] For each first vignetting sample image, the brightness compensation value of the first vignetting sample image is determined in the first feature processing layer based on the features of the first vignetting sample image and the multiple mask images.

[0016] For each first vignetting sample image, the brightness of the first vignetting sample image is compensated based on the brightness compensation value of the first vignetting sample image in the second feature processing layer to obtain the predicted image after removing the vignetting corresponding to the first vignetting sample image.

[0017] Based on the difference between the predicted image after vignetting corresponding to each first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image, the parameters of the neural network model are adjusted until a preset convergence condition is met, thus obtaining the vignetting removal model.

[0018] In some embodiments, the number of features in the first vignetting sample image is the same as the number of the plurality of mask images; determining the brightness compensation value of the first vignetting sample image in the first feature processing layer based on the features of the first vignetting sample image and the plurality of mask images includes:

[0019] In the first feature processing layer, each feature of the first vignetting sample image is normalized to obtain each normalized feature.

[0020] In the first feature processing layer, one of the multiple mask images is multiplied by a normalized feature, and the results of multiple multiplications are added together to obtain the vignetting simulation feature value;

[0021] Based on the vignetting simulation feature value, the brightness compensation value of the first vignetting sample image is determined; wherein the brightness compensation value is negatively correlated with the vignetting simulation feature value.

[0022] In some embodiments, extracting features of the first vignetting sample image from the feature extraction layer to obtain the features of the first vignetting sample image includes:

[0023] The feature extraction layer extracts features from the first vignetting sample image based on convolution operations to obtain the convolutional features of the first vignetting sample image;

[0024] The convolutional features of the first vignetting sample image are subjected to dimensionality reduction processing to obtain a one-dimensional feature vector of the first vignetting sample image;

[0025] The number of features in the one-dimensional feature vector is mapped to the number of the multiple mask images, and the mapped one-dimensional feature vector is used as the feature of the first vignetting sample image.

[0026] In some embodiments, the step of masking the vignette sample image using each target mask image associated with the vignette sample image to obtain a first vignette sample image corresponding to the vignette sample image includes:

[0027] The target pixels in the no-dark-corner sample image are masked using each target mask image associated with the no-dark-corner sample image, and the pixel values ​​of the pixels other than the target pixels are retained to obtain the first dark-corner sample image corresponding to the no-dark-corner sample image; wherein the pixel value of the target pixel is less than a preset pixel threshold.

[0028] In some embodiments, the training sample dataset further includes second vignetting sample images captured by photographs; the vignetting removal model is obtained by training the neural network model collaboratively using a first sample set consisting of vignetting-free sample images and first vignetting sample images associated with the vignetting-free sample images, and a second sample set consisting of second vignetting sample images.

[0029] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0030] The first acquisition module is configured to acquire the image to be processed.

[0031] The processing module is configured to process the image to be processed based on a preset vignetting removal model to obtain an image after vignetting removal; wherein, the vignetting removal model is obtained by training a neural network model using a training sample dataset, the training sample dataset includes multiple sets of paired vignetting-free sample images and a first vignetting sample image, the first vignetting sample image in the same set is generated by processing the vignetting-free sample image using a preset vignetting mask.

[0032] In some embodiments, the apparatus further includes:

[0033] The second acquisition module is configured to acquire vignetting-free sample images from the training sample dataset.

[0034] The third acquisition module is configured to acquire multiple mask images obtained by taking pictures of a preset scene based on different shooting parameters;

[0035] The fusion module is configured to perform a preset number of weighted fusions on any number of mask images among the multiple mask images for each sample image without dark corners, so as to obtain the target mask image corresponding to the preset number of fusions.

[0036] The mask processing module is configured to perform mask processing on each vignetting-free sample image using each target mask image associated with the vignetting-free sample image, thereby obtaining a first vignetting sample image corresponding to the vignetting-free sample image; wherein, the number of the first vignetting sample images corresponding to the vignetting-free sample image is the same as the preset number of times;

[0037] The training module is configured to train the neural network model using vignetting-free sample images from the training sample dataset and a first vignetting sample image associated with the vignetting-free sample images, to obtain the vignetting removal model.

[0038] In some embodiments, the neural network model includes a feature extraction layer, a first feature processing layer, and a second feature processing layer; the training module is further configured to, for each first vignetting sample image, extract features of the first vignetting sample image in the feature extraction layer to obtain features of the first vignetting sample image; for each first vignetting sample image, determine a brightness compensation value of the first vignetting sample image in the first feature processing layer based on the features of the first vignetting sample image and the multiple mask images; for each first vignetting sample image, perform brightness compensation on the first vignetting sample image in the second feature processing layer based on the brightness compensation value of the first vignetting sample image to obtain a vignetting-removed predicted image corresponding to the first vignetting sample image; adjust the parameters of the neural network model based on the difference between the vignetting-removed predicted image corresponding to each first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image, until a preset convergence condition is met to obtain the vignetting removal model.

[0039] In some embodiments, the number of features in the first vignetting sample image is the same as the number of the plurality of mask images; the training module is further configured to normalize each feature of the first vignetting sample image in the first feature processing layer to obtain each normalized feature; in the first feature processing layer, multiply one of the mask images in the plurality of mask images by one normalized feature, and add the multiple multiplication results to obtain a vignetting simulation feature value; based on the vignetting simulation feature value, determine the brightness compensation value of the first vignetting sample image; wherein the brightness compensation value is negatively correlated with the vignetting simulation feature value.

[0040] In some embodiments, the training module is further configured to extract features of the first vignetting sample image based on convolution operations at the feature extraction layer to obtain convolutional features of the first vignetting sample image; perform dimensionality reduction processing on the convolutional features of the first vignetting sample image to obtain a one-dimensional feature vector of the first vignetting sample image; map the number of features in the one-dimensional feature vector to the number of the multiple mask images, and use the mapped one-dimensional feature vector as the feature of the first vignetting sample image.

[0041] In some embodiments, the mask processing module is further configured to perform mask processing on the target pixels in the vignetting sample image using each target mask image associated with the vignetting sample image, and retain the pixel values ​​of the pixels other than the target pixels to obtain a first vignetting sample image corresponding to the vignetting sample image; wherein the pixel value of the target pixel is less than a preset pixel threshold.

[0042] In some embodiments, the training sample dataset further includes second vignetting sample images captured by photographs; the vignetting removal model is obtained by training the neural network model collaboratively using a first sample set consisting of vignetting-free sample images and first vignetting sample images associated with the vignetting-free sample images, and a second sample set consisting of second vignetting sample images.

[0043] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0044] processor;

[0045] Memory used to store computer programs or instructions;

[0046] The processor executes the computer program or instructions to implement the steps of the image processing method described in the first aspect above.

[0047] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions which, when executed by a processor, implement the steps of the image processing method described in the first aspect above.

[0048] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the image processing method described in the first aspect above.

[0049] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0050] In this embodiment, an electronic device acquires an image to be processed and processes it based on a preset vignetting removal model to obtain an image with vignetting removed. On one hand, since the vignetting removal model is trained on a neural network model using a training sample dataset, it can learn the characteristics and distribution patterns of vignetting in the image. This allows for more accurate identification and removal of vignetting in the image, rather than relying solely on simple pixel brightness thresholds. This not only improves the vignetting removal effect but also reduces over-processing of non-vignetting areas in the image due to incorrect threshold settings. On the other hand, since the first vignetting sample images are generated by applying a preset vignetting mask to vignetting-free sample images, a clear pairing relationship is formed between them. This pairing relationship helps the neural network model learn more accurately how to recover vignetting-free images from vignetting images, thereby improving the model's training effect.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0053] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0054] Figure 2 This is a schematic diagram of a vignetting mask image according to an exemplary embodiment.

[0055] Figure 3 This is a schematic diagram illustrating the principle of an image processing method according to an exemplary embodiment.

[0056] Figure 4This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0057] Figure 5 This is an example diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0058] Figure 6 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0059] Figure 7 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0060] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0061] It's important to note that vignetting refers to a gradual decrease in image brightness from the optical center to the corners, potentially accompanied by color shift at the corners. The causes of vignetting are varied. For example, during image capture, the angle between the light rays in the corner areas and the lens's optical axis may be greater than the angle in the center. Since the path length of light in an optical system increases with the angle, and the light intensity decreases, the light reaching the image plane from the corners is weaker than from the center. This means the light from the corners experiences more obstruction and scattering as it passes through the lens, resulting in light loss and vignetting. Vignetting can also be caused by inadequate lens design or improper use of filters and lens hoods during image capture.

[0062] One related technology involves a method for removing vignetting in images. This method sets a pixel brightness threshold and compares the brightness of each pixel in the image with the threshold. Pixels with brightness less than the threshold are compensated for. The effectiveness of this method in removing vignetting is highly dependent on the set pixel brightness threshold. If the threshold is set improperly, it may result in incomplete or excessive vignetting removal, affecting the overall image quality. Furthermore, this method requires traversing all pixels in the image, consuming significant computation time and resources, resulting in low efficiency.

[0063] In response, this disclosure provides an image processing method. Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 As shown, the method mainly includes the following steps:

[0064] S11. Obtain the image to be processed;

[0065] S12. The image to be processed is processed based on a preset vignetting removal model to obtain an image after vignetting removal; wherein, the vignetting removal model is obtained by training a neural network model using a training sample dataset, the training sample dataset includes multiple sets of paired vignetting-free sample images and a first vignetting sample image, the first vignetting sample image in the same set is generated by processing the vignetting-free sample image using a preset vignetting mask.

[0066] In this embodiment of the disclosure, the image processing method can be applied to electronic devices with image processing functions, such as user equipment (UE), mobile devices, user terminals, mobile phones, tablets, personal digital assistants (PDAs), handheld devices, computing devices, and vehicle-mounted devices. It can also be applied to electronic devices with photo or video recording functions, such as wearable devices, cameras, camcorders, and cameras.

[0067] In step S11, the electronic device acquires the image to be processed. In some embodiments, the electronic device may be equipped with an image acquisition component, which acquires the image to be processed, such as a front-facing camera or a rear-facing camera. In other embodiments, the electronic device may also acquire the image to be processed from other devices for processing; in still other embodiments, the electronic device may also acquire the image to be processed from an image library. The embodiments disclosed herein do not limit the means and methods by which the electronic device acquires the image to be processed.

[0068] The method of this disclosure can be applied to the photography scenario of an electronic device. For example, the electronic device can respond to the detection of a photography command, acquire an image captured by an image acquisition component, use the acquired image as an image to be processed, and obtain and display an image after removing vignetting based on the method of this disclosure. The photography command can be a touch command detected by the electronic device or a voice command detected by the electronic device.

[0069] The method of this disclosure can also be applied to image post-processing scenarios, such as an electronic device performing the method of this disclosure on an image that has been photographed and stored in the electronic device or on an image received from another device to obtain an image after removing vignetting.

[0070] In this embodiment of the disclosure, after the electronic device acquires the image to be processed, it can first perform preprocessing such as binarization, image enhancement, image denoising, color conversion, and geometric correction on the image to be processed.

[0071] In step S12, the electronic device processes the image to be processed based on a preset vignetting removal model to obtain an image with vignetting removed. The vignetting removal model is obtained by training a neural network model using a training sample dataset. The neural network model can be a Convolutional Neural Network (CNN), a Generative Adversarial Network (GAN), a Residual Network (ResNet), a Recurrent Neural Network (RNN), etc. This embodiment of the disclosure may select a neural network model for training according to specific circumstances, and this embodiment of the disclosure does not specifically limit the neural network model.

[0072] In this embodiment, the training sample dataset includes multiple pairs of paired vignetting-free sample images and a first vignetting sample image. The first vignetting sample image within the same group is generated by processing the vignetting-free sample image using a preset vignetting mask. The electronic device can acquire the vignetting-free sample images from an image database or using over-the-air (OTA) technology. For example, the electronic device can acquire high-resolution images from a known dataset (such as the DIV2K dataset) and use these images as vignetting-free sample images. It should be noted that the DIV2K dataset is a high-quality dataset specifically designed for image super-resolution (SR) tasks. These images have extremely high clarity and detail. The images in the DIV2K dataset have undergone rigorous screening and correction during acquisition and processing to ensure image quality.

[0073] In this embodiment of the disclosure, the first vignetting sample image within the same group is generated by processing a vignetting-free sample image using a preset vignetting mask. In some embodiments, the preset vignetting mask may be a matrix whose size is related to the resolution of the vignetting-free sample image, wherein the element values ​​of the matrix characterize the vignetting intensity of the corresponding pixel in the vignetting-free sample image. For example, the preset vignetting mask may be a mask with higher element values ​​in the central region (indicating higher brightness, i.e., no vignetting or weak vignetting) and gradually decreasing element values ​​in the edge region (indicating gradually decreasing brightness, i.e., gradually increasing vignetting).

[0074] In other embodiments, the preset vignetting mask can be a mask based on image edge detection. For example, edge detection can be performed on the vignetting-free sample image to obtain the edge image corresponding to the vignetting-free sample image. Then, a vignetting mask corresponding to the vignetting-free sample image can be generated based on the edge image. The mask value is lower in the edge region and higher in the center region. This mask can simulate the vignetting effect caused by the optical characteristics of the lens.

[0075] In other embodiments, the electronic device can acquire multiple mask images obtained by taking pictures of a preset scene with different shooting parameters, and perform normalization and other processing on the pixel values ​​in the multiple mask images to obtain a preset vignetting mask.

[0076] In this embodiment of the present disclosure, the electronic device can multiply the vignetting-free sample image with a preset vignetting mask pixel by pixel, and use the multiplied image as the first vignetting sample image; after multiplying the vignetting-free sample image with the preset vignetting mask pixel by pixel, the electronic device can also perform normalization, smoothing, brightness adjustment, contrast adjustment and other processing on the pixel values ​​of the multiplied image pixels to obtain the first vignetting sample image.

[0077] In this embodiment of the disclosure, after obtaining the image with vignetting removed through the vignetting removal model, the electronic device can also perform post-processing such as color adjustment, detail enhancement, and composition adjustment on the image with vignetting removed in order to improve the image quality.

[0078] In this embodiment, an electronic device acquires an image to be processed and processes it based on a preset vignetting removal model to obtain an image with vignetting removed. On one hand, since the vignetting removal model is trained on a neural network model using a training sample dataset, it can learn the characteristics and distribution patterns of vignetting in the image. This allows for more accurate identification and removal of vignetting in the image, rather than relying solely on simple pixel brightness thresholds. This not only improves the vignetting removal effect but also reduces over-processing of non-vignetting areas in the image due to incorrect threshold settings. On the other hand, since the first vignetting sample images are generated by applying a preset vignetting mask to vignetting-free sample images, a clear pairing relationship is formed between them. This pairing relationship helps the neural network model learn more accurately how to recover vignetting-free images from vignetting images, thereby improving the model's training effect.

[0079] In some embodiments, the method further includes:

[0080] Obtain vignetting-free sample images from the training sample dataset;

[0081] Obtain multiple mask images obtained by taking pictures of a preset scene based on different shooting parameters;

[0082] For each sample image without dark corners, a weighted fusion is performed on any number of mask images from the multiple mask images a preset number of times to obtain the target mask image corresponding to the preset number of times;

[0083] For each vignetting-free sample image, a masking process is performed on the vignetting-free sample image using each target mask image associated with the vignetting-free sample image to obtain a first vignetting sample image corresponding to the vignetting-free sample image; wherein, the number of the first vignetting sample images corresponding to the vignetting-free sample image is the same as the preset number of times;

[0084] The neural network model is trained using vignetting-free sample images from the training sample dataset and a first vignetting sample image associated with the vignetting-free sample images to obtain the vignetting removal model.

[0085] In this embodiment of the disclosure, the electronic device acquires vignetting-free sample images from the training sample dataset. As mentioned above, the electronic device can acquire vignetting-free sample images from an image database or by using over-the-air (OTA) download technology. For example, the electronic device can acquire high-resolution images from a known dataset (such as the DIV2K dataset) and use these images as vignetting-free sample images.

[0086] In this embodiment of the disclosure, the electronic device also acquires multiple mask images obtained by taking pictures of a preset scene based on different shooting parameters. It is understood that the vignetting of the image is highly correlated with the shooting parameters when the image is taken. Therefore, the vignetting features in the multiple mask images obtained by taking pictures of the preset scene based on different shooting parameters are also different.

[0087] In this embodiment of the disclosure, the photographing parameters can be camera type, aperture, shutter speed, ISO sensitivity, focal length, exposure compensation, white balance parameters, etc. The preset image can be a scene with little content, such as a white paper or a white wall. For example, the electronic device can acquire mask images of six different types of cameras at apertures of F2.0 and F5.6, and at ISO 50, 200, 400, 800, 1600, 3200, 6400, and 12800.

[0088] In this embodiment of the disclosure, after the electronic device acquires multiple mask images, it can normalize the pixel values ​​of each pixel in the mask images to facilitate subsequent operations. For example, for each mask image in the multiple mask images, the electronic device can set the pixel value of the center pixel of the mask image to 1, and normalize the pixel values ​​of each pixel based on the relationship between the pixel value of the center pixel and the pixel values ​​of the pixels other than the center pixel, and determine the normalized result of the pixel values ​​of the pixels other than the center pixel. Figure 2 This is a schematic diagram of a vignetting mask image according to an exemplary embodiment, such as... Figure 2 As shown, Figure 2 (a), (b), and (c) are mask images obtained after taking pictures of white paper under different shooting parameters and performing the above normalization operation. As can be seen from the figure, the characteristics of the vignetting in the mask images corresponding to different shooting parameters are not the same. For example, the intensity of the vignetting in (a) is less than that in (b), and the intensity of the vignetting in (b) is less than that in (c).

[0089] In this embodiment of the disclosure, the electronic device performs a weighted fusion of any number of mask images from multiple mask images for each sample image without vignetting, to obtain a target mask image corresponding to the preset number of vignetting operations. The number of mask images corresponding to each weighted fusion operation varies, for example, it can be any value between 3 and 10, and the preset number of vignetting operations can be a set value, such as 5 or 10.

[0090] It should be noted that when weighted fusing any number of mask images, the sum of the weights of each mask image is 1. For example, when weighted fusing 3 mask images, the weights of each mask image can be set to 0.3, 0.3, and 0.4 respectively. During weighted fusing, the pixel value of each pixel in each mask image can be multiplied by the weight of each mask image, and then the mask images with updated pixel values ​​are added together to obtain the target mask image.

[0091] In this embodiment of the disclosure, after performing a preset number of weighted fusions on any number of mask images, the pixel value of each pixel in the weighted fusion image can be normalized to obtain a normalized image, and the normalized image can be used as the target mask image.

[0092] In this embodiment of the present disclosure, the electronic device performs masking processing on each dark corner-free sample image using each target mask image associated with the dark corner-free sample image to obtain a first dark corner sample image corresponding to the dark corner-free sample image, wherein the number of first dark corner sample images corresponding to the dark corner-free sample image is the same as the preset number of times.

[0093] In this embodiment of the disclosure, for each masking process performed on the vignetting sample image, after determining the target mask image, the target mask image can be processed first to make the size and resolution of the target mask image the same as the size and resolution of the vignetting sample image. Then, the pixel value of each pixel in the vignetting sample image is multiplied by the pixel value of the corresponding pixel in the target mask image to obtain the first vignetting sample image corresponding to the vignetting sample image.

[0094] In some embodiments, the step of masking the vignette sample image using each target mask image associated with the vignette sample image to obtain a first vignette sample image corresponding to the vignette sample image includes:

[0095] The target pixels in the no-dark-corner sample image are masked using each target mask image associated with the no-dark-corner sample image, and the pixel values ​​of the pixels other than the target pixels are retained to obtain the first dark-corner sample image corresponding to the no-dark-corner sample image; wherein the pixel value of the target pixel is less than a preset pixel threshold.

[0096] In this embodiment of the disclosure, before performing masking processing on the no-dark-angle sample image using each target mask image associated with the no-dark-angle sample image, the electronic device determines the pixel value of the pixel in the no-dark-angle sample image and takes the pixel value less than a preset pixel threshold as the target pixel. During the masking process, only the target pixel is masked, that is, the pixel value of the target pixel is multiplied by the pixel value of the corresponding pixel in the target mask image, and the pixels other than the target pixel are not masked, that is, the pixel values ​​of the pixels other than the target pixel are retained. The preset pixel threshold is a set value, such as setting the preset pixel threshold to 255.

[0097] In this embodiment of the disclosure, since there may be overflow pixels (pixel values ​​exceeding the valid range of pixel values ​​when storing the image) in the sample image without dark corners, if the pixel is masked based on the target mask image, the image content may become abnormal. However, this embodiment of the disclosure improves the accuracy of masking by determining the target pixel and only masking the target pixel, while reducing the possibility of abnormal image content, and has a high degree of intelligence.

[0098] In this embodiment of the disclosure, the electronic device trains a neural network model using vignetting-free sample images from the training sample dataset and a first vignetting sample image associated with the vignetting-free sample image to obtain a vignetting removal model. For example, the electronic device can use the neural network model to process the first vignetting sample image corresponding to the vignetting-free sample image to obtain a first vignetting sample image after vignetting removal. Then, based on the difference between the first vignetting sample image after vignetting removal and the vignetting-free sample image, the parameters of the neural network model are adjusted until the difference is less than a preset difference threshold, thus obtaining the vignetting removal model.

[0099] In this embodiment, multiple mask images are obtained by taking pictures of a preset scene based on different shooting parameters. For each vignetting-free sample image, a weighted fusion is performed a preset number of times based on any number of mask images from the multiple mask images to obtain a target mask image corresponding to the preset number of times. Then, for each vignetting-free sample image, a masking process is performed on the vignetting-free sample image using each target mask image associated with the vignetting-free sample image to obtain a first vignetting sample image corresponding to the vignetting-free sample image. Finally, a neural network model is trained using the vignetting-free sample image and the first vignetting sample image associated with the vignetting-free sample image. The method for obtaining the vignetting removal model has two aspects. On the one hand, by using multiple mask images with different shooting parameters for weighted fusion and generating multiple first vignetting sample images for different vignetting-free sample images, it is possible to simulate the vignetting effects that may occur under different conditions, thereby increasing the diversity of training data and enhancing the model's generalization ability to remove different types of vignetting, thus improving the accuracy of the trained vignetting removal model in removing vignetting from images. On the other hand, since multiple mask images are obtained by taking pictures under different shooting parameters, they can simulate possible vignetting features in real shooting environments, improving the realism of the first vignetting sample images.

[0100] In some embodiments, the neural network model includes a feature extraction layer, a first feature processing layer, and a second feature processing layer; training the neural network model using vignetting-free sample images from the training sample dataset and first vignetting sample images associated with the vignetting-free sample images to obtain the vignetting removal model includes:

[0101] For each first vignetting sample image, the features of the first vignetting sample image are extracted in the feature extraction layer to obtain the features of the first vignetting sample image.

[0102] For each first vignetting sample image, the brightness compensation value of the first vignetting sample image is determined in the first feature processing layer based on the features of the first vignetting sample image and the multiple mask images.

[0103] For each first vignetting sample image, the brightness of the first vignetting sample image is compensated based on the brightness compensation value of the first vignetting sample image in the second feature processing layer to obtain the predicted image after removing the vignetting corresponding to the first vignetting sample image.

[0104] Based on the difference between the predicted image after vignetting corresponding to each first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image, the parameters of the neural network model are adjusted until a preset convergence condition is met, thus obtaining the vignetting removal model.

[0105] In this embodiment of the disclosure, the electronic device extracts features of each first vignetting sample image in a feature extraction layer to obtain the features of the first vignetting sample image. The features of the first vignetting sample image can be represented as a feature image, feature matrix, feature vector, feature descriptor, etc.

[0106] In some embodiments, the electronic device can extract features of the first vignetting sample image based on the scale-invariant feature transformation method at the feature extraction layer to obtain the features of the first vignetting sample image. The first vignetting sample image can be preprocessed first, such as by binarizing the first vignetting sample image, to reduce the interference of color information on feature extraction.

[0107] In some embodiments, the electronic device may extract features of the first vignetting sample image based on the directional gradient histogram method at the feature extraction layer; in other embodiments, the electronic device may also extract features of the first vignetting sample image based on edge detection and shape analysis at the feature extraction layer.

[0108] In other embodiments, the step of extracting features of the first vignetting sample image in the feature extraction layer to obtain the features of the first vignetting sample image includes:

[0109] The feature extraction layer extracts features from the first vignetting sample image based on convolution operations to obtain the convolutional features of the first vignetting sample image;

[0110] The convolutional features of the first vignetting sample image are subjected to dimensionality reduction processing to obtain a one-dimensional feature vector of the first vignetting sample image;

[0111] The number of features in the one-dimensional feature vector is mapped to the number of the multiple mask images, and the mapped one-dimensional feature vector is used as the feature of the first vignetting sample image.

[0112] In this embodiment of the disclosure, features of the first vignetting sample image are extracted based on convolution operations at the feature extraction layer to obtain the convolutional features of the first vignetting sample image. For example, a 3*3 convolution kernel can be used to extract features from the first vignetting sample image to obtain the convolutional features of the first vignetting sample image. For example, the feature image corresponding to the first sample image is obtained. It should be noted that multiple convolution operations can be performed on the first sample image to extract its features. Since the convolution operation intelligently captures the linear relationship in the first vignetting sample image, after each feature extraction based on the convolution operation, an activation function can be used to capture the nonlinear relationship in the first vignetting sample image. The activation function can be a rectified linear unit (ReLU), a leaky rectified linear unit (Leaky ReLU), a parametric rectified linear unit (PReLU), etc.

[0113] In this embodiment of the disclosure, the first vignetting sample image can also be downsampled in the feature extraction layer to reduce the resolution of the first vignetting sample image and reduce the amount of data contained in the first vignetting sample image, so as to improve the efficiency of subsequent processing.

[0114] In this embodiment of the disclosure, after obtaining the convolutional features of the first vignetting sample image, the convolutional features of the first vignetting sample image are subjected to dimensionality reduction processing to obtain a one-dimensional feature vector of the first vignetting sample image. In some embodiments, a 1*1 convolutional kernel can be used to perform dimensionality reduction processing on the convolutional features of the first vignetting sample image, and then the convolutional features are subjected to global average pooling (GAP) or global max pooling (GMP) to obtain a one-dimensional feature vector.

[0115] In other embodiments, methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), and autoencoders can be used to reduce the dimensionality of the convolutional features of the first vignetting sample image to obtain a one-dimensional feature vector of the first vignetting sample image. The embodiments of this disclosure do not limit the method of dimensionality reduction.

[0116] In this embodiment of the disclosure, after obtaining the one-dimensional feature vector, the number of features in the one-dimensional feature vector is mapped to the number of multiple mask images, and the mapped one-dimensional feature vector is used as the feature of the first vignetting sample image. In some embodiments, fully connected layers (FC layers) can be used to map the number of features in the one-dimensional feature vector to the number of multiple mask images. The parameters of the weight matrix and bias vector in the fully connected layers can be adjusted using vignetting-free sample images from the training sample dataset and the first vignetting sample image associated with the vignetting-free sample images.

[0117] In other embodiments, methods such as convolution kernels, linear transformations, attention mechanisms, and embedding space methods can be used to map the number of features in a one-dimensional feature vector to the number of multiple mask images, and the mapped one-dimensional feature vector can be used as the feature of the first vignetting sample image. This disclosure does not limit the scope of the embodiments.

[0118] In this embodiment, features of a first vignetting sample image are extracted based on convolution operations at the feature extraction layer to obtain convolutional features of the first vignetting sample image. Then, the convolutional features of the first vignetting sample image are subjected to dimensionality reduction processing to obtain a one-dimensional feature vector of the first vignetting sample image. The number of features in the one-dimensional feature vector is mapped to the number of multiple mask images, and the mapped one-dimensional feature vector is used as the feature of the first vignetting sample image. On the one hand, dimensionality reduction processing can remove redundant information in the convolutional features and retain key features, thereby reducing the computational complexity of the subsequent process. On the other hand, mapping the number of features in the one-dimensional feature vector to the number of multiple mask images can achieve a precise correspondence between features and mask images, which is highly intelligent.

[0119] In this embodiment of the disclosure, for each first vignetting sample image, a brightness compensation value for the first vignetting sample image is determined in the first feature processing layer based on the features of the first vignetting sample image and multiple mask images. In some embodiments, a brightness mapping relationship can be pre-constructed to characterize the correspondence between features and brightness compensation values, and the brightness compensation value of the first vignetting sample image is determined based on the feature-brightness mapping relationship of the first vignetting sample image. The brightness mapping relationship can be constructed based on multiple mask images, such as by extracting features from each mask image and constructing a brightness mapping relationship based on the brightness compensation values ​​corresponding to the features of the mask images. The brightness compensation values ​​corresponding to the features of the mask images can be calculated based on the shooting parameters and the brightness distribution of the preset image.

[0120] In other embodiments, for each first vignetting sample image, multiple mask images can be compared based on the features of the first vignetting sample image to determine the mask image that is most similar to the features of the first vignetting sample image. Based on the mask image that is most similar to the features of the first vignetting sample image, the brightness compensation value of the first vignetting sample image is determined.

[0121] In other embodiments, the number of features in the first vignetting sample image is the same as the number of the plurality of mask images; the step of determining the brightness compensation value of the first vignetting sample image in the first feature processing layer based on the features of the first vignetting sample image and the plurality of mask images includes:

[0122] In the first feature processing layer, each feature of the first vignetting sample image is normalized to obtain each normalized feature.

[0123] In the first feature processing layer, one of the multiple mask images is multiplied by a normalized feature, and the results of multiple multiplications are added together to obtain the vignetting simulation feature value;

[0124] Based on the vignetting simulation feature value, the brightness compensation value of the first vignetting sample image is determined; wherein the brightness compensation value is negatively correlated with the vignetting simulation feature value.

[0125] In this embodiment of the disclosure, after obtaining the features of the first vignetting sample image, the relationship between the number of features of the first vignetting sample image and the number of multiple mask images can be determined. If they are not the same, the method described above can be used to map the features of the first vignetting sample image to the number of multiple mask images. Alternatively, the number of features to be extracted can be set to the number of multiple mask images when extracting the features of the first vignetting sample image.

[0126] In this embodiment of the disclosure, each feature of the first vignetting sample image is normalized in the first feature processing layer to obtain each normalized feature. For example, when the feature of the first vignetting sample image is a one-dimensional feature vector, each element value in the one-dimensional feature vector can be normalized to obtain the normalized feature corresponding to each element value of the first feature vector. The normalization method can be the Sigmoid method, the min-max scaling method, the standardization method, etc.

[0127] In this embodiment of the present disclosure, in the first feature processing layer, one of the multiple mask images is multiplied by a normalized feature, and the results of multiple multiplications are added together to obtain the vignetting simulation feature value. The electronic device can pre-set the order of the multiple mask images and multiply them by the normalized feature in the order of the multiple masks. In this case, one normalized feature is multiplied by one mask image, and the normalized feature and the mask image have a one-to-one correspondence.

[0128] In this embodiment of the disclosure, since the normalized features and the mask images are in a one-to-one correspondence, the number of multiplication results is the same as the number of multiple mask images. After obtaining multiple multiplication results, the multiple multiplication results are added together to obtain the vignetting simulation feature value. The vignetting simulation feature value characterizes the vignetting distribution characteristics of the first vignetting sample image.

[0129] In this embodiment of the disclosure, the brightness compensation value of the first vignetting sample image is determined based on the vignetting simulation feature value; wherein the brightness compensation value is negatively correlated with the vignetting simulation feature value. For example, the brightness compensation value can be obtained by taking the reciprocal of the vignetting simulation feature value.

[0130] In this embodiment of the disclosure, the normalized feature is multiplied with the mask image, and the result of the multiplication is added to obtain the vignetting simulation feature value, which is an image with the same size and resolution as the mask image. The brightness compensation value can be obtained by dividing an image with the same resolution as the first vignetting sample image and all pixel values ​​of the pixels being 1 by the vignetting simulation feature value.

[0131] In this embodiment, each feature of the first vignetting sample image is normalized to obtain a normalized feature. Then, one of the mask images from multiple mask images is multiplied by a normalized feature, and the results of multiple multiplications are added together to obtain a vignetting simulation feature value. Based on the vignetting simulation feature value, the brightness compensation value of the first vignetting sample image is determined. This method can fuse the features of the first sample image with the vignetting features of the mask images under different shooting parameters, and can more accurately determine the vignetting distribution information (vignetting features) of the first vignetting sample image, thereby improving the accuracy of the brightness compensation value and thus improving the accuracy of removing the vignetting in the image.

[0132] In this embodiment of the disclosure, for each first vignetting sample image, the second feature processing layer performs brightness compensation on the first vignetting sample image based on the brightness compensation value of the first vignetting sample image to obtain the vignetting-free predicted image corresponding to the first vignetting sample image. In some embodiments, the pixel values ​​of each pixel in the first vignetting sample image can be updated based on the brightness compensation value of the first vignetting sample image to obtain the vignetting-free predicted image corresponding to the first vignetting sample image.

[0133] In other embodiments, as described above, if the brightness compensation value is an image with the same resolution as the first vignetting sample image, the brightness compensation value can be superimposed on the first vignetting sample image, and the pixel values ​​of the corresponding pixels can be multiplied to obtain the vignetting-free predicted image corresponding to the first vignetting sample image.

[0134] In this embodiment, the parameters of the neural network model are adjusted based on the difference between the predicted image after vignetting for each first vignetting sample image and the corresponding vignetting-free sample image, until a preset convergence condition is met, thus obtaining the vignetting removal model. The electronic device can determine the difference between the predicted image after vignetting for each first vignetting sample image and the corresponding vignetting-free sample image using methods such as Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM), and use this difference as a loss value to adjust the parameters of the neural network model. For example, optimization algorithms such as gradient descent and Adaptive Moment Estimation (Adam) can be used to update the parameters of the neural network model.

[0135] It should be noted that the electronic device can adjust the parameters of the neural network model multiple times until the convergence condition of the neural network model is met. The convergence condition can be that the difference between the predicted image after removing the vignetting corresponding to the first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image is less than a preset difference threshold. The convergence condition can also be that the number of times the parameters of the neural network model are adjusted reaches a preset number threshold. The preset difference threshold and the preset number threshold are both set values, and this disclosure embodiment does not limit them.

[0136] In this embodiment, for each first vignetting sample image, features of the first vignetting sample image are extracted. Based on the features of the first vignetting sample image and multiple mask images, the brightness compensation value of the first vignetting sample image is determined. Brightness compensation is then performed on the first vignetting sample image based on the brightness compensation value, resulting in a predicted image after vignetting removal corresponding to the first vignetting sample image. Then, based on the difference between the predicted image after vignetting removal corresponding to each first vignetting sample image and the corresponding image without vignetting, the parameters of the neural network model are adjusted until a preset convergence condition is met, thus obtaining the vignetting removal model. On the one hand, by continuously iterating and adjusting the parameters of the neural network model, the vignetting removal model can learn the complex relationship between the vignetting sample image and the brightness compensation value, enabling the model to accurately predict the brightness compensation value of the image to be processed, thereby improving the accuracy of vignetting removal. On the other hand, integrating the feature extraction layer, the first feature processing layer, and the second feature processing layer into a single neural network model achieves end-to-end learning, improving the model's efficiency and accuracy.

[0137] In some embodiments, the training sample dataset further includes second vignetting sample images captured by photographs; the vignetting removal model is obtained by training the neural network model collaboratively using a first sample set consisting of vignetting-free sample images and first vignetting sample images associated with the vignetting-free sample images, and a second sample set consisting of second vignetting sample images.

[0138] In this embodiment of the disclosure, the training sample dataset also includes a second vignetting sample image captured by a photograph. The second vignetting sample image may be an image with vignetting captured by an electronic device; the second vignetting sample image may also be an image with vignetting captured by other devices. After the electronic device acquires the image with vignetting captured by the photograph, it can perform data cleaning on the image, such as removing images with simple content or images with overly complex content, and use the image with vignetting after data cleaning as the second vignetting sample image.

[0139] In this embodiment of the disclosure, the vignetting removal model is obtained by collaboratively training a neural network model using a first sample set consisting of a vignetting-free sample image and a first vignetting sample image associated with the vignetting-free sample image, and a second sample set consisting of a second vignetting sample image. In some embodiments, the neural network model can be trained first using the first sample set. As mentioned above, the parameters of the neural network model can be adjusted based on the difference between the vignetting-removed predicted image corresponding to each first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image, until a preset convergence condition is met to obtain the first neural network model. Then, the first neural network model is trained using the second sample set, such as by using a generator-discriminator mechanism. The generator generates the vignetting-removed predicted image corresponding to the second vignetting sample image, and the discriminator judges the image. Based on the discriminator's judgment result, the parameters of the first neural network model are adjusted until the preset convergence condition is met to obtain the vignetting removal model.

[0140] In other embodiments, the neural network model can be trained simultaneously based on the first sample set and the second sample set. When an image is detected as a first vignetting sample image in the first sample set, the parameters of the neural network model are adjusted based on the difference between the predicted image after vignetting removal corresponding to the first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image. When an image is detected as a second vignetting sample image in the second sample set, the parameters of the neural network model are adjusted based on the discrimination result of the discriminator until the convergence condition is met, thus obtaining the vignetting removal model.

[0141] In this embodiment, a neural network model is trained by using a first sample set consisting of a first vignetting sample image and a first vignetting sample image associated with the vignetting sample image, and a second sample set consisting of a second vignetting sample image, to obtain a vignetting removal model. This can improve the diversity of training data, thereby enhancing the generalization ability of the vignetting removal model and making it more intelligent.

[0142] Figure 3 This is a schematic diagram illustrating the principle of an image processing method according to an exemplary embodiment, such as... Figure 3 As shown, the embodiments of this disclosure can be applied in the process of capturing images. The natural scene L33 is the shooting scene when capturing images. The imaging device L32 includes an acquisition module L321 that acquires images and converts light signals in the natural scene L33 into electrical signals (images to be processed), a processing module L322 that removes vignetting from the images acquired by the acquisition module L321 using the image processing method of the embodiments of this disclosure, and a display module L323 that displays the processed image after removing vignetting. The human visual system L31 refers to displaying the processed image through the display module, and the user views the processed image based on a user interface (such as a screen).

[0143] Figure 4 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 4 The process includes the following steps:

[0144] S41. Collect open-source data and create a training set using real and simulated vignetting.

[0145] In this embodiment of the disclosure, collecting open-source data involves obtaining a vignetting-free sample image and a second vignetting sample image. Creating a training set using real and simulated vignetting involves processing the vignetting-free sample image using a preset vignetting mask to generate a first vignetting sample image, thus obtaining a first sample set; and a second sample set consisting of the second vignetting sample image.

[0146] S42. Construct and train a neural network to predict the feature weights of each vignetting data point.

[0147] In this embodiment of the disclosure, the neural network is constructed by determining the feature extraction layer, the first feature processing layer, and the second feature processing layer of the neural network model. The neural network is trained by collaboratively training the neural network model using a first sample set and a second sample set. The feature weights for predicting each vignetting data point are determined by defining the vignetting simulation feature value and brightness compensation value for each first or second vignetting sample image.

[0148] S43. Use the trained neural network to remove vignetting from the image to be processed.

[0149] In this embodiment of the disclosure, the trained neural network is a vignetting removal model, that is, the vignetting removal model is used to process the image to be processed to obtain an image after removing vignetting.

[0150] Figure 5 This is an example diagram illustrating the effect of an image processing method according to an exemplary embodiment, wherein (a) is the image before vignetting removal, and (b) is the image after vignetting removal operation is performed on image (a) using the method of this embodiment. Figure 5 As can be seen, the method of this embodiment has a significant effect on removing vignetting and is highly accurate, without destroying the image content and features outside the vignetting area.

[0151] Figure 6 This is a block diagram of an image processing apparatus 600 according to an exemplary embodiment. Figure 6 As shown, the device mainly includes:

[0152] The first acquisition module 601 is configured to acquire the image to be processed;

[0153] The processing module 602 is configured to process the image to be processed based on a preset vignetting removal model to obtain an image after vignetting removal; wherein, the vignetting removal model is obtained by training a neural network model using a training sample dataset, the training sample dataset includes multiple sets of paired vignetting-free sample images and a first vignetting sample image, and the first vignetting sample image in the same set is generated by processing the vignetting-free sample image using a preset vignetting mask.

[0154] In some embodiments, the apparatus further includes:

[0155] The second acquisition module is configured to acquire vignetting-free sample images from the training sample dataset.

[0156] The third acquisition module is configured to acquire multiple mask images obtained by taking pictures of a preset scene based on different shooting parameters;

[0157] The fusion module is configured to perform a preset number of weighted fusions on any number of mask images among the multiple mask images for each sample image without dark corners, so as to obtain the target mask image corresponding to the preset number of fusions.

[0158] The mask processing module is configured to perform mask processing on each vignetting-free sample image using each target mask image associated with the vignetting-free sample image, thereby obtaining a first vignetting sample image corresponding to the vignetting-free sample image; wherein, the number of the first vignetting sample images corresponding to the vignetting-free sample image is the same as the preset number of times;

[0159] The training module is configured to train the neural network model using vignetting-free sample images from the training sample dataset and a first vignetting sample image associated with the vignetting-free sample images, to obtain the vignetting removal model.

[0160] In some embodiments, the neural network model includes a feature extraction layer, a first feature processing layer, and a second feature processing layer; the training module is further configured to, for each first vignetting sample image, extract features of the first vignetting sample image in the feature extraction layer to obtain features of the first vignetting sample image; for each first vignetting sample image, determine a brightness compensation value of the first vignetting sample image in the first feature processing layer based on the features of the first vignetting sample image and the multiple mask images; for each first vignetting sample image, perform brightness compensation on the first vignetting sample image in the second feature processing layer based on the brightness compensation value of the first vignetting sample image to obtain a vignetting-removed predicted image corresponding to the first vignetting sample image; adjust the parameters of the neural network model based on the difference between the vignetting-removed predicted image corresponding to each first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image, until a preset convergence condition is met to obtain the vignetting removal model.

[0161] In some embodiments, the number of features in the first vignetting sample image is the same as the number of the plurality of mask images; the training module is further configured to normalize each feature of the first vignetting sample image in the first feature processing layer to obtain each normalized feature; in the first feature processing layer, multiply one of the mask images in the plurality of mask images by one normalized feature, and add the multiple multiplication results to obtain a vignetting simulation feature value; based on the vignetting simulation feature value, determine the brightness compensation value of the first vignetting sample image; wherein the brightness compensation value is negatively correlated with the vignetting simulation feature value.

[0162] In some embodiments, the training module is further configured to extract features of the first vignetting sample image based on convolution operations at the feature extraction layer to obtain convolutional features of the first vignetting sample image; perform dimensionality reduction processing on the convolutional features of the first vignetting sample image to obtain a one-dimensional feature vector of the first vignetting sample image; map the number of features in the one-dimensional feature vector to the number of the multiple mask images, and use the mapped one-dimensional feature vector as the feature of the first vignetting sample image.

[0163] In some embodiments, the mask processing module is further configured to perform mask processing on the target pixels in the vignetting sample image using each target mask image associated with the vignetting sample image, and retain the pixel values ​​of the pixels other than the target pixels to obtain a first vignetting sample image corresponding to the vignetting sample image; wherein the pixel value of the target pixel is less than a preset pixel threshold.

[0164] In some embodiments, the training sample dataset further includes second vignetting sample images captured by photographs; the vignetting removal model is obtained by training the neural network model collaboratively using a first sample set consisting of vignetting-free sample images and first vignetting sample images associated with the vignetting-free sample images, and a second sample set consisting of second vignetting sample images.

[0165] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0166] Figure 7 This is a structural block diagram illustrating an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, personal digital processing computer, handheld device, computing device, in-vehicle device, tablet device, wearable device, camera, camcorder, etc.

[0167] Reference Figure 7 The electronic device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0168] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0169] Memory 704 is configured to store various types of data to support operation on electronic device 700. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, and videos. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0170] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0171] Multimedia component 708 includes a screen that provides an output interface between electronic device 700 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When electronic device 700 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0172] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0173] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0174] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 may detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or one of its components, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0175] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0176] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0177] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including executable instructions or a computer program, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0178] A non-transitory computer-readable storage medium, wherein when a computer program or instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the image processing methods described above in the embodiments of this disclosure.

[0179] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0180] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0181] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is processed based on a preset vignetting removal model to obtain an image after vignetting removal; wherein, the vignetting removal model is obtained by training a neural network model using a training sample dataset, the training sample dataset includes multiple sets of paired vignetting-free sample images and a first vignetting sample image, the first vignetting sample image in the same set is generated by processing the vignetting-free sample image using a preset vignetting mask.

2. The method according to claim 1, characterized in that, The method further includes: Obtain vignetting-free sample images from the training sample dataset; Obtain multiple mask images obtained by taking pictures of a preset scene based on different shooting parameters; For each sample image without dark corners, a weighted fusion is performed on any number of mask images from the multiple mask images a preset number of times to obtain the target mask image corresponding to the preset number of times; For each vignetting-free sample image, a masking process is performed on the vignetting-free sample image using each target mask image associated with the vignetting-free sample image to obtain a first vignetting sample image corresponding to the vignetting-free sample image; wherein, the number of the first vignetting sample images corresponding to the vignetting-free sample image is the same as the preset number of times; The neural network model is trained using vignetting-free sample images from the training sample dataset and a first vignetting sample image associated with the vignetting-free sample images to obtain the vignetting removal model.

3. The method according to claim 2, characterized in that, The neural network model includes a feature extraction layer, a first feature processing layer, and a second feature processing layer; training the neural network model using vignetting-free sample images from the training sample dataset and first vignetting sample images associated with the vignetting-free sample images to obtain the vignetting removal model includes: For each first vignetting sample image, the features of the first vignetting sample image are extracted in the feature extraction layer to obtain the features of the first vignetting sample image. For each first vignetting sample image, the brightness compensation value of the first vignetting sample image is determined in the first feature processing layer based on the features of the first vignetting sample image and the multiple mask images. For each first vignetting sample image, the brightness of the first vignetting sample image is compensated based on the brightness compensation value of the first vignetting sample image in the second feature processing layer to obtain the predicted image after removing the vignetting corresponding to the first vignetting sample image. Based on the difference between the predicted image after vignetting corresponding to each first vignetting sample image and the vignetting-free sample image corresponding to the first vignetting sample image, the parameters of the neural network model are adjusted until a preset convergence condition is met, thus obtaining the vignetting removal model.

4. The method according to claim 3, characterized in that, The number of features in the first vignetting sample image is the same as the number of the multiple mask images; determining the brightness compensation value of the first vignetting sample image in the first feature processing layer based on the features of the first vignetting sample image and the multiple mask images includes: In the first feature processing layer, each feature of the first vignetting sample image is normalized to obtain each normalized feature. In the first feature processing layer, one of the multiple mask images is multiplied by a normalized feature, and the results of multiple multiplications are added together to obtain the vignetting simulation feature value; Based on the vignetting simulation feature value, the brightness compensation value of the first vignetting sample image is determined; wherein the brightness compensation value is negatively correlated with the vignetting simulation feature value.

5. The method according to claim 3 or 4, characterized in that, The step of extracting features from the first vignetting sample image in the feature extraction layer to obtain the features of the first vignetting sample image includes: The feature extraction layer extracts features from the first vignetting sample image based on convolution operations to obtain the convolutional features of the first vignetting sample image; The convolutional features of the first vignetting sample image are subjected to dimensionality reduction processing to obtain a one-dimensional feature vector of the first vignetting sample image; The number of features in the one-dimensional feature vector is mapped to the number of the multiple mask images, and the mapped one-dimensional feature vector is used as the feature of the first vignetting sample image.

6. The method according to claim 2, characterized in that, The step of performing masking processing on the vignette sample image using each target mask image associated with the vignette sample image to obtain the first vignette sample image corresponding to the vignette sample image includes: The target pixels in the no-dark-corner sample image are masked using each target mask image associated with the no-dark-corner sample image, and the pixel values ​​of the pixels other than the target pixels are retained to obtain the first dark-corner sample image corresponding to the no-dark-corner sample image; wherein the pixel value of the target pixel is less than a preset pixel threshold.

7. The method according to claim 1, characterized in that, The training sample dataset also includes second vignetting sample images captured by photography; the vignetting removal model is obtained by training the neural network model using a first sample set composed of vignetting-free sample images and first vignetting sample images associated with vignetting-free sample images, and a second sample set composed of second vignetting sample images.

8. An image processing apparatus, characterized in that, The device includes: The first acquisition module is configured to acquire the image to be processed. The processing module is configured to process the image to be processed based on a preset vignetting removal model to obtain an image after vignetting removal; wherein, the vignetting removal model is obtained by training a neural network model using a training sample dataset, the training sample dataset includes multiple sets of paired vignetting-free sample images and a first vignetting sample image, the first vignetting sample image in the same set is generated by processing the vignetting-free sample image using a preset vignetting mask.

9. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the image processing method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the image processing method according to any one of claims 1 to 7.

Images