Image processing method and related equipment

Abstract

The present application provides an image processing method and related equipment. The method generates a noise image based on the parameters of a sensor used to determine the noise size of an original (RAW) image, and superimposes the noise image on a red, green, blue (RGB) image obtained by image signal processing of the RAW image. This is equivalent to simulating the noise of the RAW image on the RGB image, thereby adding granular texture details to the RGB image, improving the texture of image details, and making the image look better.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related equipment. Background Art

[0002] Image texture is an evaluation index of image expressiveness. An important factor affecting image texture is the richness of the texture in the image. The richer the texture in the image, the richer the local details in the image, so the image has more detailed texture and the image looks better.

[0003] The process of taking photos by electronic devices, that is, the process from the image sensor acquiring a noisy RAW image to the final photo output, includes the process of performing noise reduction algorithm processing on the image. At present, neural network noise reduction methods, traditional noise reduction methods, etc. cannot or cannot accurately control the noise reduction intensity for the details of the image to be processed, which can easily lead to smearing (that is, some areas of the image to be processed appear blurred or unnatural), resulting in less image detail information and poor image texture. Summary of the invention

[0004] The present application provides an image processing method and related equipment, which can improve image quality.

[0005] In a first aspect, an image processing method is provided, which is applied to an electronic device, and the method includes: generating a first noise image according to a first RGB image and sensor parameters, wherein the first RGB image is obtained by performing image signal processing on a RAW image, the sensor parameters include the sensitivity used when the RAW image is captured by the image sensor, and Poisson noise parameters and Gaussian noise parameters inherent in the image sensor, and the noise size of the RAW image is determined based on the sensor parameters; and superimposing the first noise image on the first RGB image to obtain a second RGB image.

[0006] The above solution can improve the texture of image details, thereby making the picture look better.

[0007] It can be understood that the noise level of the RAW image is determined based on the sensor parameters, or in other words, the noise of the RAW image is associated with the sensor parameters. Therefore, the above solution can be understood as simulating the noise of the RAW image based on the first RGB image and the sensor parameters; simulating the noise of the RAW image on the first RGB image to increase the grainy texture details on the first RGB image can improve the image detail texture, so that the picture has a better look and feel.

[0008] A related technology may be to extract noise from an actually captured image (e.g., a RAW image) and then superimpose it on an RGB image. It is understandable that in addition to noise, the actually captured image also has the original information of the image. The noise extracted from the actually captured image is likely to carry a portion of the original information of the image. When this portion of information is superimposed on the RGB image, it is likely to produce artifacts (antifacts), thereby reducing the image quality. The image processing method provided in the present application can reduce the possibility of artifacts while enhancing the image texture by simulating the noise of the RAW image, compared to the related technology.

[0009] In a possible embodiment, a first noise image is generated according to a first RGB image and sensor parameters, including: obtaining a second noise image according to the first RGB image, sensitivity, Poisson noise parameters, and Gaussian noise parameters; determining a noise adjustment weight corresponding to the first pixel according to a brightness value of the first pixel of the first RGB image, the noise adjustment weight being used to adjust the noise level of the second noise image; and multiplying the second noise image by the noise adjustment weight to obtain the first noise image.

[0010] It is understandable that the noise on a RAW image is generally much greater than the noise on an RGB image. If it is directly superimposed on the first RGB image, the signal-to-noise ratio will be significantly reduced, thereby reducing the image quality. In the above scheme, the second noise image is the noise on the RAW image simulated based on the sensor parameters. The noise size of the second noise image is adjusted by the brightness value of the RGB image, which can avoid excessive noise and reduced image quality after the first RGB image is superimposed with the second noise image.

[0011] In a possible embodiment, obtaining a second noise image according to a first RGB image, sensitivity, Poisson noise parameters, and Gaussian noise parameters includes: sampling Gaussian noise and Poisson noise corresponding to the first RGB image according to the first RGB image, sensitivity, Poisson noise parameters, and Gaussian noise parameters to obtain a third noise image, the third noise image including three sub-noise images corresponding to three color channels of the first RGB image respectively; downsampling the three sub-noise images to obtain a fourth noise image, the fourth noise image is a single-channel image, and pixels of the fourth noise image are arranged in a Bayer format; and stacking three fourth noise images to obtain a second noise image, the three fourth noise images corresponding to the three color channels respectively.

[0012] In the above scheme, the third noise image obtained by sampling the first RGB image contains color noise, which seriously affects the image quality. After downsampling to a single-channel noise image (i.e., the fourth noise image), there is no color noise, but noise that changes with brightness. Therefore, the subsequent noise image superimposed on the first RGB image also has no color noise, increasing the granular texture details of the image without reducing the image quality, further ensuring the texture of the image.

[0013] In a possible embodiment, according to the first RGB image, sensitivity, Poisson noise parameters and Gaussian noise parameters, Gaussian noise and Poisson noise corresponding to the first RGB image are sampled to obtain a third noise image, including: determining the Poisson noise intensity of the three color channels according to the pixel values, sensitivity and Poisson noise parameters of the three color channels of the first RGB image; determining the Gaussian noise intensity of the three color channels according to the sensitivity and Gaussian noise parameters; and randomly selecting corresponding Poisson noise values ​​and Gaussian noise values ​​pixel by pixel for each color channel of the three color channels based on the Poisson noise intensity and Gaussian noise intensity of the three color channels, respectively, to obtain a third noise map.

[0014] In a possible embodiment, the noise adjustment weight of the first pixel is determined according to the brightness value of the first pixel of the first RGB image, including: generating a corresponding first grayscale image according to the first RGB image, the brightness value of the first RGB image corresponds to the grayscale value of the first grayscale image pixel by pixel; setting the grayscale values ​​of the second pixel and the third pixel in the first grayscale image to 0 to obtain a second grayscale image, the grayscale value of the second pixel is greater than the first grayscale threshold, the grayscale value of the third pixel is less than the second grayscale threshold, and the first grayscale threshold is greater than the second grayscale threshold; determining the noise adjustment weight according to the grayscale value of the fourth pixel of the second grayscale image.

[0015] It can be understood that the second pixel is a pixel that is too bright in the first grayscale image, and the third pixel is a pixel that is too dark in the first grayscale image. Restoring these two parts of pixels to 0 can control the noise in the overexposed and underexposed areas of the first RGB image and reduce the occurrence of artifacts.

[0016] In a possible embodiment, the noise adjustment weight is determined according to the grayscale value of the fourth pixel of the second grayscale image, including: determining a preset noise adjustment gain according to the brightness value of the first RGB image, the preset noise adjustment gain being less than 1; and multiplying the preset noise adjustment gain by the grayscale value of the fourth pixel to obtain the noise adjustment weight.

[0017] It can be understood that the noise on a RAW image is generally much larger than that on an RGB image. The preset noise adjustment gain is less than 1, which can overall darken the noise image to be superimposed on the first RGB image. In other words, it can reduce the noise level of the first noise image, and can reduce the situation in which the noise is too large and the signal-to-noise ratio is too poor after the first RGB image is superimposed with the noise, thereby reducing the image quality, thereby further ensuring the image texture.

[0018] It can also be understood that the preset noise adjustment gain adjusts the noise size of the final second RGB image by adjusting the noise size of the first noise image. In the second grayscale image, the larger the grayscale value of the fourth pixel is, the higher the grayscale value of a pixel in the noise weight image is, which means that the pixel is brighter and the noise adjustment weight of the pixel is higher.

[0019] In some embodiments, from the user's perspective, the brighter areas (eg Figure 1A If the content presented in the nose bridge area 101 in the image is too smooth and has no graininess, the three-dimensional sense of the area will be significantly weakened, the image realism will be reduced, and the image texture will be affected. Figure 1B The brighter the pixel in the bridge of the nose region 201), the higher the noise adjustment weight of the pixel, the more obvious the graininess will be, and the obvious graininess will make it more three-dimensional, more in line with the real face, and enhance the realism and texture of the image. For darker areas, since the brightness of the dark area itself is relatively low, if a relatively large noise is superimposed, the content in the darker area may have a heavier graininess, which may make the content in the darker area appear more messy, resulting in a worse signal-to-noise ratio and a poor user experience. In the present application, the darker the pixel in the darker area, the lower the noise adjustment weight of the pixel, and the increase in graininess of the second RGB image compared to the first RGB image will not be too obvious. For example, Figure 1B The zygomatic area 203 in the Figure 1A In the cheekbone area 103, the added graininess is not obvious, which can reduce the deterioration of the signal-to-noise ratio in the darker area and ensure the user's viewing experience.

[0020] In some embodiments, the preset noise adjustment gain is determined according to the proportion of pixels in the darker area, pixels in the brighter area, and pixels in the remaining area in the first grayscale image.

[0021] In some embodiments, the preset noise adjustment gain is determined according to an average brightness value (or an average grayscale value) of pixels in the first grayscale image.

[0022] In some embodiments, a preset noise adjustment gain is determined based on sensor parameters using a neural network model.

[0023] In a possible embodiment, the second noise image is multiplied by the noise adjustment weight to obtain the first noise image, including: the fifth pixel of each fourth noise image corresponds one-to-one to the first pixel of each color channel of the three color channels, and each fifth pixel is multiplied by the noise adjustment weight corresponding to each first pixel to obtain three fifth noise images, and the first noise image includes the three fifth noise images.

[0024] In a possible embodiment, superimposing the first noise image on the first RGB image to obtain the second RGB image includes: superimposing the fifth noise image on the image of each of the three color channels in the first RGB image to obtain the second RGB image.

[0025] In a possible embodiment, the method further includes: in response to a user's photo-taking operation, acquiring a first RGB image, a RAW image, and sensor parameters.

[0026] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0027] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0029] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0030] It can be understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1A A schematic diagram of an example of how image texture is affected due to lack of image detail information;

[0032] Figure 1B A schematic diagram of an example of improving image texture using an image processing method provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of an image processing method 300 provided in an embodiment of the present application;

[0034] Figure 3 An implementation of S310 in the method 300 provided in the embodiment of the present application;

[0035] Figure 4 An implementation of S311 in the method 300 provided in the embodiment of the present application;

[0036] Figure 5 A schematic diagram of an example of downsampling (S3112) provided in an embodiment of the present application;

[0037] Figure 6 An implementation of S312 in the method 300 provided in the embodiment of the present application;

[0038] Figure 7 A schematic diagram of an image processing method 300' provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the architecture (including software system and some hardware) of the embodiment of the present application;

[0040] Fig. 9 The figure shows a hardware system of an electronic device 400 suitable for the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0042] Image texture is an evaluation index of image expressiveness. An important factor affecting image texture is the richness of the texture in the image. The richer the texture in the image, the richer the local details in the image, so the image has more detailed texture and the image looks better.

[0043] The process of taking photos by electronic devices, that is, the process from the image sensor obtaining the raw format (RAW) image with noise to the final photo output, includes the process of performing noise reduction algorithm processing on the image. At present, neural network noise reduction methods, traditional noise reduction methods, etc. cannot or cannot accurately control the noise reduction intensity for the details of the image to be processed, which can easily lead to smearing (that is, some areas of the image to be processed appear blurred or unnatural), resulting in less image detail information and poor image texture.

[0044] The present application provides an image processing method, which generates a noise image based on the parameters of a sensor used to determine the noise size of a RAW image, and superimposes the noise image on an RGB image obtained by image signal processing of the RAW image. This is equivalent to simulating the noise of the RAW image on the RGB image, thereby adding granular texture details to the RGB image, improving the image detail texture, and making the image look better.

[0045] A related technology may be to extract noise from an actually captured image (e.g., a RAW image) and then superimpose it on an RGB image. It is understandable that in addition to noise, the actually captured image also has the original information of the image. The noise extracted from the actually captured image is likely to carry a portion of the original information of the image. When this portion of information is superimposed on the RGB image, it is likely to produce artifacts (antifacts), thereby reducing the image quality. The image processing method provided in the present application can reduce the possibility of artifacts while enhancing the image texture by simulating the noise of the RAW image, compared to the related technology.

[0046] It should be noted that the image sensor for acquiring the RAW image may be the image sensor of the electronic device or the image sensor of another electronic device. That is to say, the RAW image and the RGB image may be obtained by taking pictures with the camera device of the electronic device; or the RAW image and the RGB image may be obtained by the electronic device from another electronic device, and the RAW image and the RGB image may be obtained by the other electronic device based on its own camera device (the RAW image is taken by the other electronic device based on its own image sensor).

[0047] Figure 1A Schematic diagram of an example in which image texture is affected by the lack of image detail information. Figure 1B A schematic diagram of an example of improving image texture using the image processing method provided in an embodiment of the present application.

[0048] Figure 1A and Figure 1B Each image contains a portrait (actually an RGB image, illustrated by black and white line drawings), as well as enlarged images of three local positions (for example, the bridge of the nose, the wing of the nose, and the cheekbone area) in the image (actually an RGB image, illustrated by grayscale images; the contrast effect of RGB images is more obvious).

[0049] like Figure 1A As shown, the nose bridge area 101, nose wing area 102 and cheekbone area 103 are all obviously smeared, which makes the three-dimensional sense of the nose bridge, nose wing and cheekbone (areas with obvious ups and downs on these people's faces) obviously weakened, the image realism is reduced, and the image texture is affected.

[0050] like Figure 1BAs shown, compared with the nose bridge area 101, the nose wing area 102 and the cheekbone area 103, the nose bridge area 201, the nose wing area 202 and the cheekbone area 203 all have obvious graininess, and the three-dimensional sense is more consistent with the real face, which improves the image realism and texture. In the RGB image, the image realism and texture are more intuitive.

[0051] It should be noted that the above only uses three local areas as examples to illustrate the granular texture brought by the image processing method provided by the present application. Figure 1B The images shown have an overall increase in grainy texture detail, and the texture of the entire image has been improved.

[0052] In order to more clearly illustrate the embodiments of the present application, some technical terms involved in the embodiments of the present application are first introduced below.

[0053] 1. Sensor or image sensor: uses the photoelectric conversion function of a photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. Alternatively, it may also be called a photosensitive element or a photosensitive device in this application.

[0054] Image sensors are mainly divided into two categories: charge coupled device camera (CCD) and complementary metal-oxide-semiconductor (CMOS) phototransistors. Exemplarily, the image sensor involved in the present application may be a CMOS phototransistor.

[0055] RAW image: The original (RAW) format image (or RAW data) is the original data that the CMOS or CCD image sensor converts the captured light source signal into a digital signal. It is lossless and contains the original color information of the object. The original image can be one frame or multiple frames. The original image can be a Bayer format image or a grayscale image, or part of it can be a Bayer format image and part of it can be a grayscale image. It can be obtained as needed, and the embodiments of the present application do not impose any restrictions on this.

[0056] 2. Grayscale image is also called grayscale image. Grayscale image is a single-channel image used to represent different brightness levels, with the brightest being all white and the darkest being all black. In other words, each pixel in the grayscale image corresponds to a different degree of brightness between black and white.

[0057] 3. ISO (International Organization for Standardization): ISO is a way of expressing the sensitivity of a camera. The larger the ISO value, the more sensitive the camera is to light, and the brighter the photo will be, and vice versa. However, too high an ISO value will increase the noise in the photo. While keeping other conditions (such as ambient brightness, exposure time, etc.) unchanged, the larger the ISO of the image sensor, the brighter the image, that is, the greater the exposure; conversely, the smaller the ISO value, the smaller the image, that is, the smaller the exposure.

[0058] 4. Poisson noise, Poisson noise intensity, Poisson noise value and Poisson noise parameters.

[0059] Poisson noise, or shot noise, is a type of noise that can be modeled by a Poisson process. In electronics, Poisson noise is due to the properties of discrete electric charges. Poisson noise is also generated when optical devices count photons, which is related to the uncertainty of photons.

[0060] The Poisson noise intensity (or Poisson distribution) includes the mean and variance of the Poisson distribution.

[0061] The Poisson noise value in this application is a noise value randomly selected based on the Poisson noise intensity, or a value randomly selected in the Poisson distribution.

[0062] The Poisson noise parameter is an inherent noise parameter of the image sensor. In the present application, the Poisson noise parameter is used to calculate the Poisson noise intensity.

[0063] 5. Gaussian noise, Gaussian noise intensity, Gaussian noise value and Gaussian noise parameters.

[0064] Gaussian noise refers to a type of noise whose probability density function follows a Gaussian distribution (i.e., normal distribution). Common Gaussian noises include fluctuation noise, cosmic noise, thermal noise, shot noise, etc. In addition to the commonly used noise suppression methods, mathematical statistics methods are often used to suppress Gaussian noise.

[0065] The Gaussian noise intensity (or Gaussian distribution) includes the expectation and variance of the Gaussian distribution.

[0066] The Gaussian noise value in this application is a noise value randomly selected based on the Gaussian noise intensity, or a value randomly selected in the Gaussian distribution.

[0067] The Gaussian noise parameter is an inherent noise parameter of the image sensor. In the present application, the Gaussian noise parameter is used to calculate the Gaussian noise intensity.

[0068] 6. Sampling and downsampling.

[0069] In signal processing, sampling is the process of converting a continuous-time signal into a discrete-time signal.

[0070] In the present application, the sampling of Gaussian noise and Poisson noise can be understood as randomly collecting Gaussian noise values ​​and Poisson noise values ​​in Gaussian noise distribution and Poisson noise distribution.

[0071] In the field of digital signal processing, downsampling, also known as acquisition reduction, is a multi-rate digital signal processing technique or the process of reducing the signal sampling rate, usually used to reduce the data transmission rate or data size.

[0072] In this application, the noise image is downsampled to obtain a noise image with fewer pixel values. Figure 5 Related description.

[0073] 7. Noise reduction: refers to the process of reducing noise in an image. Common methods include mean filtering, Gaussian filtering, bilateral filtering, etc. The noise reduction involved in this application can be included in the ISP process.

[0074] Figure 2 A schematic diagram of an image processing method 300 provided in an embodiment of the present application.

[0075] S310, generating a first noise image according to a first RGB image and sensor parameters.

[0076] The first RGB image is obtained by performing image signal processing (ISP) on the RAW image. Exemplarily, the ISP process includes noise reduction processing.

[0077] The sensor parameters include the ISO used when shooting RAW images based on the image sensor, and the Poisson noise parameters and Gaussian noise parameters inherent in the image sensor.

[0078] It should be noted that the image sensor for acquiring the RAW image may be the image sensor of the electronic device executing method 300, or may be the image sensor of other electronic devices. That is, the RAW image and the first RGB image may be obtained by photographing the camera device of the electronic device; or may be obtained by the electronic device from other electronic devices, and the RAW image and the first RGB image are obtained by the other electronic device based on its own camera device (the RAW image is photographed by the other electronic device based on its own image sensor).

[0079] Exemplarily, ISO is set by an automatic exposure algorithm based on the shooting environment.

[0080] Exemplarily, the Poisson noise parameters and the Gaussian noise parameters may be provided by the manufacturer of the image sensor, and the Poisson noise parameters and the Gaussian noise parameters of the same model of image sensors may be the same. Optionally, when the image sensor in method 300 is an image sensor of another electronic device, method 300 further includes: the electronic device executing method 300 obtains the sensor parameters from the electronic device where the image sensor is located.

[0081] It can be understood that the noise level of the RAW image is determined based on the sensor parameters, or in other words, the noise of the RAW image is associated with the sensor parameters. Therefore, S310 can be understood as simulating the noise of the RAW image based on the first RGB image and the sensor parameters.

[0082] It can also be understood that compared with other image noises (such as salt and pepper noise, multiplicative noise, etc.), Gaussian noise and Poisson noise are more evenly distributed on the image, more integrated with the image, and more natural when superimposed on the image, with less impact on the content information of the image itself.

[0083] S320: Superimpose the first noise image on the first RGB image to obtain a second RGB image.

[0084] Exemplarily, the first noise image includes three sub-noise images corresponding to the three color channels (R channel, G channel and B channel) of the first RGB image, and the corresponding sub-noise images are superimposed on the images of the three color channels to obtain the second RGB image.

[0085] S320 can be understood as simulating the noise of the RAW image on the first RGB image to increase the grainy texture details on the first RGB image.

[0086] Therefore, the above solution can improve the texture of image details, thereby making the picture look better.

[0087] In one implementation, before S310, method 300 further includes: in response to a user's operation of taking a photo, acquiring a first RGB image, a RAW image and sensor parameters from itself. The second RGB image is the image that is finally sent for display.

[0088] Alternatively, the method 300 may be applied to a video scene. In response to a user's operation of shooting a video, the first RGB image, the RAW image, and the sensor parameters are obtained from the sensor itself. Each image frame included in the video sent for display may be a second RGB image obtained by processing based on the method of the present application.

[0089] Further examples are given below for method 300 .

[0090] Figure 3This is an implementation of S310 in the method 300 provided in the embodiment of the present application. Figure 3 As shown, it includes S311 to S313.

[0091] S311, obtaining a second noise image according to the first RGB image, sensitivity, Poisson noise parameters and Gaussian noise parameters.

[0092] It can be understood that the second noise image has the same size as the first RGB image, and the pixels of the second noise image correspond one-to-one with the first pixels. The Poisson noise parameter is used to determine the Poisson noise related to the first RGB image, and the Gaussian noise parameter is used to determine the Gaussian noise unrelated to the first RGB image.

[0093] A possible example of S311 will be given below.

[0094] S312: Determine a noise adjustment weight corresponding to the first pixel according to the brightness value of the first pixel of the first RGB image.

[0095] The noise adjustment weight is used to adjust the noise level of the second noise image.

[0096] Exemplarily, the noise adjustment weight may be expressed as a weight value corresponding to the first pixel, or as a noise adjustment weight map having the same size as the first RGB.

[0097] It is understandable that the brightness of an image is generally represented by a grayscale image (grayscale image) of the image. A grayscale image is a single-channel image used to represent different degrees of brightness, with the brightest being all white and the darkest being all black. In other words, each pixel in a grayscale image corresponds to a different degree of brightness between black and white, and the specific brightness of the pixel can be reflected by the grayscale value of the pixel. The larger the grayscale value, the brighter the pixel. For example, the noise adjustment weight map is generated based on the grayscale image of the first RGB image.

[0098] A possible example of S312 will be given below.

[0099] S313: multiply the second noise image by the noise adjustment weight to obtain a first noise image.

[0100] Exemplarily, given that the pixels of the second noise image correspond one-to-one with the first pixels, the second noise image can also be understood as a three-channel (not completely equivalent to the color channel) noise image (or a noise image corresponding to the three color channels of the first RGB image). Since the noise adjustment weight corresponds to the first pixel, the pixels of the second noise image also correspond to the noise adjustment weight. The second noise image is multiplied pixel by pixel by the noise adjustment weight to obtain the first noise image, and the third noise image can also be understood as a three-channel noise image.

[0101] Exemplarily, when the noise adjustment weight is implemented in the form of a single-channel grayscale image, the images of the three channels in the second noise image are multiplied pixel by pixel with the noise adjustment weight image.

[0102] It is understandable that the noise on a RAW image is generally much greater than the noise on an RGB image. If it is directly superimposed on the first RGB image, the signal-to-noise ratio will be significantly reduced, thereby reducing the image quality. In the above scheme, the second noise image is the noise on the RAW image simulated based on the sensor parameters. The noise size of the second noise image is adjusted by the brightness value of the RGB image, which can avoid excessive noise and reduced image quality after the first RGB image is superimposed with the second noise image.

[0103] Figure 4 This is an implementation of S311 in the method 300 provided in the embodiment of the present application. Figure 4 As shown, it includes S3111 to S3113.

[0104] S3111, according to the first RGB image, sensitivity, Poisson noise parameters and Gaussian noise parameters, Gaussian noise and Poisson noise corresponding to the first RGB image are sampled to obtain a third noise image.

[0105] The third noise image includes three sub-noise images corresponding to the three color channels of the first RGB image respectively.

[0106] It can be understood that the third noise image of the same size is generated according to the size of the first RGB image, that is, the sizes of the three sub-noise images corresponding to the three color channels of the first RGB image are the same as the size of the first RGB image. The first pixel of an image of a certain color channel in the first RGB image corresponds one-to-one to a pixel point on the corresponding sub-noise image, that is, the first pixel of the first RGB image corresponds one-to-one to a pixel of the third noise image.

[0107] It can also be understood that sampling is performed on each pixel of the third noise image to obtain a Poisson noise value and a Gaussian noise value, that is, sampling is performed to obtain Gaussian noise and Poisson noise corresponding to the first RGB image.

[0108] As an implementation of S3111, it may include step 1 to step 3. Specifically, the Poisson noise intensity and the Gaussian noise intensity are determined respectively through step 1 and step 2, and then the Gaussian noise value and the Poisson noise value corresponding to the first RGB image are randomly selected according to the Poisson noise intensity and the Gaussian noise intensity.

[0109] Step 1: Determine the Poisson noise intensity of the three color channels according to the pixel values, sensitivities and Poisson noise parameters of the three color channels of the first RGB image.

[0110] It can be understood that the Poisson noise intensity includes the mean and variance of the Poisson noise. The Poisson noise intensity in this application can also be understood as the Poisson noise distribution.

[0111] Exemplarily, the mean and variance of Poisson noise are shown in Formula 1 below.

[0112] σ possion (i, j, c) = K × ISO × I (i, j, c) (Formula 1)

[0113] Wherein, (i, j, c) represents the pixel of the third noise image, wherein c represents the channel number corresponding to the pixel of the third noise image (the value can be 0, 1, 2, corresponding to the R channel, G channel and B channel respectively), i represents the number of rows of pixels of the third noise image, j represents the number of columns of pixels of the third noise image, K is the Poisson noise parameter, ISO is the sensitivity, I(i, j, c) represents the pixel value of the first pixel, σ possion (i, j, c) represents the mean or variance of the Poisson noise of the first pixel.

[0114] It can be understood that the intensity of Poisson noise is related to the pixel value of the image. For example, in Formula 1, the intensity of the Poisson noise of the RAW image is simulated based on the pixel value of the first RGB image. Generally speaking, the brightness of an RGB image is higher than that of its corresponding RAW image. Therefore, the intensity of the Poisson noise simulated according to Formula 1 is generally greater than the intensity of the Poisson noise of the RAW image. In the present application, the noise intensity can be reduced by adjusting the weight of the noise (that is, in the present application, the noise size of the second noise image is reduced by adjusting the weight of the noise), which can avoid excessive noise and reduced image quality after the first RGB image is superimposed on the second noise image.

[0115] Step 2: Determine the Gaussian noise intensity of the three color channels according to the sensitivity and Gaussian noise parameters.

[0116] It can be understood that the Gaussian noise intensity includes the mean and variance of the Gaussian noise. The Gaussian noise intensity in this application can also be understood as the Gaussian noise distribution.

[0117] Exemplarily, the mean of Gaussian noise is 0, and the variance is as shown in Formula 2 below.

[0118] σ gauss (i, j, c) = B × ISO × ISO (Formula 2)

[0119] Where (i, j, c) and ISO refer to the relevant description of formula 1, B is the Gaussian noise parameter, σ gauss (i, j, c) represents the variance of the Gaussian noise of the first pixel.

[0120] Step 3: Based on the Poisson noise intensity and the Gaussian noise intensity of the three color channels, respectively, for each color channel in the three color channels, the pixel-by-pixel Poisson noise intensity and the Gaussian noise intensity are obtained to obtain a third noise map.

[0121] Exemplarily, after determining the Poisson noise intensity based on the above formula 1, sampling is performed channel by channel and pixel by pixel to obtain the Poisson noise value (or the size of the Poisson noise), that is, the Poisson noise of the three sub-noise images of the third noise image is obtained, that is, the Poisson noise of the third noise image is composed of the Poisson noise value of each pixel.

[0122] Exemplarily, after determining the Gaussian noise intensity based on the above formula 2, sampling is performed channel by channel and pixel by pixel to obtain the Gaussian noise value (or the size of the Gaussian noise), that is, the Gaussian noise of the sub-noise image of the third noise image is obtained, that is, the Gaussian noise of the third noise image is composed of the Gaussian noise value of each pixel.

[0123] The Poisson noise sampled in the above manner is stacked with the Gaussian noise to obtain the final third noise map.

[0124] It can be understood that in S3111, the third noise image obtained by sampling the first RGB image appears as color noise, so the third noise image can be understood as an RGB image including three color channels.

[0125] S3112, downsampling the three sub-noise images to obtain a fourth noise image.

[0126] The fourth noise image is a single-channel image, and the pixels of the fourth noise image are arranged according to the RAW Bayer pattern. Therefore, the fourth noise image can be understood as a grayscale image.

[0127] Figure 5 A schematic diagram of an example of downsampling (S3112) provided in an embodiment of the present application.

[0128] like Figure 5 As shown, the third noise image can be understood as an RGB image including three color channels (as described above), and the fourth noise image is a single-channel image arranged in a Bayer format. Therefore, the downsampling involved in the present application can also be understood as mapping the third noise image in a Bayer format.

[0129] like Figure 5As shown, the fourth noise image is arranged in a Bayer format (taking 4 pixels as an example, the protection scope is not limited). The position where the R pixel is arranged in the fourth noise image (for example, (i#1, j#1)) collects the pixel value of the R pixel at the corresponding position (i#1, j#1) of the R channel image in the third noise image. Similarly, the position where the G pixel is arranged in the fourth noise image collects the pixel value of the G pixel at the corresponding position of the G channel image in the third noise image; the position where the B pixel is arranged in the fourth noise image collects the pixel value of the B pixel at the corresponding position of the B channel image in the third noise image.

[0130] It can be understood that, through the downsampling method in the present application, the color noise in the third noise image can be removed, and the obtained fourth noise image appears as noise that varies with brightness.

[0131] S3113, stack three fourth noise images to obtain a second noise image.

[0132] The three fourth noise images correspond to the three color channels respectively, so as to facilitate the subsequent superposition of noise on the three color channels of the first RGB image respectively.

[0133] In the above scheme, the third noise image obtained by sampling the first RGB image contains color noise, which seriously affects the image quality. After downsampling to a single-channel noise image (i.e., the fourth noise image), there is no color noise, but noise that changes with brightness. Therefore, the subsequent noise image superimposed on the first RGB image also has no color noise, increasing the granular texture details of the image without reducing the image quality, further ensuring the texture of the image.

[0134] Figure 6 This is an implementation of S312 in the method 300 provided in the embodiment of the present application. In this implementation, the noise adjustment weight is expressed as a grayscale image for reflecting the noise adjustment weight.

[0135] S3121, generating a corresponding first grayscale image according to the first RGB image.

[0136] The brightness value of the first RGB image corresponds to the grayscale value of the first grayscale image pixel by pixel.

[0137] S3122, setting the grayscale values ​​of the second pixel and the third pixel in the first grayscale image to 0 to obtain a second grayscale image.

[0138] The grayscale value of the second pixel is greater than the first grayscale threshold, the grayscale value of the third pixel is less than the second grayscale threshold, and the first grayscale threshold is greater than the second grayscale threshold.

[0139] It can be understood that the second pixel is a pixel that is too bright in the first grayscale image, and the third pixel is a pixel that is too dark in the first grayscale image. Restoring these two parts of pixels to 0 can control the noise in the overexposed and underexposed areas of the first RGB image and reduce the occurrence of artifacts.

[0140] S3123: Determine a noise adjustment weight map according to the grayscale value of the fourth pixel of the second grayscale image.

[0141] Exemplarily, S3121 may include: Step A, determining a preset noise adjustment gain according to the brightness value of the first RGB image, wherein the preset noise adjustment gain is less than 1. Step B, multiplying the preset noise adjustment gain by the gray value of the fourth pixel to obtain a noise adjustment weight.

[0142] It can be understood that the noise on a RAW image is generally much larger than that on an RGB image. The preset noise adjustment gain is less than 1, which can overall darken the noise image to be superimposed on the first RGB image. In other words, it can reduce the noise level of the first noise image, and can reduce the situation in which the noise is too large and the signal-to-noise ratio is too poor after the first RGB image is superimposed with the noise, thereby reducing the image quality, thereby further ensuring the image texture.

[0143] It can also be understood that the preset noise adjustment gain adjusts the noise size of the final second RGB image by adjusting the noise size of the first noise image. In the second grayscale image, the larger the grayscale value of the fourth pixel is, the higher the grayscale value of a pixel in the noise weight image is, which means that the pixel is brighter and the noise adjustment weight of the pixel is higher.

[0144] The following will continue to introduce relevant optional solutions for step A.

[0145] Optionally, for pixels in the first grayscale image whose grayscale values ​​are greater than or equal to the second grayscale threshold and whose grayscale values ​​are less than or equal to the first grayscale threshold (referred to as pixel #A for convenience of explanation), they can be further divided into pixels in the darker area, pixels in the brighter area, and pixels in the remaining area with reference to the above-mentioned method of defining the grayscale threshold.

[0146] It can be understood that from the user's perspective, the brighter areas of the first grayscale image (e.g. Figure 1A If the content presented by the bridge of the nose 101 in the image is too smooth and has no graininess, the three-dimensional sense of the area will be significantly weakened, the image realism will be reduced, and the image texture will be affected. Figure 1BThe brighter the pixel of the bridge of the nose 201), the higher the noise adjustment weight of the pixel, the more obvious the graininess will be, and the obvious graininess will make it more three-dimensional, more in line with the real face, and enhance the realism and texture of the image. For darker areas, since the brightness of the dark area itself is relatively low, if a relatively large noise is superimposed, the content in the darker area may have a heavier graininess, which may make the content in the darker area appear more messy, making the signal-to-noise ratio worse and the user experience worse. In the present application, the darker the pixel in the darker area, the lower the noise adjustment weight of the pixel, and the increase in graininess of the second RGB image compared to the first RGB image will not be too obvious. For example, Figure 1B The zygomatic bone 203 in the Figure 1A The zygomatic bone 103 in the image does not increase the graininess, which can reduce the deterioration of the signal-to-noise ratio in darker areas and ensure the user's viewing experience.

[0147] Optionally, the preset noise adjustment gain is determined according to the proportion of pixels in the darker area, pixels in the brighter area, and pixels in the remaining area in pixel #A.

[0148] For example, if the pixel ratio of the darker area is greater than the first ratio threshold, it means that the darker area accounts for a large proportion. To avoid the final second RGB image having a poor signal-to-noise ratio, the preset noise adjustment gain is smaller, such as the preset noise adjustment gain is smaller than the first gain threshold. For example, if the pixel ratio of the brighter area is greater than the first ratio threshold, it means that the brighter area accounts for a large proportion. To avoid the final second RGB image lacking stereoscopic sense and image realism, the preset noise adjustment gain is larger, such as the preset noise adjustment gain is greater than the second gain threshold, and the second gain threshold is greater than the first gain threshold. For example, if the pixel ratio of the remaining area is greater than the first ratio threshold, it means that the remaining area accounts for a large proportion. For example, the preset noise adjustment gain is less than or equal to the second gain threshold and greater than or equal to the first gain threshold.

[0149] Optionally, the preset noise adjustment gain is determined according to the average brightness value (or average grayscale value) of pixel #A.

[0150] For example, if the average grayscale value is greater than the third grayscale threshold, it means that the first grayscale image is brighter as a whole. To avoid the lack of stereoscopic effect and image realism of the final second RGB image, the preset noise adjustment gain is larger, such as the preset noise adjustment gain is greater than the third gain threshold. For example, if the average grayscale value is less than or equal to the third grayscale threshold, it means that the first grayscale image is darker as a whole. To avoid the poor signal-to-noise ratio of the final second RGB image, the preset noise adjustment gain is smaller, such as the preset noise adjustment gain is less than the fourth gain threshold, and the third gain threshold is greater than the fourth gain threshold.

[0151] Optionally, a preset noise adjustment gain is determined according to sensor parameters.

[0152] For example, the preset noise adjustment gain is determined according to the sensor parameters by a neural network model. The sensor parameters are input into the neural network model, and the output result of the model is the preset noise adjustment gain.

[0153] Optionally, the input data for training the neural network model is sample sensor parameters, and the target data is sample preset noise adjustment gain. The sample sensor parameters include sample ISO, sample Poisson noise parameters, and sample Gaussian noise parameters. It can be understood that Poisson noise parameters and Gaussian noise parameters vary with the model of the image sensor, and ISO varies with the shooting scene. The sample Poisson noise parameters and sample Gaussian noise parameters include Poisson noise parameters and Gaussian noise parameters of multiple models of image sensors; for each model of image sensor, the sample ISO includes ISO used in multiple shooting scenes. The sample preset noise adjustment gain corresponds to the sample sensor parameters.

[0154] Figure 7 This is a schematic diagram of an image processing method 300 ′ provided in an embodiment of the present application. The method 300 ′ can be understood as a specific implementation of the method 300 .

[0155] like Figure 7 As shown, on the one hand, referring to S3111, S3112 and S3113 in method 300, a second noise image is obtained according to the first RGB image and the sensor parameters. Specifically, a third noise image is obtained according to the first RGB image and the sensor parameters, a fourth noise image is obtained according to the third noise image, and a second noise image is obtained according to the fourth noise image.

[0156] On the other hand, referring to S3121, S3122 and S3123 in method 300, a noise adjustment weight map is obtained according to the first RGB image. Specifically, a first grayscale image is obtained according to the first RGB image, a second grayscale image is obtained according to the first grayscale image, and a noise adjustment weight map is obtained according to the second grayscale image.

[0157] Subsequently, referring to S313 in method 300 , a first noise image is obtained according to the second noise image and the noise adjustment weight map.

[0158] As an implementation of S313, Figure 7 As shown, the three fourth noise images included in the second noise image correspond to the three color channels of the first RGB image one by one, and the fifth pixel of each fourth noise image corresponds to the first pixel of each color channel one by one. Multiplying the second noise image with the noise adjustment weight map can be understood as multiplying each fifth pixel with the noise adjustment weight corresponding to each first pixel to obtain three fifth noise images. In other words, the first noise image includes three fifth noise images.

[0159] Finally, referring to S320 in method 300 , the first noise image is superimposed on the first RGB image to obtain a second RGB image.

[0160] As an implementation manner of S320, the fifth noise image is superimposed on the image of each channel of the three color channels in the first RGB image to obtain the second RGB image.

[0161] Corresponding to the methods given in the above-mentioned method embodiments, the embodiments of the present application also provide corresponding devices, such as a server or an electronic device, which includes a module for executing the corresponding methods in the above-mentioned method embodiments. The module can be software, hardware, or a combination of software and hardware. It can be understood that the technical features described in the above-mentioned method embodiments are also applicable to the following device embodiments. Therefore, the contents not described in detail can be referred to the above method embodiments, and for the sake of brevity, they will not be repeated here.

[0162] Figure 8 It is a schematic diagram of the architecture (including software system and some hardware) of the embodiment of the present application. Figure 8 As shown, the application architecture is divided into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the application architecture can be divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0163] like Figure 8 As shown, the application layer includes the camera and the gallery. It can be understood that Figure 8 The application layer includes some applications, and in fact, the application layer may also include other applications, which are not limited in this application. For example, the application layer also includes information, alarm clock, weather, stopwatch, compass, timer, flashlight, calendar, Alipay and other applications.

[0164] like Figure 8 As shown, the application framework layer includes a camera access interface. The camera access interface includes camera management and camera devices. The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. Among them, the camera hardware abstraction layer includes multiple camera devices. The camera algorithm library includes a granular noise algorithm module, which is used to execute the image processing method provided in this application, including generating a noise image according to RGB image sensor parameters, and superimposing the noise image on the RGB image. Optionally, the camera algorithm library may also include an automatic exposure algorithm for sending ISO to the sensor.

[0165] It should be understood that the decision module can also be placed in other layers. As a possible implementation, the decision module can be placed in the application layer or the application framework layer.

[0166] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. Figure 8 As shown, the driver layer includes camera device driver, digital signal processor driver and graphics processor driver, etc.

[0167] The hardware layer includes sensors, image signal processors, digital signal processors, and graphics processors. The sensors include multiple sensors, TOF cameras, and multi-spectral sensors.

[0168] For example, the user can click on the camera application. When the user clicks on the camera to take a photo, the photo taking instruction can be sent to the camera hardware abstraction layer through the camera access interface. The camera hardware abstraction layer calls the camera device driver and calls the camera algorithm library. The camera algorithm library sends the configured parameters (including the sensor image output mode and the parameter configuration of each ISP module) to the camera hardware abstraction layer. The camera hardware abstraction layer sends these configured parameters to the camera device driver. The camera device driver sends the configuration parameters sent by the camera hardware abstraction layer to the hardware layer, for example, sends the sensor image output mode to the sensor, and sends the parameter configuration of each ISP module to the image signal processor. The sensor outputs the image based on the sensor image output mode. The image signal processor performs corresponding processing based on the parameter configuration of each ISP module. The camera algorithm library is also used to send digital signals to the digital signal processor driver in the driver layer, so that the digital signal processor driver calls the digital signal processor in the hardware layer to process the digital signal. The digital signal processor can return the processed digital signal to the camera algorithm library through the digital signal processor driver. The camera algorithm library is also used to send digital signals to the graphics signal processor driver in the driver layer so that the graphics signal processor driver calls the graphics processor in the hardware layer to process digital signals. The graphics processor can return the processed graphics data to the camera algorithm library through the graphics processor driver.

[0169] In addition, the grainy noise algorithm in the camera algorithm library can obtain the ISO used by the sensor when taking the RAW image from the automatic exposure algorithm. The grainy noise algorithm also presets the sensor parameters of the sensor (see the description of the embodiment above). The RGB image output by the image signal processor can be sent to the camera device driver. The camera device driver can send the RGB image output by the image signal processor to the camera hardware abstraction layer. The camera hardware abstraction layer can send the RGB image to the post-processing algorithm module, and the post-processing algorithm module further processes the RGB image based on the grainy noise algorithm (see the description of the embodiment above for details). Figures 2 to 7The camera access interface can send the RGB image returned by the camera hardware abstraction layer to the camera.

[0170] The above describes in detail the software system used in the embodiment of the present application. Fig. 9 The hardware system of the electronic device 400 is described.

[0171] Fig. 9 The figure shows a hardware system of an electronic device 400 suitable for the present application.

[0172] The electronic device 400 may be a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any limitation on the specific type of the electronic device 400.

[0173] The electronic device 400 may include a processor 410, an external memory interface 420, an internal memory 424, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 1, an antenna 2, a mobile communication module 450, a wireless communication module 460, an audio module 470, a speaker 470A, a receiver 470B, a microphone 470C, an earphone interface 470D, a sensor module 480, a button 490, a motor 491, an indicator 492, a camera 493, a display screen 494, and a subscriber identification module (SIM) card interface 495, etc. The sensor module 480 may include a pressure sensor 480A, a gyroscope sensor 480B, an air pressure sensor 480C, a magnetic sensor 480D, an acceleration sensor 480E, a distance sensor 480F, a proximity light sensor 480G, a fingerprint sensor 480H, a temperature sensor 480J, a touch sensor 480K, an ambient light sensor 480L, a bone conduction sensor 480M, etc.

[0174] It should be noted that Fig. 9 The structure shown does not constitute a specific limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may include Fig. 9The electronic device 400 may include more or fewer components than those shown. Fig. 9 A combination of some of the components shown, or the electronic device 400 may include Fig. 9 Subcomponents of some of the components shown. For example, Fig. 9 The illustrated proximity light sensor 480G may be optional. Fig. 9 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0175] The processor 410 may include one or more processing units. For example, the processor 410 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an ISP, a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural network processor. Different processing units may be independent devices or integrated devices.

[0176] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0177] The processor 410 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that the processor 410 has just used or cyclically used. If the processor 410 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0178] In the present application, the processor 410 generates a noise image based on the parameters of the sensor used to determine the noise size of the RAW image, and superimposes the noise image on the RGB image obtained by image signal processing of the RAW image. This is equivalent to simulating the noise of the RAW image on the RGB image, thereby adding granular texture details to the RGB image, improving the image detail texture, and making the picture look better.

[0179] Fig. 9 The connection relationship between the modules shown is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the electronic device 400. Optionally, the modules of the electronic device 400 may also adopt a combination of multiple connection modes in the above embodiments.

[0180] The electronic device 400 can realize the display function through the GPU, the display screen 494 and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or change display information.

[0181] Display screen 494 may be used to display images or videos. In some embodiments, electronic device 400 may include 1 or N display screens 494, where N is a positive integer greater than 1.

[0182] The electronic device 400 can realize the shooting function through ISP, camera 493, video codec, GPU, display screen 494 and application processor.

[0183] The ISP is used to process the data fed back by the camera 493. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithmic optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be set in the camera 493.

[0184] The camera 493 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 400 may include 1 or N cameras 493, where N is a positive integer greater than 1.

[0185] The embodiment of the present application provides a chip system, which includes one or more processors for calling and running instructions stored in the memory from the memory, so that the method of the embodiment of the present application is executed. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0186] Among them, the chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0187] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0188] The computer program product can be stored in a memory, and is finally converted into an executable target file that can be executed by a processor after preprocessing, compiling, assembling and linking.

[0189] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0190] The computer-readable storage medium may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM).

[0191] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0192] In several embodiments provided in the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0193] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0194] It should be understood that the "plurality" mentioned in this application refers to two or more than two. The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0195] The terms (or numbers) "first", "second", ..., etc. that appear in the embodiments of the present application are only used for descriptive purposes, that is, only to distinguish different objects, such as different "coordinates", etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", ..., etc. may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one of the following (items)" or similar expressions thereof refers to any combination of these items, including any combination of a single (item) or a plurality of (items).

[0196] In short, the above is only a preferred embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the protection scope of this application.

Claims

1. An image processing method, applied to an electronic device, characterized in that: The method comprises: generating a first noise image according to a first RGB image and sensor parameters, wherein the first RGB image is obtained by performing image signal processing on a RAW image, the sensor parameters include a sensitivity used when an image sensor is used to capture the RAW image, and Poisson noise parameters and Gaussian noise parameters inherent to the image sensor, and the noise level of the RAW image is determined based on the sensor parameters; superimposing the first noise image on the first RGB image to obtain a second RGB image; The step of generating a first noise image according to the first RGB image and the sensor parameters includes: Generate Poisson noise intensity according to the pixel value of the first RGB image, the sensitivity and the Poisson noise parameter; Generate Gaussian noise intensity according to the sensitivity and the Gaussian noise parameter; Sampling the Poisson noise intensity to obtain a Poisson noise value, sampling the Gaussian noise intensity to obtain a Gaussian noise value, and obtaining a third noise image according to the Poisson noise value and the Gaussian noise value; Downsampling and stacking the third noise image to obtain a second noise image; Determine a noise adjustment weight corresponding to the first RGB image, where the noise adjustment weight is used to adjust the noise level of the second noise image; The first noise image is obtained according to the second noise image and the noise adjustment weight.

2. The method according to claim 1, characterized in that The determining the noise adjustment weight corresponding to the first RGB image includes: determining the noise adjustment weight corresponding to the first pixel according to a brightness value of the first pixel of the first RGB image; The obtaining the first noise image according to the second noise image and the noise adjustment weight includes: multiplying the second noise image by the noise adjustment weight to obtain the first noise image.

3. The method according to claim 2, characterized in that The downsampling and stacking processing is performed on the third noise image to obtain a second noise image, including: According to the first RGB image, the sensitivity, the Poisson noise parameter, and the Gaussian noise parameter, Gaussian noise and Poisson noise corresponding to the first RGB image are sampled to obtain a third noise image, wherein the third noise image includes three sub-noise images corresponding to three color channels of the first RGB image respectively; Downsampling the three sub-noise images to obtain a fourth noise image, wherein the fourth noise image is a single-channel image, and pixels of the fourth noise image are arranged in a Bayer format; The three fourth noise images are stacked to obtain the second noise image, wherein the three fourth noise images correspond to the three color channels respectively.

4. The method according to claim 3, characterized in that Generating the Poisson noise intensity according to the pixel values ​​of the first RGB image, the sensitivity and the Poisson noise parameter comprises: determining the Poisson noise intensity of the three color channels according to the pixel values ​​of the three color channels of the first RGB image, the sensitivity and the Poisson noise parameter; Generating the Gaussian noise intensity according to the sensitivity and the Gaussian noise parameter includes: determining the Gaussian noise intensity of the three color channels according to the sensitivity and the Gaussian noise parameter; The sampling of the Poisson noise intensity to obtain a Poisson noise value, the sampling of the Gaussian noise intensity to obtain a Gaussian noise value, and obtaining a third noise image according to the Poisson noise value and the Gaussian noise value include: based on the Poisson noise intensity and the Gaussian noise intensity of the three color channels, for each of the three color channels, randomly selecting corresponding Poisson noise values ​​and Gaussian noise values ​​pixel by pixel to obtain the third noise image.

5. The method according to claim 2, characterized in that The determining the noise adjustment weight of the first pixel according to the brightness value of the first pixel of the first RGB image includes: Generate a corresponding first grayscale image according to the first RGB image, wherein the brightness value of the first RGB image corresponds to the grayscale value of the first grayscale image pixel by pixel; Setting the grayscale values ​​of the second pixel and the third pixel in the first grayscale image to 0 to obtain a second grayscale image, wherein the grayscale value of the second pixel is greater than a first grayscale threshold, the grayscale value of the third pixel is less than a second grayscale threshold, and the first grayscale threshold is greater than the second grayscale threshold; The noise adjustment weight is determined according to the grayscale value of a fourth pixel of the second grayscale image.

6. The method according to claim 5, characterized in that The determining the noise adjustment weight according to the grayscale value of the fourth pixel of the second grayscale image includes: Determine a preset noise adjustment gain according to the brightness value of the first RGB image, wherein the preset noise adjustment gain is less than 1; The preset noise adjustment gain is multiplied by the grayscale value of the fourth pixel to obtain the noise adjustment weight.

7. The method according to claim 3 or 4, characterized in that The step of multiplying the second noise image by the noise adjustment weight to obtain the first noise image includes: The fifth pixels of each of the fourth noise images correspond one-to-one to the first pixels of each color channel of the three color channels, and each of the fifth pixels is multiplied by the noise adjustment weight corresponding to each of the first pixels to obtain three fifth noise images, wherein the first noise image includes three fifth noise images.

8. The method according to claim 7, characterized in that The step of superimposing the first noise image on the first RGB image to obtain a second RGB image includes: The fifth noise image is superimposed on the image of each channel of the three color channels in the first RGB image to obtain the second RGB image.

9. The method according to any one of claims 1 to 6, characterized in that The method further comprises: In response to a user's photo-taking operation, the first RGB image, the RAW image, and the sensor parameters are acquired.

10. An electronic device comprising an image sensor, a memory, and one or more processors, characterized in that: The memory is used to store a computer program; the processor is used to call the computer program so that the electronic device executes the method according to any one of claims 1 to 9.

11. A computer storage medium, characterized in that: include: Computer instructions; when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

Images