Image processing methods, apparatus, electronic devices and storage media

Abstract

This application discloses an image processing method, apparatus, electronic device, and storage medium. The method includes: generating a sparse index map based on the blurring intensity corresponding to each background pixel in a background region of an image to be processed; the sparse index map is used to indicate target pixels selected from the background pixels, wherein the number of target pixels indicated by the sparse index map is less than the number of background pixels; and performing point diffusion processing on each target pixel included in the background region based on the sparse index map to obtain a blurred image. Implementing this application embodiment can reduce the computational load of background blurring and improve real-time performance.

Description

Technical Field

[0001] This application relates to the field of imaging technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Some smartphones can use software algorithms to simulate the effect of a camera aperture when shooting portraits, achieving a sharp subject and blurred background. The key lies in the blurring of the background area. Currently, mainstream background blurring algorithms suffer from poor clarity and detail in the blurred areas on low-resolution images. While background blurring on high-resolution images can improve the clarity and detail of the blurred areas to some extent, it significantly increases the computational load, resulting in poor real-time performance. Summary of the Invention

[0003] This application discloses an image processing method, apparatus, electronic device, and storage medium, which can reduce the computational load of background blurring and improve real-time performance.

[0004] This application discloses an image processing method, the method comprising: generating a sparse index map based on the blur intensity corresponding to each background pixel in the background region of the image to be processed; the sparse index map being used to indicate target pixels selected from the background pixels, wherein the number of target pixels indicated by the sparse index map is less than the number of background pixels; and performing point diffusion processing on each target pixel included in the background region based on the sparse index map to obtain a blurred image.

[0005] This application discloses an image processing apparatus, comprising:

[0006] A generation module is used to generate a sparse index map based on the blur intensity corresponding to each background pixel in the background region of the image to be processed; the sparse index map is used to indicate the target pixels selected from the background pixels, and the number of target pixels indicated by the sparse index map is less than the number of background pixels; a diffusion module is used to perform point diffusion processing on each target pixel included in the background region based on the sparse index map to obtain a blurred image.

[0007] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor implements any of the image processing methods disclosed in this application.

[0008] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the image processing methods disclosed in this application.

[0009] Compared with related technologies, the embodiments of this application have the following beneficial effects:

[0010] Electronic devices can generate sparse index maps based on the blur intensity corresponding to background pixels in the background region. This reduces the number of target pixels that need to participate in blur processing, reduces the computational load of background blur, improves the real-time performance of background blur, and lowers the difficulty of implementing real-time preview of blurred images in electronic devices with limited computing power, such as smartphones.

[0011] Simultaneously, the reduction in the number of target pixels required for blurring allows electronic devices to directly perform dot diffusion processing on high-resolution images. This leverages the richer image information contained in high-resolution images to improve the sharpness and clarity of the light spots in the blurred background area, enhancing the depth of the light spots. Furthermore, using dot diffusion processing for target pixel blurring calculations can reduce color leakage, making the blurred image closer to the effect of optical imaging, resulting in a more realistic and natural appearance. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the method flow of a depth-of-field rendering method disclosed in one embodiment;

[0014] Figure 2 This is a schematic flowchart of an image processing method disclosed in one embodiment;

[0015] Figure 3 This is an example diagram of a background region and its corresponding sparse index map disclosed in one embodiment;

[0016] Figure 4 This is a schematic flowchart of another image processing method disclosed in one embodiment;

[0017] Figure 5 This is an example diagram of superimposing the point diffusion results of two target pixels in one embodiment;

[0018] Figure 6 This is a schematic flowchart of another image processing method disclosed in one embodiment;

[0019] Figure 7This is a schematic diagram of the structure of an image processing apparatus disclosed in one embodiment;

[0020] Figure 8 This is a schematic diagram of the structure of an electronic device disclosed in one embodiment. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0023] This application discloses an image processing method, apparatus, electronic device, and storage medium, which can reduce the computational load of background blurring and improve real-time performance. These will be described in detail below.

[0024] To more clearly illustrate the image processing method disclosed in the embodiments of this application, the following section first introduces a depth-of-field rendering method disclosed in the embodiments of this application. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of a depth-of-field rendering method disclosed in one embodiment. This method can be applied to any electronic device, such as smartphones or tablets, and is not specifically limited thereto. Figure 1 As shown:

[0025] The electronic device can perform foreground segmentation and depth estimation on the image to be processed 110, and obtain foreground segmentation result 120 and depth estimation result 130.

[0026] The image to be processed 110 can be image data captured by the imaging device (such as a camera module) of an electronic device, or image data stored in the image memory of the electronic device, and there is no specific limitation.

[0027] The foreground segmentation result 120 can be used to indicate the foreground and background regions of the image 110 to be processed. The electronic device can perform target object (such as a human portrait) recognition on the image 110 to be processed to obtain the foreground segmentation result. Taking portrait photography as an example, the foreground segmentation result 120 can be a binary mask, the foreground region can refer to the region where the human portrait is located, and the background region can refer to other image regions other than the human portrait region.

[0028] The depth estimation result 130 can be used to indicate the depth value corresponding to each pixel in the image to be processed. The electronic device can calculate the depth value corresponding to each pixel in the image to be processed by methods such as monocular depth estimation, binocular depth estimation, or depth estimation based on depth sensor measurement, without any specific limitation.

[0029] In some embodiments, the electronic device may also generate a foreground segmentation result 120 based on the depth estimation result 130. For example, pixels with depth values ​​greater than a depth threshold indicated by the depth estimation result 130 may be identified as background pixels in the background region; or pixels with depth values ​​less than or equal to the depth threshold indicated by the depth estimation result 130 may be identified as foreground pixels in the foreground region. The depth threshold can be set according to actual business needs and is not specifically limited.

[0030] The electronic device can generate a blur intensity map 140 based on the foreground segmentation result and depth estimation result 120. The blur intensity map 140 can include the blur intensity corresponding to each pixel in the image to be processed. Optionally, the first blur intensity corresponding to each foreground pixel in the foreground region of the image to be processed can be positively correlated with the depth value corresponding to the foreground pixel, and the second blur intensity corresponding to each background pixel in the background region of the image to be processed can be positively correlated with the depth value corresponding to the background pixel, and the first blur intensity is always less than the second blur intensity. The first blur intensity can be selected to have a smaller value to avoid the foreground being too blurred.

[0031] In other words, the blurriness of the background area after blurring is greater than that of the foreground area; and the further away the background pixels are from the imaging device and the greater their depth value, the more blurred they become after blurring; similarly, the foreground pixels with greater depth values ​​become more blurred after blurring.

[0032] The electronic device can further combine the image information in the image to be processed 110 with the blur intensity map 140 to perform background blur rendering on the background area of ​​the image to be processed, and obtain background blur result 150; and combine the image information in the image to be processed 110 with the blur intensity map 140 to perform depth of field (DOF) rendering on the foreground area of ​​the image to be processed, and obtain DOF rendering result 160.

[0033] The electronic device can combine the background blur result 150 and the DOF rendering result 160 to obtain a blurred image 170. In the blurred image 170, a shooting effect with a clear foreground and a blurred background is presented.

[0034] When the electronic device performs background blurring on the background area of ​​the image 110 to be processed, if the resolution of the image 110 to be processed is large, such as the original image data collected by the imaging device, or the compressed image obtained after downsampling the original image data by one-half or more, the background blurring rendering is prone to problems such as large computational load and poor real-time performance.

[0035] The image processing method disclosed in this application can reduce the amount of computation required for background blurring and improve real-time performance.

[0036] Please see Figure 2 , Figure 2 This is a schematic diagram of the process flow of an image processing method disclosed in one embodiment. Figure 2 The method shown can be one implementation of the aforementioned depth-of-field rendering method for blurring the background area. For example... Figure 2 As shown, the following steps may be included:

[0037] 210. Generate a sparse index map based on the blur intensity of each background pixel in the background region of the image to be processed.

[0038] The electronic device can determine the background region of the image to be processed and the blurring intensity corresponding to each background pixel in the background region according to any of the embodiments disclosed in the foregoing embodiments. The following will not be repeated.

[0039] In some embodiments, the image to be processed can be a single-frame image captured by the imaging device, or a fused image obtained by the ISP processor after fusing multiple frames captured by the imaging device; the specific method is not limited. For example, the image to be processed can be obtained by fusing two images with different exposure values.

[0040] A sparse index map is used to indicate the target pixels selected from each background pixel for bokeh calculation. The number of target pixels indicated by the sparse index map is less than the number of background pixels in the background region. The sparse index map corresponds to the image to be processed. That is, the sparse index map is sparse; some background pixels in the background region participate in the bokeh calculation, while others do not. Furthermore, the selection of target pixels for bokeh calculation from the background pixels is related to the bokeh intensity corresponding to the background pixel.

[0041] In one embodiment, the number of target pixels indicated by the sparse index map can be negatively correlated with the blur intensity corresponding to each background pixel in the background region of the image to be processed. That is, the greater the blur intensity corresponding to the background pixel, the sparser the sparse index map. The electronic device can select the target pixel based on the blur intensity corresponding to any one background pixel in the background region, or it can select the target pixel based on the average blur intensity corresponding to multiple background pixels in the background region; the specific method is not limited.

[0042] In typical application scenarios, the background area of ​​the image to be processed is usually quite complex. For example, in portrait photography with a complex background, the background area of ​​the image to be processed may include a background panel close to the imaging device, or it may include a dense forest far away from the imaging device. Therefore, the blur intensity corresponding to each background pixel in the background area may vary considerably.

[0043] In one embodiment, the background region of the image to be processed can be divided into multiple blurred sub-regions according to the blur intensity corresponding to each background pixel. The difference between the blur intensities of the background pixels in each blurred sub-region is less than a first difference threshold. That is, the blur intensity corresponding to the background pixels in each blurred sub-region is similar, while the blur intensity corresponding to the background pixels in different blurred sub-regions varies significantly. The number of background pixels included in each blurred sub-region can be the same or different, and there is no specific limitation.

[0044] For each blurred sub-region included in the background region, the number of first target pixels corresponding to the blurred sub-region indicated by the sparse index map can be negatively correlated with the blur intensity corresponding to the background pixels of the blurred sub-region in the image to be processed.

[0045] For example, the background region may include a first blurred sub-region and a second blurred sub-region. Background pixels in the first blurred sub-region correspond to a first blurred intensity, and background pixels in the second blurred sub-region correspond to a second blurred intensity; the first blurred intensity is less than the second blurred intensity. The first target pixel corresponding to the first blurred sub-region indicated by the sparse index map is a first quantity, and the first target pixel corresponding to the second blurred sub-region is a second quantity; assuming that the first blurred sub-region and the second blurred sub-region include the same number of background pixels, then the first quantity is greater than the second quantity.

[0046] It can be seen that for a local blurred sub-region, the greater the blur intensity corresponding to the background pixels in the blurred sub-region, the sparser the blurred sub-region indicated by the sparse index map, and the fewer the number of target pixels in the blurred sub-region.

[0047] Accordingly, for any blurred sub-region, the electronic device can select the first target pixel based on the blur intensity corresponding to any background pixel included in the blurred sub-region, or it can select the first target pixel based on the average blur intensity corresponding to multiple background pixels included in the blurred sub-region, without any specific limitation.

[0048] Optionally, the electronic device can select a first target pixel from the background pixels included in any blurred sub-region based on an interval-based selection method. The blurring intensity corresponding to the background pixels included in the blurred sub-region can be positively correlated with the selected first interval. That is, the greater the blurring intensity, the larger the selected first interval, and the sparser the blurred sub-region indicated by the sparse index map.

[0049] The first interval can refer to the number of background pixels between two adjacent first target pixels, such as a interval of one pixel, two pixels, or four pixels, but is not limited to these.

[0050] Alternatively, the first interval can also refer to the length of the gap between two adjacent first target pixels. It is understood that image resolution can be used to indicate how many pixels are included per inch of image, and the length and width of the background region of the image to be processed can be determined based on the number of background pixels and the resolution of the image to be processed. Therefore, electronic devices also select first target pixels from multiple background pixels at length intervals.

[0051] For example, suppose the background region of the image to be processed is divided into region A and region B. The blur intensity corresponding to each background pixel in region A can be within a first range; the blur intensity corresponding to each background pixel in region B can be within a second range; the upper limit of the first range can be less than the lower limit of the second range, that is, the value of the first range is less than the value of the second range.

[0052] The sparse index map can indicate that the first target pixel in region A can be selected from the background pixels included in region A every two pixels; and the first target pixel in region B can be selected from the background pixels included in region B every four pixels.

[0053] 220. Based on the sparse index map, perform point diffusion processing on each target pixel in the background region to obtain a blurred image.

[0054] Electronic devices can perform point diffusion processing on each target pixel in the background region according to the indication of the sparse index map. Point diffusion processing is a method of blurring computation that simulates realistic lens blur using a point spread function (PSF), diffusing the original pixel value of the target pixel in the image to be processed according to the diffusion range indicated by the diffusion rendering kernel. The size of the diffusion range can be represented by the number of pixels within the diffusion range.

[0055] Understandably, after point diffusion processing of each target pixel, a corresponding point diffusion result can be obtained. The point diffusion result for each target pixel can include the diffusion range corresponding to the target pixel and the diffusion pixel values ​​of each background pixel within the diffusion range. Specifically, the diffusion pixel values ​​of each background pixel within the diffusion range corresponding to the target pixel are calculated based on the original pixel value of the target pixel in the image to be processed. That is, the pixel value of the target pixel in the image to be processed diffuses to a diffusion range centered on the target pixel.

[0056] After performing point diffusion processing on each target pixel, the electronic device can obtain a background blurring result corresponding to the background region. Optionally, the electronic device can merge the background blurring result with a clear, unblurred foreground region to obtain a blurred image; or, the electronic device can merge the background blurring result with the DOF rendering result to obtain a blurred image where the foreground is also slightly blurred, without any specific limitation.

[0057] It should be noted that although only some target pixels in the background region of the image being processed participate in the blurring calculation, the result of point diffusion processing can affect several background pixels surrounding the target pixels, causing these background pixels to also be blurred. Therefore, reducing the number of pixels participating in the blurring calculation does not affect the blurring effect of the background region after blurring, and point diffusion processing can also reduce color leakage caused by blurring. The blurred effect after blurring is closer to the effect of optical imaging, and the blurred image is more realistic and natural.

[0058] In some embodiments, some pixels in the background area may not be blurred. However, in high-resolution images, the number of unblurred pixels is small and difficult to detect with the naked eye, thus having little impact on the overall blurring effect of the background area. Furthermore, the problem of some pixels not being blurred can be avoided by reasonably designing the selection interval of the target pixels.

[0059] As can be seen, in the aforementioned embodiments, the electronic device can generate a sparse index map based on the blur intensity corresponding to the background pixels in the background region, thereby reducing the number of target pixels that need to participate in the blurring process, reducing the computational load of background blurring, improving the real-time performance of background blurring, and reducing the difficulty of implementing real-time preview of blurred images in electronic devices with limited computing power, such as smartphones.

[0060] Simultaneously, the reduction in the number of target pixels required for blurring allows electronic devices to directly perform dot diffusion processing on high-resolution images. This leverages the richer image information contained in high-resolution images to improve the sharpness and clarity of the light spots in the blurred background area, enhancing the depth of the light spots. Furthermore, performing blurring calculations on target pixels through dot diffusion reduces computational load while maintaining the blurred effect in the background area, and also reduces color leakage, more closely resembling the effect of optical imaging, resulting in a more realistic and natural blurred image.

[0061] In one embodiment, when performing step 210, in order to reduce the computational load of background blur rendering while improving the edge sharpness, clarity, and layering of the light spot rendering, the electronic device may further generate a sparse index map based on the blur intensity and image features corresponding to each background pixel in the background region. The image features may include brightness and / or texture sharpness.

[0062] In other words, the target pixel indicated by the sparse index map may include a first target pixel selected based on the blur intensity and any one of the following two types of pixels: a second target pixel selected based on brightness and a third target pixel selected based on texture sharpness.

[0063] Electronic devices can convert image data of an image to be processed into YUV data format for representation, where Y / U / V represent three color channels. The Y component of a background pixel can be used to indicate the brightness of that background pixel. For example, the brightness of the background pixel corresponding to a light spot in the background area can be higher than the brightness of other background pixels.

[0064] Electronic devices can extract texture features from images to be processed using methods such as gray-level co-occurrence matrix calculation, synchronous autoregressive models, syntactic texture analysis, and local Fourier transform to determine the texture sharpness. For example, the texture sharpness of background pixels corresponding to the edge of a light spot in the background region can be higher than that of background pixels corresponding to the center of the light spot.

[0065] Optionally, when the image features include brightness, the background region can be divided into at least two brightness sub-regions based on the brightness of each background pixel. The brightness difference between the background pixels in each brightness sub-region is less than a second difference threshold. That is, the brightness of the background pixels in each brightness sub-region is similar, while the brightness difference between background pixels in different brightness sub-regions is significant. The brightness sub-regions divided based on brightness may be different from or the same as the blur sub-regions divided based on blur intensity; no specific limitation is imposed.

[0066] For each luminance sub-region within the background region, the number of second target pixels corresponding to that luminance sub-region, as indicated by the sparse index map, is positively correlated with the luminance of the background pixels included in that luminance sub-region. That is, the closer a region in the background region is to the light source, the more it needs to participate in the blurring calculation.

[0067] For example, the background region may include a first brightness sub-region and a second brightness sub-region. Background pixels in the first brightness sub-region correspond to a first brightness, and background pixels in the second brightness sub-region correspond to a second brightness, where the first brightness is less than the second brightness. The second target pixel count corresponding to the first brightness sub-region, as indicated by the sparse index map, is a third number, and the second target pixel count corresponding to the second brightness sub-region, as indicated by the sparse index map, is a fourth number. Assuming the first and second brightness sub-regions contain the same number of background pixels, then the third number is less than the fourth number.

[0068] It is understandable that, for any luminance sub-region, the electronic device can select the second target pixel based on the luminance corresponding to any background pixel included in the luminance sub-region, or it can select the second target pixel based on the average luminance corresponding to multiple background pixels included in the luminance sub-region; there is no specific limitation.

[0069] Optionally, the electronic device can select a second target pixel from the background pixels included in any brightness sub-region based on an interval-based selection method. The brightness of the background pixels included in the brightness sub-region may be negatively correlated with the selected second interval. That is, the higher the brightness, the smaller the selected second interval, and the denser the brightness sub-region indicated by the sparse index map. Furthermore, the second interval selected based on brightness may be the same as or different from the first interval selected based on blur intensity; the specific setting can be determined according to business requirements, and this embodiment does not impose limitations.

[0070] Optionally, when the image features include texture sharpness, the background region can be divided into at least two texture sub-regions based on the texture sharpness of each background pixel. The difference in texture sharpness among the pixels in each texture sub-region is less than a third difference threshold. That is, the texture sharpness of background pixels in each texture sub-region is similar, while the texture sharpness of background pixels in different texture sub-regions varies significantly. The texture sub-regions divided based on texture sharpness may be different from or the same as the blur sub-regions divided based on blur intensity; no specific limitation is imposed.

[0071] For each texture sub-region included in the background region, the number of third target pixels corresponding to that texture sub-region, as indicated by the sparse index map, is positively correlated with the texture sharpness of the background pixels included in that texture sub-region. That is, the sharper the texture in the background region, the more likely it is to participate in the blurring calculation.

[0072] For example, the background region may include a first texture sub-region and a second texture sub-region. Background pixels in the first texture sub-region correspond to a first texture sharpness level, and background pixels in the second brightness sub-region correspond to a second texture sharpness level. The first texture sharpness level is lower than the second texture sharpness level. The third target pixel indicated by the sparse index map corresponding to the first texture sub-region is the fifth number, and the third target pixel indicated by the sparse index map corresponding to the second texture sub-region is the sixth number. Assuming the first and second texture sub-regions include the same number of background pixels, then the fifth number is less than the sixth number.

[0073] It is understandable that, for any texture sub-region, the electronic device may select the third target pixel based on the texture sharpness corresponding to any background pixel included in the texture sub-region, or it may select the third target pixel based on the average texture sharpness corresponding to multiple background pixels included in the texture sub-region, without any specific limitation.

[0074] Optionally, the electronic device can select a third target pixel from the background pixels included in any texture sub-region based on an interval-based selection method. The texture sharpness of the background pixels included in the texture sub-region may be negatively correlated with the selected third interval. That is, the higher the brightness, the smaller the selected third interval, and the denser the brightness sub-region indicated by the sparse index map. Furthermore, the third interval selected based on texture sharpness may be the same as or different from the first interval selected based on blur intensity; the aforementioned third interval may be the same as or different from the second interval selected based on brightness, and can be set according to business needs; this embodiment does not limit the specific settings.

[0075] It should be noted that the sparse index map generated by the electronic device can be a set of first target pixels corresponding to each blurred sub-region, second target pixels corresponding to each brightness sub-region, and / or third target pixels corresponding to each texture sub-region.

[0076] For example, please refer to Figure 3 , Figure 3 This is an example diagram of a background region and its corresponding sparse index map disclosed in one embodiment. For example... Figure 3 As shown, the background region 30 may include a light spot 30a and a dark background 30b other than the light spot 30a. The sparse index map corresponding to the background region 30 may be as shown in the sparse index map 37.

[0077] Based on the blur intensity corresponding to the background pixels, the background region 30 can be divided into two blur sub-regions: blur sub-region 31a and blur sub-region 31b. The blur intensity corresponding to the background pixels in blur sub-region 31a can be less than the blur intensity corresponding to the background pixels in blur sub-region 31b.

[0078] Correspondingly, the distribution of the first target pixel 341 in the sparse index map 37, selected based on the blurring intensity, can be shown in the distribution map 34. In the distribution map 34, the first target pixel 341 corresponding to the blurred sub-region 31a is relatively dense, while the first target pixel 341 corresponding to the blurred sub-region 31b is relatively sparse.

[0079] Based on the brightness of the background pixels, the background region 30 can be divided into two brightness sub-regions: brightness sub-region 32a and brightness sub-region 32b. The brightness of the background pixels in brightness sub-region 32a can be greater than the brightness of the background pixels in brightness sub-region 32b.

[0080] Correspondingly, the distribution of the second target pixel 351 selected based on brightness in the sparse index map 37 can be shown in the distribution map 35. In the distribution map 35, the second target pixel 351 corresponding to the brightness sub-region 32a is relatively dense, while the second target pixel 351 corresponding to the brightness sub-region 32b is relatively sparse.

[0081] Based on the texture sharpness of the background pixels, the background area can be divided into three texture sub-regions: texture sub-region 33a, texture sub-region 33b, and texture sub-region 33c. Texture sub-region 33a can be the central area of ​​the light spot 30a, texture sub-region 33b can be the boundary area between the edge of the light spot 30a and the dark background 30b, and texture sub-region 33b contains some background pixels from the light spot 30a and some background pixels from the dark background 30b. Among them, the texture sharpness of the background pixels in texture sub-region 33b is the highest, the texture sharpness of the background pixels in texture sub-region 33a is the second highest, and the texture sharpness of the background pixels in texture sub-region 33c is the lowest.

[0082] Correspondingly, the distribution of the third target pixel 361 selected based on texture sharpness in the sparse index map 37 is shown in the distribution map 36. The third target pixel 361 corresponding to the texture sub-region 33b is the most dense, the third target pixel 361 corresponding to the texture sub-region 33a is relatively sparse, and the target pixel 361 corresponding to the texture sub-region 33c is the sparsest.

[0083] In the aforementioned embodiments, the electronic device can select target pixels for blurring calculation from the background region based on the blurring intensity corresponding to each background pixel. This reduces the computational load required for blurring, improving the real-time performance of background blurring and allowing application to high-resolution images. By leveraging the information density of high-resolution images, the blurring effect of light spots can be improved. Furthermore, areas with higher light source and texture sharpness in the background region participate more in the blurring calculation, further optimizing the blurring effect of light spots and making the background blurring more realistic and natural.

[0084] Please see Figure 4 , Figure 4 This is a schematic flowchart of another image processing method disclosed in one embodiment, which can be applied to any of the aforementioned electronic devices. For example... Figure 4 As shown, the method may include the following steps:

[0085] 410. Generate a sparse index map based on the blur intensity of each background pixel in the background region of the image to be processed.

[0086] The implementation method for generating a sparse index map corresponding to the image to be processed by the electronic device can be found in the foregoing embodiments, and will not be repeated below.

[0087] 420. For each target pixel in the background region indicated by the sparse index map, obtain the diffusion rendering kernel corresponding to the target pixel and the diffusion weight corresponding to the target pixel based on the blur intensity of the target pixel.

[0088] In this embodiment, a rendering kernel table can be pre-set. This table can include diffusion rendering kernels corresponding to different blur intensities, and different diffusion rendering kernels can be used to indicate different diffusion ranges. Optionally, the blur intensity and the radius of the diffusion rendering kernel can be positively correlated. That is, the greater the blur intensity, the larger the diffusion range indicated by the diffusion rendering kernel, and the more pixels the diffusion range covers.

[0089] For each target pixel, the electronic device can look up the corresponding diffusion rendering kernel from the rendering kernel table based on the blur intensity of the target pixel. Different blur intensities correspond to different diffusion rendering kernel sizes.

[0090] In addition, each target pixel can also correspond to a diffusion weight. Different target pixels may have the same or different diffusion weights. The diffusion weight corresponding to each target pixel can be calculated based on the original pixel value of that target pixel in the image to be processed.

[0091] For example, the diffusion weight may include luminance weight and chrominance weight, and the original pixel values ​​of each background pixel in the background region of the image to be processed may include luminance component values ​​and chrominance component values. The luminance component values ​​may be Y channel component values, and the chrominance component values ​​may correspond to U / V channel component values.

[0092] Electronic devices can calculate the luminance weight of each background pixel based on its luminance component value. The luminance component value of a background pixel is positively correlated with its corresponding luminance weight. That is, the brighter the background pixel in the image to be processed, the higher its diffusion weight will be during point diffusion processing.

[0093] Electronic devices can calculate the chromaticity weight of each background pixel based on its chromaticity component values. The chromaticity component values ​​and chromaticity weights of background pixels are positively correlated. That is, background pixels with higher chromaticity in the image to be processed will have higher diffusion weights during point diffusion processing.

[0094] Optionally, when calculating the diffusion weight corresponding to each background pixel, the electronic device may further determine the diffusion weight corresponding to the background pixel based on the pixel value of the background pixel in the image to be processed and the exposure value corresponding to the image to be processed.

[0095] The exposure value corresponding to the image to be processed can be represented by the sensitivity (International Standardization Organization, ISO) parameter of the imaging device when the image to be processed is captured.

[0096] In some embodiments, the image to be processed can be a fused image obtained by fusing multiple frames. The exposure value corresponding to the image to be processed can be the photosensitivity parameter of the imaging device when any frame of the image before fusion is captured. For example, the image to be processed can be obtained by fusing a first image and a second image, where the first image corresponds to a first exposure value, the second image corresponds to a second exposure value, and the second exposure value is less than the first exposure value. The exposure value corresponding to the image to be processed can be either the first exposure value or the second exposure value.

[0097] The electronic device can map the pixel values ​​of background pixels based on a preset weight mapping curve, and determine the mapped values ​​as the weight values ​​corresponding to the background pixels. The weight mapping curve can be related to the exposure value of the image to be processed; the smaller the exposure value of the image, the greater the stretching of the weight mapping curve.

[0098] 430. Based on the diffusion rendering kernel and the corresponding diffusion weight corresponding to the target pixel, perform point diffusion processing on the target pixel to obtain the point diffusion result of the target pixel.

[0099] The point diffusion result of the target pixel can include the point diffusion result corresponding to the pixel value and the point diffusion result corresponding to the number of diffusions. Specifically, multiplying the original pixel value of the target pixel in the image to be processed by the diffusion weight corresponding to the target pixel, and then multiplying by the diffusion rendering kernel, yields the point diffusion result corresponding to the pixel value; multiplying the diffusion weight corresponding to the target pixel by the diffusion rendering kernel yields the point diffusion result corresponding to the number of diffusions.

[0100] 440. Overlay the point diffusion results of each target pixel indicated by the sparse index map to obtain a blurred image based on the overlay results.

[0101] When an electronic device overlays the point diffusion results of each target pixel, it can generate a pixel value overlay map and a frequency statistics map corresponding to the image to be processed, respectively. The pixel value overlay map records the point diffusion results corresponding to the pixel values, while the frequency statistics map records the point diffusion results corresponding to the number of diffusion cycles.

[0102] The electronic device superimposes the point diffusion results of each target pixel, which can refer to recording the point diffusion results of each target pixel in a pixel superposition map and a number diffusion map.

[0103] For example, the diffusion ranges corresponding to adjacent target pixels may overlap. When the diffusion results of adjacent target pixels are superimposed, the diffusion pixel values ​​of the background pixels in the overlapping part are also superimposed accordingly. It should be noted that, due to the sparsity of the sparse index map, there may be several non-target pixel background pixels between two adjacent target pixels.

[0104] Please see Figure 5 , Figure 5 This is an example diagram illustrating the superposition of point diffusion results of two target pixels, as disclosed in one embodiment. Figure 5 As shown, the diffusion range corresponding to target pixel 510 can include 9 background pixels, and the diffusion pixel values ​​of the 9 background pixels can be A1-A9 respectively. The diffusion range corresponding to target pixel 520 can also include 9 second pixels, and the diffusion pixel values ​​of the 9 background pixels can be B1-B9 respectively. Target pixel 510 and target pixel 520 can be two adjacent target pixels. In the image to be processed, target pixel 510 and target pixel 520 are separated by a non-target pixel background pixel.

[0105] In the diffusion range corresponding to target pixel 510, background pixels with diffusion pixel values ​​of A9, A2, and A3 overlap with background pixels in the diffusion range corresponding to target pixel 520, with diffusion pixel values ​​of B7, B6, and B5. Therefore, after superimposing the point diffusion results of target pixel 510 and target pixel 520, the pixel values ​​corresponding to the overlapping background pixels are A9+B7, A2+B6, and A3+B5, respectively.

[0106] The implementation method of recording the point diffusion results of each target pixel and the number of diffusions in the diffusion map of the electronic device can refer to the implementation method of recording the point diffusion results of each target pixel and the pixel value in the pixel value superposition map. The following content will not be repeated.

[0107] Therefore, the superposition result obtained by the electronic device may include: a pixel value superposition map that records the point diffusion results corresponding to each target pixel and pixel value; and a number statistics map that records the point diffusion results corresponding to each target pixel and number of diffusion times.

[0108] For each background pixel in the background region, the electronic device can divide the superimposed pixel value of the background pixel in the pixel value superimposed image by the number of diffusions of the pixel in the number of diffusions in the number of diffusions image to obtain the third pixel value of the background pixel in the blurred image.

[0109] In some embodiments, the overlay result corresponding to the point diffusion process can indicate that each pixel in the background region is blurred. In this case, the aforementioned process of obtaining a blurred image based on the overlay result can include: directly using the overlay result as the background blurring result of the background region to generate a blurred image.

[0110] In other embodiments, the overlay result corresponding to the dot diffusion processing can indicate that some pixels in the background region are not blurred. Therefore, obtaining the blurred image based on the overlay result may include:

[0111] Identify any missing pixels in the background region that have not been blurred, and recalculate the blurring by performing point diffusion or Gaussian blurring on these missing pixels to obtain the blurring result for the missing pixels; combine the aforementioned superposition result and the blurring result of the missing pixels to obtain the background blurring result for the background region, and generate a blurred image.

[0112] In the aforementioned embodiments, the electronic device can perform point diffusion processing under the guidance of a sparse index map, using the diffusion weights corresponding to each target pixel indicated by the sparse index map. With the overlapping of the point diffusion results of each target pixel, brighter spots in the image to be processed form bright light spots in the blurred image, while darker spots in the image to be processed form hazy diffuse circles in the blurred image. This reduces computational load, improves the real-time performance of background blurring, and makes the background blurring effect more realistic and natural.

[0113] In one embodiment, the aforementioned image to be processed can be obtained by fusing two images with different exposure values. See also... Figure 6 , Figure 6 This is a schematic flowchart of another image processing method disclosed in one embodiment, which can be applied to any of the aforementioned electronic devices. For example... Figure 6 As shown, the method may include the following steps:

[0114] 610. By fusing the first image and the second image, the image to be processed can be obtained.

[0115] The first exposure value corresponding to the first image can be greater than the second exposure value corresponding to the second image. For example, the first image can be a normally exposed image, and the second image can be a dark frame captured using a shutter speed setting with a lower exposure value.

[0116] Optionally, the electronic device can identify a first pixel in the first image whose brightness exceeds a brightness threshold, and identify a second pixel in the second image that matches the first pixel. The brightness threshold can be set according to actual business needs and is not specifically limited. The electronic device can identify the second pixel in the second image, excluding the one matching the first pixel, using algorithms such as Scale-invariant Feature Transform (SIFT) matching and Speeded-Up Robust Features (SURF) matching, without further limitation.

[0117] After identifying the matching second pixel, the electronic device can replace the first pixel value of the first pixel in the first image with the second pixel value of the matching second pixel in the second image, thereby fusing the first and second images to obtain the image to be processed. In other words, the image information in the first image with a higher exposure value can be restored using the second image with a lower exposure value, making the colors of the light source points in the fused image to be processed more realistic, which is beneficial for improving the bokeh effect in the blurred image.

[0118] Optionally, the brightness threshold can be set to 255. In scenes with excessively bright light sources, the first image with a higher exposure value may contain pixels truncated by light source saturation. For example, among the pixels with a brightness component value of 255 in the first image, some may be pixels truncated by light source saturation. Therefore, fusing two images with different exposure values ​​is beneficial for recovering the color information of pixels truncated by light source saturation.

[0119] Alternatively, before identifying a second pixel in the second image that matches the first pixel, the electronic device may also register the first and second images to align the first and second images, thereby improving the accuracy of pixel matching.

[0120] The implementation methods for steps 620-650 described below can be found in the aforementioned embodiments, and will not be repeated hereafter.

[0121] 620. Generate a sparse index map based on the blur intensity of each background pixel in the background region of the image to be processed, and the image features of each background pixel in the image to be processed.

[0122] 630. For each target pixel in the background region indicated by the sparse index map, obtain the diffusion rendering kernel that matches the blur intensity corresponding to the target pixel, and obtain the diffusion weight corresponding to the target pixel.

[0123] 640. Based on the diffusion rendering kernel and the corresponding diffusion weight corresponding to the target pixel, perform point diffusion processing on the target pixel to obtain the point diffusion result of the target pixel.

[0124] 650. The point diffusion results of each target pixel indicated by the sparse index map are superimposed to obtain a blurred image based on the superimposed results.

[0125] In one embodiment, the diffusion rendering kernel obtained in step 630 can be a differential diffusion rendering kernel. Therefore, in step 650, after obtaining the superposition result, the electronic device further integrates the superposition result and compresses it to a preset size, such as 8 bits, to obtain the background blurring result.

[0126] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing apparatus disclosed in one embodiment. This image processing apparatus can be applied to any of the aforementioned electronic devices. Figure 7 As shown, the image processing apparatus 700 may include a generation module 710 and a diffusion module 720.

[0127] The generation module 710 is used to generate a sparse index map based on the blur intensity corresponding to each background pixel in the background region of the image to be processed; the sparse index map is used to indicate the target pixel selected from each background pixel, and the number of target pixels indicated by the sparse index map is less than the number of background pixels.

[0128] The diffusion module 720 is used to perform point diffusion processing on each target pixel in the background region according to the sparse index map to obtain a blurred image.

[0129] In one embodiment, the background region is divided into at least two fuzzy sub-regions according to the fuzzing intensity corresponding to each background pixel, and the target pixel includes a first target pixel selected according to the fuzzing intensity; for each fuzzy sub-region, the number of first target pixels corresponding to the fuzzy sub-region indicated by the sparse index map is negatively correlated with the fuzzing intensity corresponding to the background pixels included in the fuzzy sub-region.

[0130] In one embodiment, the generation module 710 can also be used to select target pixels corresponding to the blurred sub-region from the background pixels included in the blurred sub-region at intervals; the blurring intensity corresponding to the background pixels included in the blurred sub-region is positively correlated with the selected interval.

[0131] In one embodiment, the generation module 710 can also be used to generate a sparse index map based on the blur intensity corresponding to each background pixel in the background region of the image to be processed, and the image features of each background pixel in the image to be processed.

[0132] In one embodiment, the image features include: brightness; the background region is divided into at least two brightness sub-regions according to the brightness of each background pixel, and the target pixel includes a second target pixel selected according to the brightness; for each brightness sub-region, the number of second target pixels corresponding to the brightness sub-region indicated by the sparse index map is positively correlated with the brightness of the background pixels included in the brightness sub-region.

[0133] In one embodiment, the image features include: texture sharpness; the background region is divided into at least two texture sub-regions according to the texture sharpness of each background pixel, and the target pixel includes a third target pixel selected according to the texture sharpness; for each texture sub-region, the number of third target pixels corresponding to the texture sub-region indicated by the sparse index map is positively correlated with the texture sharpness of the background pixels included in the texture sub-region.

[0134] In one embodiment, the diffusion module 720 may include: an acquisition unit, a diffusion unit, and a superposition unit.

[0135] The acquisition unit can be used to acquire the diffusion rendering kernel corresponding to each target pixel in the background region indicated by the sparse index map, based on the blur intensity corresponding to the target pixel, and to acquire the diffusion weight corresponding to the target pixel; the radius of the diffusion rendering kernel is positively correlated with the blur intensity corresponding to the target pixel;

[0136] The diffusion unit can be used to perform point diffusion processing on the target pixel based on the diffusion rendering kernel and diffusion weight to obtain the point diffusion result of the target pixel.

[0137] The overlay unit can be used to overlay the point diffusion results of each target pixel indicated by the sparse index map to obtain a blurred image based on the overlay result.

[0138] In one embodiment, the acquisition unit can also be used to determine the diffusion weight corresponding to the target pixel based on the exposure value corresponding to the image to be processed and the pixel value of the target pixel in the image to be processed.

[0139] In one embodiment, the image processing apparatus 700 may further include a fusion module.

[0140] The fusion module can be used to fuse the first image and the second image to obtain the image to be processed. The first image corresponds to the first exposure value, and the second image corresponds to the second exposure value. The second exposure value is less than the first exposure value.

[0141] In one embodiment, the fusion module can also be used to identify a first pixel in the first image whose brightness is higher than a brightness threshold, and to identify a second pixel in the second image that matches the first pixel; and to replace the first pixel value of the first pixel in the first image with the second pixel value of the matching second pixel in the second image to obtain the image to be processed.

[0142] As can be seen, by implementing the aforementioned image processing device, a sparse index map can be generated based on the blur intensity corresponding to the background pixels in the background region. This reduces the number of target pixels that need to participate in the blurring process, reduces the computational load of background blurring, improves the real-time performance of background blurring, and reduces the difficulty of implementing real-time preview of blurred images in electronic devices with limited computing power, such as smartphones.

[0143] Simultaneously, the reduction in the number of target pixels required for blurring allows the image processing device to directly perform point diffusion processing on the high-resolution image to be processed. This leverages the richer image information contained in high-resolution images to improve the sharpness and clarity of the light spots in the blurred background area, enhancing the layering of the light spots. Furthermore, using point diffusion processing for target pixel blurring calculations can reduce color leakage, more closely resembling the effect of optical imaging, resulting in a more realistic and natural blurred image.

[0144] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of another electronic device disclosed in one embodiment.

[0145] like Figure 8 As shown, the electronic device may include:

[0146] Memory 810 storing executable program code;

[0147] Processor 820 coupled to memory 810;

[0148] The processor 820 calls the executable program code stored in the memory 810 to execute any of the image processing methods disclosed in the embodiments of this application.

[0149] It should be noted that, Figure 8 The electronic device shown may also include components not shown, such as a power supply, input buttons, camera, speaker, screen, RF circuit, Wi-Fi module, Bluetooth module, and sensors, which will not be described in detail in this embodiment.

[0150] This application discloses a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements any of the image processing methods disclosed in this application.

[0151] This application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform any of the image processing methods disclosed in this application.

[0152] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0153] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0154] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of this application.

[0157] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0158] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized by, The method comprises: generating a sparse index map according to the blur intensity corresponding to each background pixel point in the background region of the image to be processed; the sparse index map is used to indicate target pixel points selected from the background pixel points, and the number of target pixel points indicated by the sparse index map is less than the number of background pixel points; performing point spread processing on each target pixel point included in the background region according to the sparse index map to obtain a blurred image; wherein the performing point spread processing on each target pixel point included in the background region according to the sparse index map to obtain a blurred image comprises: for each target pixel point indicated by the sparse index map, obtaining a diffusion rendering kernel corresponding to the target pixel point and a diffusion weight corresponding to the target pixel point according to the blur intensity corresponding to the target pixel point; the radius of the diffusion rendering kernel is positively correlated with the blur intensity corresponding to the target pixel point; performing point spread processing on the target pixel point according to the diffusion rendering kernel and the diffusion weight to obtain a point spread result of the target pixel point; and superimposing the point spread results of each target pixel point indicated by the sparse index map to obtain a blurred image according to the superimposed result.

2. The method of claim 1, wherein, The background region is divided into a plurality of blur sub-regions according to the blur intensity corresponding to each background pixel point, and the target pixel points include first target pixel points selected according to the blur intensity; the number of first target pixel points corresponding to the blur sub-region is negatively correlated with the blur intensity corresponding to the background pixel points included in the blur sub-region.

3. The method of claim 2, wherein, The first target pixel points corresponding to the blur sub-region are selected at intervals from the background pixel points included in the blur sub-region; the blur intensity corresponding to the background pixel points included in the blur sub-region is positively correlated with the interval.

4. The method of claim 1, wherein, The generating a sparse index map according to the blur intensity corresponding to each background pixel point in the background region of the image to be processed comprises: generating a sparse index map according to the blur intensity corresponding to each background pixel point in the background region of the image to be processed and the image features of the background pixel points in the image to be processed.

5. The method of claim 4, wherein, The image features include brightness; the background region is divided into a plurality of brightness sub-regions according to the brightness of each background pixel point, and the target pixel points include second target pixel points selected according to the brightness; the number of second target pixel points corresponding to the brightness sub-region is positively correlated with the brightness of the background pixel points included in the brightness sub-region; and / or, the image features include texture sharpness; the background region is divided into a plurality of texture sub-regions according to the texture sharpness of each background pixel point; the target pixel points include third target pixel points selected according to the texture sharpness; the number of third target pixel points corresponding to the texture sub-region is positively correlated with the texture sharpness of the background pixel points included in the texture sub-region.

6. The method of claim 1, wherein, The obtaining a diffusion weight corresponding to the target pixel point comprises: The diffusion weight corresponding to the target pixel point is determined according to the exposure value corresponding to the image to be processed and the pixel value of the target pixel point in the image to be processed.

7. The method according to any one of claims 1 to 6, characterized in that, Before the sparse index map is generated according to the blurring intensity corresponding to each background pixel point in the background region of the image to be processed, the method further comprises: The first image and the second image are fused to obtain the image to be processed, the first image corresponding to a first exposure value, and the second image corresponding to a second exposure value, the second exposure value being less than the first exposure value.

8. The method of claim 7, wherein, The first image and the second image are fused to obtain the image to be processed, comprising: identifying a first pixel point in the first image whose luminance is higher than a luminance threshold, and identifying a second pixel point in the second image that matches the first pixel point; replacing a first pixel value of the first pixel point in the first image with a second pixel value of the matching second pixel point in the second image to obtain the image to be processed.

9. An image processing apparatus characterized by comprising: comprising: generating a sparse index map according to the blurring intensity corresponding to each background pixel point in the background region of the image to be processed; the sparse index map is used to indicate a target pixel point selected from the background pixel points, and the number of target pixel points indicated by the sparse index map is less than the number of background pixel points; a diffusion module is configured to perform point diffusion processing on each target pixel point included in the background region according to the sparse index map to obtain a blurred image, and is further configured to, for each target pixel point indicated by the sparse index map, obtain a diffusion kernel corresponding to the target pixel point according to the blurring intensity corresponding to the target pixel point, and obtain a diffusion weight corresponding to the target pixel point; the radius of the diffusion kernel is in a positive correlation with the blurring intensity corresponding to the target pixel point; performing point diffusion processing on the target pixel point according to the diffusion kernel and the diffusion weight to obtain a point diffusion result of the target pixel point; superimposing the point diffusion results of each target pixel point indicated by the sparse index map to obtain a blurred image according to a superimposed result.

10. An electronic device, comprising: The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

Images