Image processing method and device, electronic equipment, and computer readable storage medium

Abstract

Embodiments of the present application relate to the technical field of image processing, and disclose an image processing method and device, electronic equipment and computer readable storage medium. The method comprises: performing portrait beautification processing on a portrait region of a first image to be processed by a first thread to obtain a second image; performing blurring processing on a background region of the first image by a second thread to obtain a third image, the background region being a region of the first image other than the portrait region, the first thread and the second thread being two threads running in parallel; and generating a target image corresponding to the first image according to the second image and the third image. The method disclosed in the embodiments of the present application can shorten the time length of image processing operation of the electronic equipment, thereby improving the efficiency of image processing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of electronic device shooting technology, electronic devices can now perform various image processing operations on captured images to improve image quality.

[0003] In practice, it has been found that electronic devices in related technologies require a large number of image processing operations, which leads to long processing times and reduces the efficiency of image processing. Summary of the Invention

[0004] This application discloses an image processing method and apparatus, an electronic device, and a computer-readable storage medium, which can shorten the time for electronic devices to perform image processing operations, thereby improving the efficiency of image processing.

[0005] The first aspect of this application discloses an image processing method, including:

[0006] The first thread performs portrait enhancement processing on the portrait area of ​​the first image to be processed to obtain the second image;

[0007] The background area of ​​the first image is blurred by the second thread to obtain the third image. The background area is the area in the first image other than the portrait area. The first thread and the second thread are two threads that run in parallel.

[0008] Generate a target image corresponding to the first image based on the second image and the third image.

[0009] A second aspect of this application discloses an image processing apparatus, comprising:

[0010] The first processing unit is used to perform portrait beautification processing on the portrait area of ​​the first image to be processed through the first thread to obtain the second image;

[0011] The second processing unit is used to blur the background area of ​​the first image through a second thread to obtain a third image. The background area is the area in the first image other than the portrait area. The first thread and the second thread are two threads that run in parallel.

[0012] The generation unit is used to generate a target image corresponding to the first image based on the second image and the third image.

[0013] A third aspect of this application discloses an electronic device, comprising:

[0014] Memory containing executable program code;

[0015] A processor coupled to the memory;

[0016] The processor calls the executable program code stored in the memory to execute the image processing method disclosed in the first aspect of the embodiments of this application.

[0017] The fourth aspect of this application discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the image processing method disclosed in the first aspect of this application.

[0018] Compared with related technologies, the embodiments of this application have the following beneficial effects:

[0019] The image processing method disclosed in this application decouples the previously single-threaded portrait beautification and background blurring operations. Instead, it uses a first thread and a second thread, executed in parallel, to perform portrait beautification in the portrait area and background blurring in the background area, respectively. Finally, the portrait beautification and background blurring results are fused to obtain an optimized image. It is understood that since the portrait area and background area are two independent regions, decoupling them for processing does not affect the image processing effect. Furthermore, splitting the single thread into two threads can shorten the time for image processing operations on electronic devices, thereby improving image processing efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of an image processing method disclosed in an embodiment of this application;

[0022] Figure 1A This is a schematic diagram disclosed in an embodiment of this application for illustrating the human portrait area;

[0023] Figure 2 This is a schematic flowchart of another image processing method disclosed in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of a portrait segmentation map and a depth map disclosed in an embodiment of this application;

[0025] Figure 4 This is a schematic flowchart of another image processing method disclosed in the embodiments of this application;

[0026] Figure 5 This is a flowchart illustration of an image processing method disclosed in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of an image processing apparatus disclosed in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be noted that the terms "first," "second," "third," and "fourth," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0031] This application discloses an image processing method and apparatus, an electronic device, and a computer-readable storage medium, which can shorten the time for electronic devices to perform image processing operations, thereby improving the efficiency of image processing.

[0032] The technical solution of this application will be described in detail below with reference to specific embodiments.

[0033] To more clearly illustrate the image processing method, apparatus, electronic device, and computer-readable storage medium disclosed in the embodiments of this application, a scenario diagram applicable to the image processing method is first introduced. Optionally, the image processing method can be applied to various electronic devices, including but not limited to portable electronic devices such as mobile phones and tablets, wearable devices such as smartwatches and smart bracelets, and desktop electronic devices such as desktop computers and televisions, without limitation herein.

[0034] With the continuous development of electronic device imaging technology, electronic devices can now perform a series of image processing steps after capturing images to optimize image quality. Specifically, the typical image processing workflow includes: using an Image Signal Processor (ISP) to perform image processing operations such as black level dead pixel correction, lens distortion correction, and scaling on the image to be processed; then performing portrait enhancement on the foreground area of ​​the processed image; and finally, blurring the background of the enhanced image to obtain an optimized image.

[0035] In related technologies, the aforementioned series of image processing operations are typically executed using a single thread, meaning that each image processing step is performed sequentially. In practice, it has been found that as the number of image processing steps increases, the time required for electronic devices to process images becomes increasingly longer, thereby reducing image processing efficiency.

[0036] To address this, this application discloses an image processing method that decouples the previously single-threaded portrait beautification and background blurring operations. Instead, it uses a first thread and a second thread, executed in parallel, to perform portrait beautification in the portrait area and background blurring in the background area, respectively. Finally, the beautification and blurring results are fused to obtain an optimized image. It is understood that since the portrait area and background area are two independent regions, decoupling them for processing does not affect the image processing effect. Furthermore, splitting the single-threaded operation into a dual-threaded operation can shorten the time required for image processing by the electronic device, thereby improving image processing efficiency.

[0037] Please see Figure 1 , Figure 1 This is a schematic flowchart of an image processing method disclosed in an embodiment of this application. The method can be applied to the aforementioned electronic device or other execution entity, and is not limited thereto. The method may include the following steps:

[0038] 102. Perform portrait enhancement processing on the portrait area of ​​the first image to be processed through the first thread to obtain the second image.

[0039] In this embodiment, the first image may include, but is not limited to, an initial image captured by a camera device of an electronic device, an initial image received from another electronic device, or an initial image downloaded from the Internet. The initial image refers to an image that has not undergone image processing. In another embodiment, the first image may be a processed image after processing by an image processor (ISP). Optionally, the processed image may be an image in YUV or RGB format.

[0040] It should be noted that a thread is a single sequential control flow for executing tasks in an electronic device; one thread can be used to execute a corresponding task. Optionally, in this embodiment, the electronic device can create multiple threads simultaneously to execute different image processing operations in parallel, thereby shortening the image processing time. The aforementioned first thread can be any one of the multiple threads.

[0041] Optionally, the electronic device can determine the human figure region of the first image to be processed through a first thread. See also Figure 1A , Figure 1A This is a schematic diagram illustrating a portrait area disclosed in an embodiment of this application. The first image 200 may include a portrait area 210 and a background area 220. It is understood that the portrait area 210 is typically closer to the shooting source, so in some embodiments, the portrait area 210 is the foreground area. The background area 220 in the first image 200, excluding the portrait area 210, is farther from the shooting source, so the background area 220 is the background area. It should be noted that this embodiment uses a portrait area and a background area as examples for illustration and should not be construed as limiting the scope of this application.

[0042] Furthermore, after determining the portrait area of ​​the first image, the electronic device can perform portrait beautification processing on the portrait area through a first thread to obtain a beautified second image. The portrait beautification processing may include image processing operations such as beautification, skin smoothing, and face slimming, which are not limited here.

[0043] 104. The background area of ​​the first image is blurred by the second thread to obtain the third image. The background area is the area in the first image excluding the human figure area. The first thread and the second thread are two threads that run in parallel.

[0044] In this embodiment, while the electronic device performs portrait beautification processing on the portrait area through the first thread, it can also perform background blur processing on the background area of ​​the first image through the second thread to obtain a third image with a blurred background. The blurring processing may include Gaussian blur, dot diffusion, and other processing operations, which are not limited here.

[0045] Optionally, the first time point for starting the first thread and the second time point for starting the second thread can be the same; in another embodiment, the difference between the first time point for starting the first thread and the second time point for starting the second thread can be less than a difference threshold (e.g., 1 second, 2 seconds, which is not limited here). That is, the first thread and the second thread can start simultaneously or not simultaneously, but the interval between their starts is very short, allowing the first thread and the second thread to have a parallel execution phase.

[0046] 106. Generate the target image corresponding to the first image based on the second and third images.

[0047] In this embodiment of the application, after the electronic device obtains the second image through the first thread and the third image through the second thread, it can perform image fusion processing on the second image and the third image to obtain a target image with a clear foreground and a blurred background.

[0048] Image fusion processing refers to the technique of extracting feature information from multiple different images and generating a composite image based on the extracted feature information according to certain weights. Image fusion processing can include algorithms such as alpha blend and alphatest, and is not limited here.

[0049] For example, an electronic device can fuse the portrait region of a second image and the background region of a third image according to certain weights to obtain a target image. The portrait region in the target image can be the portrait region in the second image that has undergone portrait enhancement processing, and the background region can be the background region in the third image that has undergone background blur processing.

[0050] By implementing the methods disclosed in the above embodiments, the portrait beautification and background blurring operations, which were originally executed in a single thread, can be decoupled. Instead, a first thread and a second thread execute the portrait beautification and background blurring operations in parallel, respectively. Finally, the portrait beautification results and the background blurring results are fused to obtain an optimized image. It is understood that since the portrait region and the background region are two independent regions, decoupling them for processing separately does not affect the image processing effect. Furthermore, splitting the single thread into two threads can shorten the time for image processing operations on the electronic device, thereby improving the efficiency of image processing.

[0051] Please see Figure 2 , Figure 2 This is a schematic flowchart of another image processing method disclosed in an embodiment of this application. This method can be applied to the aforementioned electronic device or other execution entity, and is not limited thereto. The method may include the following steps:

[0052] 202. The first thread performs edge optimization processing on the human image region in the human image segmentation map corresponding to the first image to be processed, so as to obtain an optimized human image segmentation map. The human image segmentation map is used to describe the image position of the human image region in the first image.

[0053] In this embodiment of the application, before the electronic device performs edge optimization processing on the portrait region in the portrait segmentation map corresponding to the first image to be processed through the first thread, it can obtain the portrait segmentation map corresponding to the first image (see [link]). Figure 3 , Figure 3This is a schematic diagram of a portrait segmentation map and a depth map disclosed in an embodiment of this application. The portrait segmentation map 300 corresponding to the first image 200 is used to describe the image location of the portrait region 3001 in the first image. Optionally, the electronic device can input the first image into a portrait segmentation model to obtain the portrait segmentation map corresponding to the first image through the portrait segmentation model.

[0054] The portrait segmentation model can be a model trained on a large number of sample images containing portraits, used to generate a portrait segmentation map corresponding to the image. Optionally, the portrait segmentation model can include a MonoDepth model.

[0055] Understandably, while a portrait segmentation model can quickly obtain a portrait segmentation image corresponding to the first image, the portrait regions obtained from this model are not entirely accurate. Optionally, the electronic device can perform edge optimization processing on the portrait regions in the portrait segmentation image through a first thread to obtain an optimized portrait segmentation image where the edges of the portrait regions more closely match the portrait contours. Edge optimization processing can include fast guided filtering, box filtering, etc., and is not limited here.

[0056] By implementing the above method, edge optimization processing can be performed on the portrait region in the portrait segmentation map corresponding to the first image, so that the portrait region determined by the subsequent portrait segmentation map is more accurate. This can avoid the subsequent portrait beautification operation being applied to the background region, thus destroying the image details of the background region.

[0057] 204. The first thread determines the portrait region of the first image based on the optimized portrait segmentation map, and performs portrait beautification processing on the portrait region.

[0058] In one embodiment, the electronic device can determine a portrait mask based on an optimized portrait segmentation map using a first thread. This portrait mask is used to determine the portrait region in the first image. The electronic device can then determine the portrait region of the first image based on the portrait mask using the first thread and perform portrait enhancement processing on the portrait region.

[0059] 206. The background area of ​​the first image is blurred by the second thread to obtain the third image. The background area is the area in the first image other than the human figure area. The first thread and the second thread are two threads that run in parallel.

[0060] Optionally, the electronic device can also create multiple other threads to perform image processing operations in parallel with the first and second threads. Optionally, the electronic device can use a third thread to perform de-jitter processing on the first image to obtain a fourth image; then, the electronic device generates a target image corresponding to the first image based on the second, third, and fourth images.

[0061] In another embodiment, the electronic device may further perform HDR high dynamic range rendering on the first image through a fourth thread to obtain a fifth image; then the electronic device generates a target image corresponding to the first image based on the second image, the third image, the fourth image and the fifth image.

[0062] By implementing the above method, electronic devices can create more threads to execute different image processing operations in parallel, thereby further shortening the image processing time and improving image processing efficiency.

[0063] 208. Generate the target image corresponding to the first image based on the second and third images.

[0064] It is understandable that, assuming the first and second threads are started simultaneously, their image processing times may differ due to the different workloads they perform. Optionally, the electronic device can generate the target image corresponding to the first image based on the second and third images only after the first thread has completed portrait enhancement and the second thread has completed blurring.

[0065] In one embodiment, if either the first thread or the second thread (target thread) completes the image processing operation, but the other non-target thread does not complete the image processing operation, the electronic device can wait for the non-target thread to complete the image processing operation, and generate the target image corresponding to the first image based on the second image and the third image when the non-target thread completes the image processing operation; wherein, when the target thread is the first thread, the non-target thread is the second thread, and when the target thread is the second thread, the non-target thread is the first thread.

[0066] By implementing the above method, electronic devices can set a waiting program to execute subsequent steps only after both threads have completed image processing. This avoids the electronic device using an image that has not completed image processing to generate the target image corresponding to the first image, thereby reducing the image quality of the target image.

[0067] As an optional implementation, the electronic device can acquire a weight map corresponding to the first image, and then perform image fusion on the second image and the third image based on the weight map to obtain a target image corresponding to the first image. The weight map describes the weights corresponding to the foreground region and the background region of the first image.

[0068] Combination Figure 1AFor example, if the portrait region 210 is located in the lower-middle area of ​​the first image 200, then the weight of the second image corresponding to the lower-middle area of ​​the first image 200 is 100%. Therefore, in subsequent fusion operations, the portrait region of the second image can be used to construct the lower-middle area of ​​the target image. Correspondingly, the other areas of the first image 200 are the background region 220, so the weight of the third image corresponding to these other areas is 100%. Therefore, in subsequent fusion operations, the background region of the third image can be used to construct the other areas of the target image besides the lower-middle area.

[0069] By implementing the above method, the image-processed regions in the second and third images can be merged into a target image with a clear foreground and a blurred background through image fusion, thereby improving the image quality while shortening the image processing time.

[0070] In another embodiment, a weight map is used to describe the weights corresponding to the foreground region, the background region, and the transition region of the first image. The transition region is the area where the foreground and background regions meet. To make the image processing effect transition more smoothly from the foreground region to the background region, the weight of the transition region can be 50% for the second image and 50% for the third image, or 40% for the second image and 60% for the third image, etc. The electronic device can then generate the target image based on the second and third images according to the method described above, which will not be elaborated further here.

[0071] In another embodiment, after obtaining the second image through the first thread and the third image through the second thread, the electronic device can perform hair edge optimization on the second and third images to make the hair contours in the second and third images more accurate. Then, the electronic device can generate a target image corresponding to the first image based on the second and third images with optimized hair edges.

[0072] By implementing the methods disclosed in the above embodiments, a single thread can be decomposed into two threads to shorten the time for image processing operations by the electronic device, thereby improving the efficiency of image processing; and, edge optimization processing can be performed on the portrait region in the portrait segmentation map corresponding to the first image to make the portrait region determined by the subsequent portrait segmentation map more accurate, thereby avoiding the subsequent portrait beautification operation from being applied to the background region and destroying the image details of the background region; and, by setting a waiting program to execute subsequent steps only when both threads have completed image processing, the electronic device can avoid using an image that has not completed image processing to generate the target image corresponding to the first image, thereby reducing the image quality of the target image; and, by using an image fusion method to merge the image-processed regions in the second and third images into a target image with a clear foreground and a blurred background, the image quality can be improved while shortening the image processing time.

[0073] Please see Figure 4 , Figure 4 This is a flowchart illustrating another image processing method disclosed in an embodiment of this application. This method can be applied to the aforementioned electronic device or other execution entity, and is not limited thereto. The method may include the following steps:

[0074] 402. Determine the blur intensity map corresponding to the first image to be processed. The blur intensity map is used to characterize the blur degree corresponding to each first pixel point included in the first image.

[0075] In this embodiment, before the electronic device performs blurring processing on the background area of ​​the first image through the second thread, it can first determine the blurring degree corresponding to each first pixel in the first image, and then generate a blurring intensity map based on the blurring degree corresponding to each first pixel in the first image. Optionally, the blurring degree corresponding to each first pixel in the first image can be manually calibrated or generated by the electronic device, and is not limited here.

[0076] In one embodiment, before determining the blur intensity map corresponding to the first image to be processed, the electronic device can acquire a portrait segmentation map and a depth map corresponding to the first image. The depth map is used to describe the depth information of each first pixel included in the first image. As mentioned above, the portrait segmentation map can be obtained through a portrait segmentation model, while the depth map can be obtained by inputting the first image into a depth prediction model. The depth prediction model can be a model trained on a large number of sample images to predict the depth information of the image. Optionally, the depth prediction model can include a DeepLab model. The electronic device can then generate the blur intensity map based on the portrait segmentation map and depth map corresponding to the first image.

[0077] Optionally, the electronic device can determine the first average depth of the portrait region and the second average depth of the background region of the first image based on the portrait segmentation map and depth map corresponding to the first image to be processed. Specifically, the electronic device can determine the maximum and minimum depths of the portrait region, and then determine the first average depth of the portrait region of the first image based on the maximum and minimum depths of the portrait region; similarly, the electronic device can determine the maximum and minimum depths of the background region, and then determine the second average depth of the background region of the first image based on the maximum and minimum depths of the background region.

[0078] The electronic device can then generate a relative depth map based on the first average depth of the portrait region and the second average depth of the background region of the first image, and determine the blur intensity map corresponding to the first image based on the relative depth map; wherein, the depth value of each pixel in the portrait region of the relative depth image is the first average depth, and the depth value of each pixel in the background region of the relative depth image is the second average depth.

[0079] By implementing the above method, the electronic device can determine the relative depth map corresponding to the first image based on the portrait segmentation map and depth map corresponding to the first image, which will facilitate the subsequent determination of the blurring intensity of each pixel in the first image based on the relative depth map.

[0080] Optionally, the electronic device can generate a blur intensity map corresponding to the first image based on the relative depth map corresponding to the first image and the following formula:

[0081]

[0082] Where LUT[x] represents the blur level of the Xth pixel in the first image, radius represents the initial blur level, curve represents the curve smoothness, and x represents the depth of the Xth pixel in the first image in the relative depth map; where X is a positive integer.

[0083] Please refer to it again. Figure 3 In the depth map 310, different shades of gray are typically used to represent different depths of field. The depth map 310 uses darker gray areas to represent the portrait region 210 and lighter gray areas to represent the background region 220. This should not be construed as limiting the scope of the embodiments in this application. It is understood that depth maps obtained through depth prediction models or other methods may not be accurate. Optionally, the electronic device can perform edge optimization on the edges of the portrait region in the depth map based on the portrait segmentation map to obtain an optimized depth map, and then determine the relative depth map of the first image based on the optimized depth map.

[0084] Optionally, the electronic device can determine the human image region in the depth map based on the human image segmentation map, and identify pixels in the human image region of the depth map whose depth values ​​are greater than a first depth threshold (which can be approximately equal to the maximum depth of the human image region) that need to be corrected, and revise the depth value corresponding to the pixel to be corrected to the average of the depth values ​​of its two adjacent pixels.

[0085] It is understandable that pixels with depth values ​​greater than the first depth threshold may be due to inaccurate predictions by the depth prediction model. Implementing the above method can correct the depth values ​​of pixels with abnormal depth values ​​back to normal depths, thereby making the depth information in the depth map more accurate.

[0086] Similarly, electronic devices can determine the pixels in the background region of the depth map whose depth values ​​are less than the second depth threshold based on the portrait segmentation map, and correct the pixels in the background region whose depth values ​​are less than the second depth threshold to obtain an optimized depth map.

[0087] In another embodiment, after generating a relative depth map based on the first average depth of the portrait region and the second average depth of the background region of the first image, the electronic device can determine the face region in the depth map and take the depth value with the most corresponding pixels in the face region as the target depth value; then, the pixels included in the portrait region of the relative depth image are adjusted from the first average depth to the target depth value, and the depth values ​​of the pixels included in the background region of the relative depth image are adjusted according to the difference between the first average depth and the target depth value to obtain an optimized relative depth map.

[0088] For example, suppose the first average depth of each pixel in the portrait region of the relative depth image is 10, and the second average depth of each pixel in the background region of the relative depth image is 20. If the depth of the pixel with the most pixels in the face region is determined to be 6, the electronic device can adjust the first average depth of the relative depth image from 10 to 6 and the second average depth to 20-4=16 to determine the optimized relative depth image.

[0089] By implementing the above method, the depth value with the most corresponding pixels in the face region can be used as the average depth of the portrait region. This can reduce the degree of blurring of each pixel in the portrait region in the subsequent blurring intensity map determined by the optimized relative depth map. As a result, the degree of blurring of the portrait region in the target image after blurring can be lower, so as to ensure that the portrait is clear enough.

[0090] In one embodiment, the electronic device can generate a depth histogram based on the individual pixels in the face region and the corresponding depth values ​​of each pixel. The horizontal axis of the depth histogram can represent the depth value, and the vertical axis can represent the number of pixels corresponding to each depth value. The electronic device can then determine the depth value with the most corresponding pixels in the face region as the target depth value based on this depth histogram.

[0091] As an optional implementation, before determining the bokeh intensity map corresponding to the first image to be processed, the electronic device may perform image preprocessing on the first image, and then determine the bokeh intensity map corresponding to the first image based on the preprocessed first image. Optionally, image preprocessing may include initialization, size adjustment, loading preset parameters, etc., which are not limited here.

[0092] By implementing the above method, the first image can be preprocessed to correct it, thereby making the determined blur intensity map more accurate and improving the subsequent blurring effect.

[0093] 404. Perform portrait enhancement processing on the portrait area of ​​the first image to be processed through the first thread to obtain the second image.

[0094] 406. The background area of ​​the first image is blurred using the second thread based on the blur intensity map to obtain the third image.

[0095] In one embodiment, the blur intensity map may include the blur radius corresponding to each second pixel point included in the background region of the first image. In the Gaussian blur algorithm, the blur radius represents the range of values ​​for the corresponding pixel point during blurring; a larger range indicates a higher degree of blur, and vice versa. That is, the blur radius and the degree of blur are positively correlated.

[0096] Optionally, the electronic device can use a second thread to blur each second pixel according to the blur radius corresponding to each second pixel in the background area of ​​the first image to obtain a blurred image; and use the second thread to perform point diffusion processing on the highlight areas in the blurred image to obtain a third image.

[0097] Blur processing includes, but is not limited to, Gaussian blur, box blur, or double blur algorithms, which can create a "frosted glass" effect on the image. Point spread processing includes, but is not limited to, PSF (point spread function), which can create a diffused light spot effect in bright areas. Bright areas are regions in the blurred image whose brightness is greater than a brightness threshold.

[0098] By implementing the above method, the electronic device can perform blurring and dot diffusion processing on the background area of ​​the first image through a second thread, so as to process the background area of ​​the first image into a blurred result with light spots spreading in the highlight area, thereby making the background area of ​​the processed image closer to the shooting effect of an SLR camera and improving the image quality.

[0099] In another embodiment, after determining the blur intensity map corresponding to the first image to be processed, the electronic device can generate a transition region mask based on the blur intensity map through a second thread. This transition region mask is used to characterize the transition region in the first image, which is the area where the portrait area and the background area in the first image meet. Then, after performing point diffusion processing on the highlighted areas in the blurred image through the second thread to obtain a diffused image, the electronic device can fuse the diffused image with the first image based on the transition region mask to obtain a third image.

[0100] Understandably, due to the significant difference in depth between the portrait area and the background area, the transition from the sharp portrait area to the blurred background area will be very abrupt in the first image after blurring. To address this, information from the original first image can be used to merge the transition area in the diffused image with the transition area in the first image, making the transition from the sharp portrait area to the blurred background area smoother and thus improving the image quality.

[0101] Optionally, the electronic device can determine the transition region of the first image and the transition region of the diffused image through a second thread based on the transition region mask; and fuse the first sub-image corresponding to the transition region of the first image with the transition region of the diffused image through the second thread, so as to reduce the difference in the degree of blurring between the pixels included in the transition region of the diffused image, thereby making the transition from the clear portrait area to the blurred background area smoother, thereby improving the image quality.

[0102] 408. Generate the target image corresponding to the first image based on the second and third images.

[0103] By implementing the methods disclosed in the above embodiments, a single thread can be decomposed into two threads to shorten the time for image processing operations of electronic devices, thereby improving the efficiency of image processing; and, based on the portrait segmentation map and depth map corresponding to the first image, a relative depth map corresponding to the first image can be determined, which facilitates the subsequent determination of the blurring intensity of each pixel in the first image based on the relative depth map; and, the depth values ​​of pixels with abnormal depth values ​​can be corrected back to normal depths, so that the depth information in the depth map is more accurate; and, the degree of blurring of the portrait area in the target image after blurring processing can be lower, so as to ensure that the portrait is sufficiently clear; and, the background area of ​​the first image can be blurred and dot diffusion processing can be performed on the background area of ​​the first image through the second thread, so as to process the background area of ​​the first image into a blurred result with light spot diffusion in the highlight area, thereby making the background area of ​​the processed image closer to the shooting effect of an SLR camera and improving the image texture; and, the transition from the clear portrait area to the blurred background area in the target image can be smoother, thereby improving the image quality.

[0104] Please see Figure 5 , Figure 5 This is a flowchart illustrating an image processing method disclosed in an embodiment of this application. The image processing method may include the following steps:

[0105] 501. Acquire the first image using the imaging device;

[0106] 502. The first image is processed by an image processor to obtain a processed image;

[0107] 503. Input the processed image into the portrait segmentation model and the depth prediction model respectively to obtain the portrait segmentation map and the depth map;

[0108] 504. Perform image preprocessing on the processed image to obtain a preprocessed image. Image preprocessing includes at least initialization, resizing, and loading preset parameters.

[0109] 505. Generate a blur intensity map corresponding to the first image based on the portrait segmentation map and the depth map; and execute steps 506 and 507 through the first thread, and steps 508 to 510 through the second thread;

[0110] 506. The first thread performs edge optimization processing on the portrait region in the portrait segmentation image to obtain an optimized portrait segmentation image;

[0111] 507. The first thread determines the portrait region of the first image based on the optimized portrait segmentation map, and performs portrait beautification processing on the portrait region to obtain the second image;

[0112] 508. Generate a transition region mask based on the blur intensity map using the second thread;

[0113] 509. The background of the first image is blurred and dot diffusion is performed on the second thread according to the blur intensity map to obtain the diffusion image;

[0114] 510. Using the second thread, the diffusion image is fused with the first image based on the transition region mask to obtain the third image;

[0115] 511. Perform hair edge optimization on the second and third images, and generate the target image corresponding to the first image based on the second and third images after hair edge optimization.

[0116] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing apparatus disclosed in an embodiment of this application. This image processing apparatus can be applied to the aforementioned electronic device or other execution entity, and is not limited thereto. The image processing apparatus may include a first processing unit 601, a second processing unit 602, and a generation unit 603, wherein:

[0117] The first processing unit 601 is used to perform portrait beautification processing on the portrait area of ​​the first image to be processed through the first thread to obtain the second image;

[0118] The second processing unit 602 is used to blur the background area of ​​the first image through the second thread to obtain the third image. The background area is the area in the first image other than the human image area. The first thread and the second thread are two threads that run in parallel.

[0119] The generation unit 603 is used to generate a target image corresponding to the first image based on the second image and the third image.

[0120] By implementing the aforementioned image processing device, the previously single-threaded portrait beautification and background blurring operations can be decoupled. Instead, a first thread and a second thread execute the portrait beautification and background blurring operations in parallel, respectively. Finally, the portrait beautification and background blurring results are fused to obtain an optimized image. It is understandable that since the portrait and background regions are two independent areas, decoupling them for processing does not affect the image processing effect. Furthermore, splitting the single-threaded operation into a dual-threaded one can shorten the time required for image processing operations on the electronic device, thereby improving image processing efficiency.

[0121] As an optional implementation, the first processing unit 601 is further configured to acquire a portrait segmentation map corresponding to the first image to be processed, and to perform edge optimization processing on the portrait region in the portrait segmentation map corresponding to the first image to be processed by a first thread to obtain an optimized portrait segmentation map, wherein the portrait segmentation map is used to describe the image position of the portrait region in the first image; and to determine the portrait region of the first image based on the optimized portrait segmentation map by the first thread, and to perform portrait beautification processing on the portrait region.

[0122] By implementing the above-mentioned device, edge optimization processing can be performed on the portrait region in the portrait segmentation map corresponding to the first image, so that the portrait region determined by the subsequent portrait segmentation map is more accurate. This can prevent the subsequent portrait beautification operation from being applied to the background region and destroying the image details of the background region.

[0123] As an optional implementation method, Figure 6 The image processing apparatus shown also includes a first determining unit (not shown), wherein:

[0124] The first determining unit is used to determine the blur intensity map corresponding to the first image to be processed before blurring the background area of ​​the first image through the second thread to obtain the third image. The blur intensity map is used to characterize the blur degree corresponding to each first pixel point included in the first image.

[0125] Furthermore, the second processing unit 602 is also configured to perform blurring processing on the background area of ​​the first image according to the blurring intensity map through the second thread to obtain the third image.

[0126] By implementing the above device, a blur intensity map can be generated to guide subsequent blur processing, thereby improving the subsequent blur effect.

[0127] As an optional implementation, the blur intensity map includes the blur radius corresponding to each second pixel point included in the background region of the first image; and the second processing unit 602 is further configured to perform blur processing on each second pixel point according to the blur radius corresponding to each second pixel point included in the background region of the first image through a second thread to obtain a blurred image; and to perform point diffusion processing on the bright areas in the blurred image through the second thread to obtain a third image, wherein the bright areas are areas in the blurred image whose brightness is greater than a brightness threshold.

[0128] By implementing the above device, the background area of ​​the first image can be blurred and dot diffusion processed by the second thread to make the background area of ​​the first image blurred and the highlight area has light spot diffusion. This makes the background area of ​​the processed image closer to the shooting effect of an SLR camera and improves the image quality.

[0129] As an optional implementation method, Figure 6 The image processing apparatus shown also includes a second generation unit (not shown), wherein:

[0130] The second generation unit is used to generate a transition region mask based on the blur intensity map of the first image to be processed through the second thread after determining the blur intensity map. The transition region mask is used to characterize the transition region in the first image. The transition region is the region where the portrait region and the background region in the first image meet.

[0131] Furthermore, the second processing unit 602 is also configured to fuse the diffusion image with the first image using a second thread based on a transition region mask to obtain a third image.

[0132] By implementing the above-mentioned device, the transition from the clear human figure area to the blurred background area in the subsequently obtained target image can be smoother, thereby improving the image quality.

[0133] As an optional implementation, the first determining unit is further configured to acquire a portrait segmentation map and a depth map corresponding to the first image to be processed, and determine a first average depth of the portrait region and a second average depth of the background region of the first image based on the portrait segmentation map and the depth map corresponding to the first image to be processed, wherein the depth map is used to describe the depth information of each first pixel included in the first image; and generate a relative depth map based on the first average depth of the portrait region and the second average depth of the background region of the first image, wherein the depth value of each pixel included in the portrait region of the relative depth image is the first average depth, and the depth value of each pixel included in the background region of the relative depth image is the second average depth; and determine a blur intensity map corresponding to the first image based on the relative depth map.

[0134] By implementing the above device, the relative depth map corresponding to the first image can be determined based on the portrait segmentation map and depth map corresponding to the first image, which facilitates the subsequent determination of the blurring intensity of each pixel in the first image based on the relative depth map.

[0135] As an optional implementation, the first determining unit is further configured to input the first image to be processed into a portrait segmentation model to obtain a portrait segmentation map corresponding to the first image; and to input the first image to be processed into a depth prediction model to obtain a depth map corresponding to the first image.

[0136] By implementing the above-mentioned device, the human figure segmentation map and depth map corresponding to the first image can be quickly obtained through the model, thereby improving the speed of image processing.

[0137] As an optional implementation, the first determining unit is further configured to perform edge optimization on the edges of the portrait region of the depth map based on the portrait segmentation map to obtain an optimized depth map; and to determine the first average depth of the portrait region and the second average depth of the background region of the first image based on the optimized depth map.

[0138] By implementing the above device, the depth values ​​of pixels with abnormal depth values ​​can be corrected back to normal depths, making the depth information in the depth map more accurate.

[0139] As an optional implementation method, Figure 6 The image processing apparatus shown also includes a second determining unit and an adjusting unit (not shown), wherein:

[0140] The second determining unit is used to determine the face region in the depth map after generating a relative depth map based on the first average depth of the portrait region and the second average depth of the background region of the first image, and to take the depth value with the most corresponding pixels in the face region as the target depth value.

[0141] The adjustment unit is used to adjust the depth values ​​of each pixel in the portrait region of the relative depth image from the first average depth to the target depth value, and adjust the depth values ​​of each pixel in the background region of the relative depth image according to the difference between the first average depth and the target depth value, so as to obtain the optimized relative depth map.

[0142] Furthermore, the first determining unit is also used to determine the blur intensity map corresponding to the first image based on the optimized relative depth map.

[0143] By implementing the above device, the depth value with the most corresponding pixels in the face region can be used as the average depth of the portrait region. This can reduce the degree of blurring of each pixel in the portrait region in the subsequent blurring intensity map determined by the optimized relative depth map. As a result, the degree of blurring of the portrait region in the target image after blurring is low, so as to ensure that the portrait is clear enough.

[0144] As an optional implementation, the generation unit 603 is also used to generate a target image corresponding to the first image based on the second image and the third image, when the first thread completes the portrait beautification process and the second thread completes the blurring process.

[0145] By implementing the above device, a waiting program can be set to execute subsequent steps only after both threads have completed image processing. This can prevent the electronic device from using an image that has not completed image processing to generate the target image corresponding to the first image, thereby reducing the image quality of the target image.

[0146] As an optional implementation, the generation unit 603 is further configured to obtain a weight map corresponding to the first image, the weight map being used to describe the weights corresponding to the foreground region and the background region of the first image; and to perform image fusion on the second image and the third image according to the weight map to obtain a target image corresponding to the first image.

[0147] By implementing the above-mentioned device, the image-processed regions of the second and third images can be merged into a target image with a clear foreground and a blurred background through image fusion, thereby improving the image quality while shortening the image processing time.

[0148] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. For example... Figure 7 As shown, the electronic device may include:

[0149] Memory 701 storing executable program code;

[0150] Processor 702 coupled to memory 701;

[0151] The processor 702 calls the executable program code stored in the memory 701 to execute the image processing methods disclosed in the above embodiments.

[0152] This application discloses a computer-readable storage medium storing a computer program that causes a computer to execute the image processing methods disclosed in the above embodiments.

[0153] This application also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer performs some or all of the steps of the methods described in the above method embodiments.

[0154] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0155] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0156] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of this application.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0160] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and computer-readable storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, The method includes: The first thread performs portrait enhancement processing on the portrait area of ​​the first image to be processed to obtain the second image; Determine the blur intensity map corresponding to the first image to be processed, the blur intensity map being used to characterize the blur degree corresponding to each first pixel point included in the first image; The background region of the first image is blurred by the second thread according to the blur intensity map to obtain the third image. The background region is the region in the first image other than the human image region. The first thread and the second thread are two threads that run in parallel. The blur intensity map is determined according to the human image segmentation map and depth map corresponding to the first image to be processed. The depth map is obtained by inputting the first image to be processed into the depth prediction model. Generate a target image corresponding to the first image based on the second image and the third image; Furthermore, before blurring the background region of the first image using the second thread based on the blur intensity map, the method further includes: determining the portrait region in the depth map based on the portrait segmentation map, identifying pixels in the portrait region of the depth map whose depth values ​​are greater than a first depth threshold to be corrected, and revising the depth value corresponding to the pixel to be corrected to the average of the depth values ​​of its two adjacent pixels.

2. The method according to claim 1, characterized in that, The portrait enhancement processing operation performed on the portrait region of the first image to be processed through the first thread includes: A portrait segmentation map corresponding to the first image to be processed is obtained, and the portrait region in the portrait segmentation map is subjected to edge optimization processing through a first thread to obtain an optimized portrait segmentation map. The portrait segmentation map is used to describe the image position of the portrait region in the first image. The first thread determines the portrait region of the first image based on the optimized portrait segmentation map, and performs portrait beautification processing on the portrait region.

3. The method according to claim 1, characterized in that, The blur intensity map includes the blur radius corresponding to each second pixel point in the background region of the first image; the step of blurring the background region of the first image according to the blur intensity map using a second thread to obtain a third image includes: The second thread performs blurring processing on each second pixel point according to the blur radius corresponding to each second pixel point included in the background area of ​​the first image to obtain a blurred image. The second thread performs point diffusion processing on the bright areas in the blurred image to obtain a third image, wherein the bright areas are regions in the blurred image whose brightness is greater than a brightness threshold.

4. The method according to claim 3, characterized in that, After determining the blur intensity map corresponding to the first image to be processed, the method further includes: A transition region mask is generated by a second thread based on the blur intensity map. The transition region mask is used to characterize the transition region in the first image. The transition region is the area where the portrait area and the background area in the first image meet. And, the step of performing point diffusion processing on the highlighted areas in the blurred image through the second thread to obtain the third image includes: The second thread performs point diffusion processing on the highlighted areas in the blurred image to obtain a diffused image; The second thread fuses the diffused image with the first image based on the transition region mask to obtain a third image.

5. The method according to any one of claims 1 to 4, characterized in that, The process of determining the blur intensity map corresponding to the first image to be processed includes: Obtain the portrait segmentation map and depth map corresponding to the first image to be processed, and determine the first average depth of the portrait region and the second average depth of the background region of the first image based on the portrait segmentation map and depth map. The depth map is used to describe the depth information of each first pixel point included in the first image. A relative depth map is generated based on the first average depth of the portrait region and the second average depth of the background region of the first image. The depth value of each pixel in the portrait region of the relative depth map is the first average depth, and the depth value of each pixel in the background region of the relative depth map is the second average depth. The blur intensity map corresponding to the first image is determined based on the relative depth map.

6. The method according to claim 5, characterized in that, The step of obtaining the portrait segmentation map and depth map corresponding to the first image to be processed includes: The first image to be processed is input into the portrait segmentation model to obtain the portrait segmentation map corresponding to the first image; The first image to be processed is input into the depth prediction model to obtain the depth map corresponding to the first image.

7. The method according to claim 5, characterized in that, Determining the first average depth of the portrait region and the second average depth of the background region of the first image based on the portrait segmentation map and the depth map includes: The image region of the depth map is edge-optimized based on the image segmentation map to obtain an optimized depth map; The first average depth of the portrait region and the second average depth of the background region of the first image are determined based on the optimized depth map.

8. The method according to claim 5, characterized in that, After generating a relative depth map based on the first average depth of the human figure region and the second average depth of the background region of the first image, the method further includes: The face region is determined in the depth map, and the depth value with the most corresponding pixels in the face region is taken as the target depth value. The depth values ​​of each pixel in the portrait region of the relative depth image are adjusted from the first average depth to the target depth value, and the depth values ​​of each pixel in the background region of the relative depth image are adjusted according to the difference between the first average depth and the target depth value to obtain an optimized relative depth map. And, determining the blur intensity map corresponding to the first image based on the relative depth map includes: The bokeh intensity map corresponding to the first image is determined based on the optimized relative depth map.

9. The method according to any one of claims 1 to 4 or 6 to 8, characterized in that, The step of generating the target image corresponding to the first image based on the second image and the third image includes: When the first thread completes the portrait beautification process and the second thread completes the blurring process, a target image corresponding to the first image is generated based on the second image and the third image.

10. The method according to any one of claims 1-4 or 6-8, characterized in that, The step of generating the target image corresponding to the first image based on the second image and the third image includes: Obtain the weight map corresponding to the first image, the weight map being used to describe the weights corresponding to the foreground and background regions of the first image respectively; Image fusion is performed on the second image and the third image according to the weight map to obtain the target image corresponding to the first image.

11. An image processing apparatus, characterized in that, The device includes: The first processing unit is used to perform portrait beautification processing on the portrait area of ​​the first image to be processed through the first thread to obtain the second image; The second processing unit is used to determine the blur intensity map corresponding to the first image to be processed, the blur intensity map being used to characterize the blur degree corresponding to each first pixel in the first image; and to perform blur processing on the background area of ​​the first image according to the blur intensity map by a second thread to obtain a third image, the background area being the area in the first image excluding the portrait area, the first thread and the second thread being two threads running in parallel, the blur intensity map being determined based on the portrait segmentation map and depth map corresponding to the first image to be processed, the depth map being obtained by inputting the first image to be processed into a depth prediction model; The generation unit is configured to generate a target image corresponding to the first image based on the second image and the third image; Furthermore, the device is further configured to, before blurring the background region of the first image according to the blurring intensity map via the second thread, determine the portrait region in the depth map according to the portrait segmentation map, identify the pixel point to be corrected in the portrait region of the depth map whose depth value is greater than a first depth threshold, and revise the depth value corresponding to the pixel point to be corrected to the average of the depth values ​​of its two adjacent pixels.

12. An electronic device, characterized in that, The method includes a memory storing executable program code and a processor coupled to the memory; wherein the processor invokes the executable program code stored in the memory to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 10.

Images