Image processing and blurring method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide an image processing and image blurring method, apparatus, electronic device and storage medium. The image processing method comprises: performing stereo matching on a first image captured by a first camera and a second image captured by a second camera to obtain a disparity map; performing semantic edge detection on the first image to determine semantic edge regions and non-semantic edge regions in the first image; and performing edge-preserving smoothing filtering on the disparity map based on the gradient of pixels in the first image, the semantic edge regions and the non-semantic edge regions.

Description

Technical Field

[0001] The present application relates to a field, and specifically to image processing and blurring methods, devices, electronic devices and storage media. Background Art

[0002] Background blur is an important means to optimize the display effect of the image. Especially for large-area background objects such as lawns and stone roads, the smoothness of lawns and stone roads is particularly important. If they are not smooth enough, they will cause blurring agglomeration, unevenness, and other problems, affecting people's visual experience.

[0003] At present, mobile terminals are generally equipped with dual cameras. When shooting with dual cameras, for example, when shooting with dual cameras in portrait mode, binocular stereo matching is performed on the image captured by the main camera and the image captured by the secondary camera to obtain a disparity map, and the background of the corresponding image is blurred according to the disparity map. Usually, the background of the image captured by the main camera is blurred using the disparity map.

[0004] Before using the disparity map to blur the background, it is necessary to perform edge-preserving smoothing filtering on the disparity map.

[0005] However, for large background objects that often appear in images that need to be blurred, the current edge-preserving smoothing filtering algorithm will cause the edges of each small object in the background object to be highlighted in the corresponding blurred image obtained by using the disparity map to blur the background of the corresponding image. At the same time, the edge of each small object is not smooth, which ultimately makes the display effect of the background object unnatural and not smooth, affecting the user's visual experience. Summary of the invention

[0006] Embodiments of the present application provide an image processing method, an image blurring method, an image processing device, an electronic device, and a storage medium.

[0007] The present application provides an image processing method, including:

[0008] Performing stereo matching on a first image captured by a first camera and a second image captured by a second camera to obtain a disparity map;

[0009] Performing semantic edge detection on the first image to determine a semantic edge region and a non-semantic edge region in the first image;

[0010] Based on the gradients, semantic edge regions, and non-semantic edge regions of the pixels in the first image, edge-preserving smoothing filtering is performed on the disparity map.

[0011] The present application provides an image blurring method, including:

[0012] Processing the disparity map to obtain a processed disparity map, wherein the disparity map is obtained by stereo matching a first image captured by a first camera and a second image captured by a second camera, and the processed disparity map is obtained by processing the disparity map according to the above-mentioned image processing method;

[0013] The first image is blurred using the processed disparity map to obtain a blurred effect rendered image.

[0014] The present application provides an image processing device, including:

[0015] A generating unit is configured to perform stereo matching on a first image captured by the first camera and a second image captured by the second camera to obtain a disparity map;

[0016] A detection unit configured to perform semantic edge detection on the first image to determine a semantic edge region and a non-semantic edge region in the first image;

[0017] The processing unit is configured to perform edge-preserving smoothing filtering on the disparity map based on the gradient of pixels in the first image, semantic edge regions, and non-semantic edge regions.

[0018] The image processing method and image blurring method provided by the embodiments of the present application realize edge-preserving smoothing filtering processing of the disparity map based on the gradient, semantic edge area and non-semantic edge area of ​​the pixels in the first image. Thus, in the first image with blurred background obtained by blurring the background of the first image captured by the first camera using the processed disparity map, the semantic edge, for example, the semantic edge of the user who is the subject of the photo in the portrait mode, can be retained. At the same time, the display effect of the background objects with larger areas, such as lawns and stone roads, is smooth, and the unnatural and non-smooth display effect of the background objects caused by the edge preservation of small-sized objects such as leaves and stones in the background objects is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application is shown;

[0021] Figure 2 A flow chart of an image blurring method provided by an embodiment of the present application is shown;

[0022] Figure 3 A structural block diagram of an image processing device provided by an embodiment of the present application is shown;

[0023] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application is shown, and the method includes:

[0027] Step 101: Perform stereo matching on a first image captured by a first camera and a second image captured by a second camera to obtain a disparity map.

[0028] The stereo matching may be binocular stereo matching, and the obtained disparity map may describe the disparity between matching pixel points in the first image captured by the first camera and the second image captured by the second camera.

[0029] The first image and the second image may be color images. The first camera may be a main camera in a dual camera or a multi-camera, and the second camera may be a secondary camera in a dual camera or a multi-camera. The first camera may also be a secondary camera in a dual camera or a multi-camera, and the second camera may also be a main camera in a dual camera or a multi-camera.

[0030] Step 102: Perform semantic edge detection on the first image to determine semantic edge regions and non-semantic edge regions in the first image.

[0031] In the present application, a semantic edge detector may be used to detect semantic edges in a first image captured by a first camera. Each detected semantic edge belongs to an object. The number of detected semantic edges may be one or more.

[0032] In the present application, the area occupied by the semantic edge in the first image captured by the first camera may be referred to as a semantic edge area, and the area other than the semantic edge area in the first image captured by the first camera may be referred to as a non-semantic edge area.

[0033] When using a semantic edge detector to detect semantic edges in an image, the detection result output by the semantic edge detector is the edge of an object whose size is larger than a preset size. The semantic edge detector will only detect the edges of objects of a certain size in the image, that is, the edges of larger objects. The edges of larger objects will be detected as semantic edges, while the edges of very small objects will not be detected. This is determined by the characteristics of the semantic edge detector. In other words, the semantic edge detector will not determine the edges of very small objects as semantic edges.

[0034] For example, when photographing a person, the user is the subject of the photograph, and the first image captured by the first camera and the second image captured by the second camera both include the user and background objects in the user's environment, such as lawns, stone roads, etc.

[0035] The semantic edge detector detects the semantic edge in the first image captured by the first camera, and can detect the semantic edge of the user, which may be the entire outline or a part of the outline of the user's body. It is also possible to detect the semantic edges of other objects of larger size. However, the semantic edge detector will not detect the edges of small objects such as leaves in the lawn and stones on the stone road. In other words, the semantic edge detector will not determine the edges of small objects such as leaves in the lawn and stones on the stone road as semantic edges.

[0036] In some embodiments, performing semantic edge detection on the first image to determine semantic edge regions and non-semantic edge regions in the first image includes: performing semantic edge detection on the first image using a semantic edge detector based on machine learning to obtain the confidence that each pixel belongs to a semantic edge; comparing the confidence that each pixel belongs to a semantic edge with a confidence threshold to determine the semantic edge region and non-semantic edge region in the first image.

[0037] In the present application, a semantic edge detector based on machine learning may be used to perform semantic edge detection on the first image.

[0038] The semantic edge detector based on machine learning can be a structured-edge based edge detector (SE), a rich convolutional features edge detector (RCF), etc.

[0039] The semantic edge detector based on machine learning can output an edgeMap. The edgeMap output by the semantic edge detector includes the confidence that each pixel in the first image belongs to a semantic edge. The confidence that each pixel belongs to a semantic edge can be compared with a confidence threshold. For each pixel in the first image, if the confidence that the pixel belongs to a semantic edge is greater than a confidence threshold, such as 0.7, it can be determined that the pixel belongs to a semantic edge. If the confidence that the pixel belongs to a semantic edge is less than the confidence threshold, it can be determined that the pixel belongs to a non-semantic edge. After determining that the pixel belongs to a semantic edge or a non-semantic edge for each pixel, it can be determined which pixels in the first image captured by the first camera belong to the semantic edge and which pixels are pixels that do not belong to the semantic edge. Then, the semantic edge area and the non-semantic edge area in the first image can be further determined.

[0040] In some embodiments, semantic edge detection is performed on the first image using a semantic edge detector based on machine learning to obtain the confidence that each pixel belongs to the semantic edge, including: reducing the size of the first image to a preset size to obtain a reduced first image; inputting the reduced first image into a semantic edge detector based on machine learning to obtain the confidence that each pixel belongs to the semantic edge.

[0041] In the present application, when using a semantic edge detector based on machine learning to detect semantic edges in a first image captured by a first camera that is related to a disparity map, the size of the first image captured by the first camera can be reduced to a preset size, and the reduced first image can be used for detection to reduce the overhead of the detection process.

[0042] For example, the preset size is 1 / 32 of the original size of the first image captured by the first camera, and the size of the first image captured by the first camera is reduced to the preset size to obtain a reduced first image, and the size of the reduced first image is 1 / 32 of the original size of the first image captured by the first camera. The reduced first image is input to the semantic edge detector based on machine learning to obtain the confidence of each pixel output by the semantic edge detector based on machine learning that it belongs to a semantic edge.

[0043] In some embodiments, the first image captured by the first camera is an image captured by the first camera in portrait mode, and the second image captured by the second camera is an image captured by the second camera in portrait mode.

[0044] Portrait mode is a common shooting mode. In portrait mode, binocular stereo matching can be performed on the second image captured by the first camera and the second image captured by the second camera to obtain a disparity map. At the same time, semantic edges in the first image captured by the first camera in portrait mode are detected.

[0045] Step 103 : performing edge-preserving smoothing filtering on the disparity map based on the gradients, semantic edge regions, and non-semantic edge regions of the pixels in the first image.

[0046] In the present application, each pixel in the first image captured by the first camera corresponds to a disparity map pixel in the disparity map.

[0047] For a semantic edge region in a first image captured by a first camera, the disparity map includes a first region corresponding to the semantic edge region. The first region corresponding to the semantic edge region includes disparity map pixels corresponding to pixels in the semantic edge region.

[0048] For a non-semantic edge region in a first image captured by a first camera, the disparity map includes a second region corresponding to the non-semantic edge region. The second region corresponding to the semantic edge region includes disparity map pixels corresponding to pixels in the non-semantic edge region.

[0049] For each pixel in the first image captured by the first camera, the center point of the filter window of the pixel is the pixel, and the center point of the filter window of the disparity map pixel corresponding to the pixel is the disparity map pixel corresponding to the pixel. The range of the filter window of the pixel is the same as the range of the filter window of the disparity map pixel corresponding to the pixel. Therefore, by determining the range of the filter window of the pixel, the range of the filter window of the disparity map pixel corresponding to the pixel can be determined.

[0050] For each pixel in the first image collected by the first camera, the filtering window of the pixel can be preliminarily determined by searching for neighboring pixels that belong to the same filtering window as the pixel. For each pixel in the first image collected by the first camera, the similarity between the pixel and the neighboring pixels of the pixel can be calculated based on the color similarity and spatial similarity between the pixel and the neighboring pixels of the pixel. The spatial similarity can be the distance between the pixel and the neighboring pixels of the pixel. The color similarity and spatial similarity between the pixel and the neighboring pixels of the pixel can be weightedly calculated to obtain the similarity between the pixel and the neighboring pixels of the pixel.

[0051] For each pixel in the first image captured by the first camera, when determining the filter window of the pixel, the neighboring pixels in the filter window of the pixel to be determined can be searched according to the similarity between the pixel and the neighboring pixels of the pixel. If the similarity is greater than the similarity threshold, the search is further outward, that is, the search is continued in the direction away from the pixel until a neighboring pixel whose similarity with the pixel is less than the similarity threshold is found, and the search is stopped, so that the area pixel where the search is stopped is used as the boundary pixel of the filter window, thereby determining the range of the filter window of the pixel, and the pixel is the center point of the filter window of the pixel, and the filter window of the pixel can be further determined.

[0052] For each pixel in the first image captured by the first camera, if the pixel is in a non-edge area such as a flat area, the difference between the pixel and a certain number of neighboring pixels of the pixel in the non-edge area is small, and a large number of neighboring pixels can be searched out. Correspondingly, the number of pixels included in the filter window of the pixel is large, and the range of the filter window of the pixel is large. If, in the process of filtering the first image captured by the first camera, the filter window of the pixel is used to perform a corresponding filtering operation on the pixel, for example, the mean of the pixel values ​​of the pixels included in the filter window of the pixel is used as the pixel value after filtering of the pixel, a smoothing effect can be achieved.

[0053] If the pixel is in a non-edge area, since the range of the filtering window of the pixel is the same as the range of the filtering window of the disparity map pixel corresponding to the pixel in the disparity map, the range of the filtering window of the disparity map pixel corresponding to the pixel in the disparity map is also larger. A filtering window with a larger range can be used to perform corresponding filtering operations on the disparity map pixel corresponding to the pixel in the disparity map to achieve a smoothing effect.

[0054] It should be understood that the above description of using a larger filter window to perform a corresponding filtering operation on the pixels in the image captured by the first camera to achieve a smoothing effect is only an exemplary illustration of the correlation between the size of the filter window and the smoothing effect. In the present application, the disparity map is actually filtered, and finally the disparity map pixels are subjected to a corresponding filtering operation using the filter window of the disparity map pixels.

[0055] For each pixel in the image captured by the first camera, if the pixel is in the edge area, since the difference between adjacent pixels in the edge area is large, in other words, the similarity between adjacent pixels in the edge area is low, the search will be stopped when a small number of neighboring pixels are searched out. Accordingly, the number of pixels included in the filter window of the pixel is small, and the range of the filter window of the pixel is small, which is equivalent to achieving an edge preservation effect.

[0056] If the pixel is in the edge area, since the range of the filtering window of the pixel is the same as the range of the filtering window of the disparity map pixel corresponding to the pixel in the disparity map, the range of the filtering window of the disparity map pixel corresponding to the pixel in the disparity map is also smaller. A filtering window with a smaller range is used to perform corresponding filtering operations on the disparity map pixel corresponding to the pixel in the disparity map to achieve an edge preservation effect.

[0057] In the present application, the disparity map may be subjected to edge-preserving smoothing filtering based on the gradient of pixels in the first image, semantic edge regions, and non-semantic edge regions.

[0058] The gradient of a pixel is a parameter in a filtering algorithm that can indicate the overall degree of difference between a pixel and its neighboring pixels. The size of the gradient of a pixel is inversely proportional to the size of the range of the filtering window of the pixel determined by searching. In the present application, the gradient of each pixel in the first image captured by the first camera can be calculated based on the similarity between the pixel and each neighboring pixel within a preset size range centered on the pixel.

[0059] When the disparity map is processed by edge-preserving smoothing filtering based on the gradients of pixels in the first image, semantic edge areas and non-semantic edge areas, for each pixel in the semantic edge area, the gradient of the pixel is higher, and the range of the filtering window of the pixel should be smaller, and the range of the filtering window of the pixel preliminarily determined by searching can satisfy the smaller range of the filtering window of the pixel. Therefore, the range of the filtering window of the pixel preliminarily determined by searching can be directly used as the range of the filtering window of the pixel, and accordingly, the range of the filtering window of the pixel preliminarily determined by searching can be directly used as the range of the filtering window of the disparity map pixel corresponding to the pixel.

[0060] For each pixel in the non-semantic edge region, the gradient of the pixel may be compared with a gradient threshold.

[0061] When the gradient of the pixel in the non-semantic edge area is less than the gradient threshold, it can be determined that the pixel should be in a non-edge area such as a flat area, and the range of the filter window of the pixel should be larger. The range of the filter window of the pixel preliminarily determined by searching can satisfy the larger range of the filter window of the pixel. Therefore, the range of the filter window of the pixel preliminarily determined by searching can be directly used as the range of the filter window of the pixel. Correspondingly, the range of the filter window of the pixel preliminarily determined by searching can be directly used as the range of the filter window of the disparity map pixel corresponding to the pixel.

[0062] When the gradient of the pixel in the non-semantic edge area is greater than the gradient threshold, it may be because the pixel is in the non-edge area but is a pixel on the edge of an object of very small size. Since the pixel is located in the non-edge area, a filter window with a larger range should be used to perform corresponding filtering operations on the disparity map pixel corresponding to the pixel to achieve a smoothing effect. However, since the gradient is inversely proportional to the size of the range of the filter window determined by the search, the range of the filter window of the pixel preliminarily determined by the search is relatively small. If the range of the filter window of the pixel preliminarily determined by the search is used as the filter window of the disparity map pixel corresponding to the pixel, then after the corresponding filtering operation is performed on the disparity map pixel corresponding to the pixel, an edge preservation effect will be achieved instead of a smoothing effect.

[0063] When the gradient of the pixel in the non-semantic edge area is greater than the gradient threshold, in order to achieve a smoothing effect, the range of the filtering window of an associated pixel in the non-semantic edge area whose gradient is less than the gradient threshold can be used as the range of the filtering window of the pixel, and accordingly, the range of the filtering window of an associated pixel in the non-semantic edge area whose gradient is less than the gradient threshold can be used as the range of the filtering window of the disparity map pixel corresponding to the pixel.

[0064] For example, an associated pixel in the non-semantic edge region whose gradient is less than a gradient threshold may be a pixel that is closest to the pixel among all pixels in the non-semantic edge region whose gradient is less than the gradient threshold.

[0065] In the present application, when the disparity map is subjected to edge-preserving smoothing filtering based on the gradient of pixels in the first image, the semantic edge region and the non-semantic edge region, it can be ensured that the edge-preserving effect is achieved after the corresponding filtering operation is performed on each disparity map pixel in the first region corresponding to the semantic edge region, thereby performing edge preservation on the first region corresponding to the semantic edge region.

[0066] Thus, in the first image with blurred background obtained by using the processed disparity map to blur the background of the first image captured by the first camera, the semantic edge in the first image captured by the first camera will be preserved because the first area corresponding to the semantic edge area has been preserved.

[0067] For example, the first image captured by the first camera is the first image captured by the first camera in portrait mode. The semantic edge is the semantic edge of the user as the subject of the shooting, and in the first blurred image obtained by blurring the background of the first image captured by the first camera using the processed disparity map, the semantic edge of the user will be preserved.

[0068] When the disparity map is subjected to edge-preserving smoothing filtering based on the gradient of pixels in the first image, semantic edge regions and non-semantic edge regions, it can be ensured that a smoothing effect is achieved after the corresponding filtering operation is performed on each disparity map pixel in the second region corresponding to the non-semantic edge region, thereby smoothing the second region corresponding to the non-semantic edge region.

[0069] Therefore, in the first image with blurred background obtained by blurring the background of the first image captured by the first camera using the processed disparity map, the second area corresponding to the non-semantic edge area is smoothed, so that the non-semantic edge area is smooth.

[0070] In some embodiments, performing edge-preserving smoothing filtering on the disparity map based on the gradient of pixels in the first image, the semantic edge region and the non-semantic edge region includes: determining the range of a filtering window of disparity map pixels in the disparity map based on the gradient of pixels in the first image; and filtering a first region in the disparity map corresponding to the semantic edge region and a second region in the disparity map corresponding to the non-semantic edge region, respectively, using a preset edge-preserving smoothing filtering algorithm according to the range of the filtering window.

[0071] For each pixel in the semantic edge area, the range of the filter window of the pixel determined by the search can be determined as the range of the filter window of the pixel, and correspondingly, the range of the filter window of the pixel determined by the search can be used as the range of the filter window of the disparity map pixel corresponding to the pixel.

[0072] For each pixel in the non-semantic edge area, when the gradient of the pixel is less than the gradient threshold, the range of the filter window of the pixel determined by the search can be determined as the range of the filter window of the pixel, and accordingly, the range of the filter window of the pixel determined by the search can be used as the range of the filter window of the disparity map pixel corresponding to the pixel.

[0073] When the gradient of the pixel is less than the gradient threshold, a preset filter window range can be determined as the filter window range of the disparity map pixel corresponding to the pixel, and the preset filter window range is a larger range that can ensure a smoothing effect after filtering the pixel. The largest range of the filter windows of all pixels in the non-semantic edge region whose gradients are less than the gradient threshold can also be determined as the filter window range of the disparity map pixel corresponding to the pixel.

[0074] In the present application, the preset edge-preserving smoothing filtering algorithm may be one of the existing edge-preserving smoothing filtering algorithms such as a guided filter and a domain transform filter.

[0075] When filtering the first area corresponding to the semantic edge area based on the range of the filtering window of the disparity map pixels in the disparity map using a preset edge-preserving smoothing filtering algorithm, for each disparity map pixel in the first area corresponding to the semantic edge area, since the range of the filtering window of the disparity map pixel has been determined, the center point of the filtering window of the disparity map pixel is the disparity map pixel. Therefore, the filtering window of the disparity map pixel can be finally determined, and the filtering window of the disparity map pixel is used to perform a corresponding filtering operation on the disparity map pixel. The corresponding filtering operation performed on the disparity map pixel is a filtering operation that can achieve an edge-preserving effect when edge-preserving smoothing filtering is performed using the preset edge-preserving smoothing filtering algorithm.

[0076] When filtering the second area corresponding to the non-semantic edge area based on the range of the filtering window of the disparity map pixels in the disparity map using a preset edge-preserving smoothing filtering algorithm, for each disparity map pixel in the second area corresponding to the non-semantic edge area, since the range of the filtering window of the disparity map pixel has been determined, the center point of the filtering window of the disparity map pixel is the disparity map pixel, and therefore, the filtering window of the disparity map pixel can be finally determined, and the filtering window of the disparity map pixel is used to perform a corresponding filtering operation on the disparity map pixel. The corresponding filtering operation performed on the disparity map pixel is a filtering operation that can achieve a smoothing effect when edge-preserving smoothing filtering is performed using the preset edge-preserving smoothing filtering algorithm.

[0077] In some embodiments, based on the gradient of the pixels in the first image, determining the range of the filtering window for the disparity map pixels in the disparity map includes: reducing the gradient of the target pixel in the non-semantic edge area, wherein the target pixel is a pixel having a gradient greater than a gradient threshold, and the reduced gradient of the target pixel is less than the gradient threshold; based on the reduced gradient of the target pixel and a preset correspondence, determining the range of the filtering window of the disparity map pixels corresponding to the target pixel in the disparity map, wherein the preset correspondence describes the correspondence between the gradient less than the gradient threshold and the range of the filtering window, and the size of the gradient less than the gradient threshold is inversely proportional to the size of the range of the filtering window corresponding to the gradient less than the gradient threshold.

[0078] In the present application, generally, the target pixels whose gradient in the non-semantic edge region is greater than the gradient threshold are pixels on the edge of a very small object.

[0079] The interval consisting of 0 and the gradient threshold, that is, the interval with 0 as the left end point and the gradient threshold as the right end point, can be pre-divided into a plurality of gradient sub-intervals, each of which corresponds to a range of a filtering window.

[0080] For endpoints of the same type in multiple gradient sub-intervals, the larger the endpoint of the gradient sub-interval is, the smaller the range of the filtering window corresponding to the gradient sub-interval is.

[0081] For example, for the left endpoints of multiple gradient sub-intervals, the larger the left endpoint of the gradient sub-interval is, the smaller the range of the filter window corresponding to the gradient sub-interval is. For a gradient sub-interval, the range of the filter window corresponding to the gradient sub-interval is smaller than the range of the filter window corresponding to the preceding gradient sub-interval of the gradient sub-interval, and the range of the filter window corresponding to the gradient sub-interval is larger than the range of the filter window corresponding to the succeeding gradient sub-interval of the gradient sub-interval.

[0082] The left endpoint of the preceding gradient subinterval of the gradient subinterval is smaller than the left endpoint of the gradient subinterval, and the left endpoint of the succeeding gradient subinterval of the gradient subinterval is larger than the left endpoint of the gradient subinterval.

[0083] For a reduced gradient and another reduced gradient, if the reduced gradient is within the gradient subinterval, the other reduced gradient is within the preceding gradient subinterval of the gradient subinterval. Since the range of the filter window corresponding to the reduced gradient is the range of the filter window corresponding to the gradient subinterval where the reduced gradient is located, and the range of the filter window corresponding to the gradient subinterval is smaller than the range of the filter window corresponding to the preceding gradient subinterval of the gradient subinterval, the range of the filter window corresponding to the reduced gradient is smaller than the range of the filter window corresponding to the preceding gradient subinterval of the gradient subinterval.

[0084] For a reduced gradient and another reduced gradient, if the reduced gradient is within the gradient subinterval, the other reduced gradient is within a subsequent gradient subinterval of the gradient subinterval. The range of the filter window corresponding to the gradient subinterval is greater than the range of the filter window corresponding to the subsequent gradient subinterval of the gradient subinterval, and therefore, the range of the filter window corresponding to the reduced gradient is greater than the range of the filter window corresponding to the other reduced gradient.

[0085] For each of the multiple gradient sub-intervals, the range of the filter window corresponding to any gradient in the gradient sub-interval is the range of the filter window corresponding to the gradient sub-interval. Thus, a preset corresponding relationship can be established.

[0086] For endpoints of the same type of multiple gradient sub-intervals, the range of the filter window corresponding to the gradient sub-interval with the largest endpoint is the smallest, and the range of the filter window corresponding to the gradient sub-interval with the largest endpoint can ensure that a smoothing effect is achieved after the corresponding disparity map pixels are filtered using the filter window with the range of the filter window.

[0087] For example, for the left endpoints of multiple gradient sub-intervals, the range of the filter window corresponding to the gradient sub-interval with the largest left endpoint can ensure that a smoothing effect is achieved after the corresponding disparity map pixels are filtered using the filter window with the range of the filter window.

[0088] Therefore, the range of the filter window corresponding to any gradient in any gradient sub-interval can ensure that a smoothing effect is achieved after the corresponding disparity map pixels are filtered using the filter window having the range of the filter window.

[0089] In the case where the non-semantic edge area includes target pixels whose gradients are greater than a gradient threshold, for the reduced gradient of any target pixel, the range of the filtering window corresponding to the reduced gradient of the target pixel is the range of the filtering window corresponding to the gradient sub-interval in which the reduced gradient of the target pixel is located.

[0090] For the reduced gradient of any target pixel, the range of the filtering window corresponding to the reduced gradient of the target pixel can ensure that a smoothing effect is achieved after a filtering operation is performed on the disparity map pixels corresponding to the target pixel using a filtering window having the range of the filtering window.

[0091] When determining the range of the filtering window for the disparity map pixels in the disparity map based on the gradients of the pixels in the first image, the gradient of each target pixel can be reduced to the same gradient, which is less than the gradient threshold, and the same gradient is used as the reduced gradient of each target pixel. Thus, the reduced gradient of each target pixel is less than the gradient threshold.

[0092] For example, the gradient of each target pixel is reduced to 0, that is, the reduced gradient of each target pixel is 0.

[0093] When determining the range of the filtering window for the disparity map pixels in the disparity map based on the gradient of the pixels in the first image, for each target pixel in the non-semantic edge area, the range of the filtering window corresponding to the reduced gradient of the target pixel in a preset correspondence relationship can be determined as the range of the filtering window of the disparity map pixels in the disparity map corresponding to the target pixel.

[0094] The following is a comparison to illustrate the difference between the existing edge-preserving smoothing filtering algorithm and the smoothing filtering process in this application:

[0095] The edges of some very small objects in the first image captured by the first camera may also be determined to be related to the edges due to the high gradient of the pixels. The disparity map pixels in the disparity map correspond one-to-one to the pixels in the first image captured by the first camera. Accordingly, the disparity map pixels in the to-be-processed area corresponding to the area occupied by the edge of the very small object in the disparity map correspond one-to-one to the pixels in the area occupied by the edge of the very small object.

[0096] The existing edge-preserving smoothing filtering algorithm determines that the pixels of the edge of the object of very small size are related to the edge. Accordingly, when the existing edge-preserving smoothing filtering algorithm performs edge-preserving smoothing filtering on the to-be-processed area corresponding to the area occupied by the edge of the object of very small size in the disparity map, each disparity map pixel in the to-be-processed area corresponding to the area occupied by the edge of the object of very small size in the disparity map is processed in accordance with the edge-preserving processing. As a result, the to-be-processed area corresponding to the area occupied by the edge of the object of very small size in the disparity map is processed in accordance with the edge-preserving processing.

[0097] Ultimately, in the first image with blurred background obtained by using the processed disparity map to blur the background of the first image captured by the first camera, the edge of each very small object will be preserved, and the edge of each very small object will be highlighted. The display effect of each very small object is unnatural, and the edge of each very small object is not smooth.

[0098] For example, in the portrait mode, the subject is a user, and the first image captured by the first camera often includes background objects such as lawns and stone roads in addition to the user.

[0099] The gradient of the pixels at the edge of the leaves in the lawn is large, and the existing edge-preserving smoothing filtering algorithm determines that the pixels belonging to the edge of the leaves are related to the edge based on the gradient of the pixels. Each disparity map pixel in the to-be-processed area corresponding to the area occupied by the edge of the leaves in the disparity map is processed in accordance with the edge-preserving processing. Thus, the to-be-processed area corresponding to the area occupied by the edge of the leaves is processed in accordance with the edge-preserving processing. In the first image with blurred background obtained by blurring the background of the first image captured by the first camera using the processed disparity map, the edge of each leaf in the lawn is highlighted, the display effect of each leaf is unnatural, and the display effect of the edge of each leaf is not smooth.

[0100] The pixel gradient of the edge of the stone on the stone road is large, and the existing edge-preserving smoothing filter algorithm determines that the pixel of the edge of the stone is related to the edge according to the pixel gradient.

[0101] Each disparity map pixel in the to-be-processed area corresponding to the area occupied by the edge of the stone in the disparity map is processed in accordance with the edge preservation. Thus, the to-be-processed area corresponding to the area occupied by the edge of the stone is processed in accordance with the edge preservation. In the first image obtained by blurring the background of the first image captured by the first camera using the processed disparity map, the edge of each stone on the stone road is highlighted, the display effect of each stone is unnatural, and the display effect of the edge of each stone is not smooth.

[0102] In the image acquired by the first camera after the background blur is obtained by using the processed disparity map to blur the background of the image acquired by the first camera, the edge of each leaf in the lawn is highlighted, and the display effect of the edge of each leaf is not smooth, resulting in an unnatural display effect of the entire lawn and an uneven display effect of the entire lawn. Similarly, the edge of each stone on the stone road is highlighted, and the display effect of each stone is not natural, and the display effect of the edge of each stone is not smooth, resulting in an unnatural display effect of the entire stone road and an uneven display effect of the entire stone road.

[0103] In the present application, when the disparity map is subjected to edge-preserving smoothing filtering based on the gradient of pixels in the first image, the semantic edge region, and the non-semantic edge region, it can be ensured that after the corresponding filtering operation is performed on each disparity map pixel in the first region corresponding to the semantic edge region, the edge-preserving effect is achieved, thereby the first region corresponding to the semantic edge region is edge-preserved. It can be ensured that after the corresponding filtering operation is performed on each disparity map pixel in the second region corresponding to the non-semantic edge region, the smoothing effect is achieved, thereby the second region corresponding to the non-semantic edge region is smoothed.

[0104] Thus, in the image acquired by the first camera with background blur obtained by using the processed disparity map to blur the background of the image acquired by the first camera, the semantic edge in the image acquired by the first camera will be preserved because the first area corresponding to the semantic edge area has been edge-preserved. In the image acquired by the first camera with background blur obtained by using the processed disparity map to blur the background of the image acquired by the first camera, the non-semantic edge area is smoothed because the second area corresponding to the non-semantic edge area is smoothed.

[0105] In the present application, the area in the disparity map corresponding to the area occupied by the edge of the small-sized object in the background object will not be edge-preserved, but smoothed, thereby avoiding the situation where the edge of the small-sized object in the background object in the image captured by the first camera after the background blur is preserved, resulting in an unnatural and uneven display effect.

[0106] For example, in portrait mode, the subject is a user, and the image captured by the first camera often includes background objects such as lawns and stone roads in addition to the user. By performing edge preservation processing on the first area corresponding to the semantic edge area in the disparity map and smoothing processing on the second area corresponding to the non-semantic edge area in the disparity map, the image captured by the first camera is blurred using the processed disparity map to obtain an image captured by the first camera with blurred background, and the semantic edge of the user can be retained. At the same time, the display effect of larger background objects such as lawns and stone roads is smooth, and the unnatural and non-smooth display effect of background objects caused by small objects such as leaves and stones in the background objects due to edge preservation is avoided.

[0107] Please refer to Figure 2 , which shows a flow chart of the image blurring method provided in an embodiment of the present application.

[0108] Step 201 , processing the disparity map to obtain a processed disparity map.

[0109] In the present application, the disparity map is obtained by stereo matching a first image captured by a first camera and a second image captured by a second camera.

[0110] The processed disparity map is obtained by processing the disparity map using the image processing method described in the above embodiment.

[0111] Step 202: blur the first image using the processed disparity map to obtain a blurred rendering image.

[0112] After obtaining the processed disparity map, when using the processed disparity map to blur the background of the image captured by the first camera, the processed disparity map can be used to blur the image captured by the first camera to obtain a blurred effect rendered image. Blurring the first image captured by the first camera can be blurring the background of the first image using a large aperture effect rendering.

[0113] Please refer to Figure 3 , which shows a structural block diagram of an image processing device provided in an embodiment of the present application. The image processing device comprises: a generating unit 301 , a detecting unit 302 , and a processing unit 303 .

[0114] The generating unit 301 is configured to perform stereo matching on a first image captured by the first camera and a second image captured by the second camera to obtain a disparity map;

[0115] The detection unit 302 is configured to perform semantic edge detection on the first image to determine a semantic edge region and a non-semantic edge region in the first image;

[0116] The processing unit 303 is configured to perform edge-preserving smoothing filtering on the disparity map based on the gradient, semantic edge region and non-semantic edge region of the pixels in the first image.

[0117] In some embodiments, the processing unit 303 is further configured to determine the range of the filtering window of the disparity map pixels in the disparity map based on the gradient of the pixels in the first image; and according to the range of the filtering window, use a preset edge-preserving smoothing filtering algorithm to filter the first area in the disparity map corresponding to the semantic edge area and the second area in the disparity map corresponding to the non-semantic edge area.

[0118] In some embodiments, the processing unit 303 is further configured to reduce the gradient of a target pixel in the non-semantic edge area, wherein the target pixel is a pixel having a gradient greater than a gradient threshold, and the reduced gradient of the target pixel is less than the gradient threshold; based on the reduced gradient of the target pixel and a preset correspondence, determine the range of a filtering window of a disparity map pixel corresponding to the target pixel in the disparity map, wherein the preset correspondence describes the correspondence between a gradient less than the gradient threshold and the range of the filtering window, and the size of the gradient less than the gradient threshold is inversely proportional to the size of the range of the filtering window corresponding to the gradient less than the gradient threshold.

[0119] In some embodiments, the detection unit 302 is further configured to perform semantic edge detection on the first image using a semantic edge detector based on machine learning to obtain the confidence that each pixel belongs to a semantic edge; and compare the confidence that each pixel belongs to a semantic edge with the confidence threshold to determine the semantic edge area and non-semantic edge area in the first image.

[0120] In some embodiments, the detection unit 302 is further configured to reduce the size of the first image to a preset size to obtain a reduced first image; input the reduced first image into a semantic edge detector based on machine learning to obtain the confidence that each pixel belongs to a semantic edge.

[0121] In some embodiments, the first image captured by the first camera is an image captured by the first camera in portrait mode, and the second image captured by the second camera is an image captured by the second camera in portrait mode.

[0122] Figure 4 4 is a block diagram of a structure of an electronic device provided by this embodiment. The electronic device includes a processing component 422, which further includes one or more processors, and a memory resource represented by a memory 432, which is used to store instructions executable by the processing component 422, such as an application. The application stored in the memory 432 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 422 is configured to execute instructions to perform the above method.

[0123] The electronic device may also include a power supply component 426 configured to perform power management of the electronic device, a wired or wireless network interface 450 configured to connect the electronic device to a network, and an input / output (I / O) interface 458. The electronic device may operate based on an operating system stored in the memory 432, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

[0124] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0125] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0126] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Performing stereo matching on a first image captured by a first camera and a second image captured by a second camera to obtain a disparity map; Performing semantic edge detection on the first image to determine a semantic edge region and a non-semantic edge region in the first image; Reducing the gradient of a target pixel in the non-semantic edge region, wherein the target pixel is a pixel having a gradient greater than a gradient threshold, and the reduced gradient of the target pixel is less than the gradient threshold; Determining, based on the reduced gradient of the target pixel and the preset corresponding relationship, a range of a filtering window of a disparity map pixel corresponding to the target pixel in the disparity map, wherein the preset corresponding relationship describes a corresponding relationship between a gradient less than a gradient threshold and the range of the filtering window, and a size of the gradient less than the gradient threshold is inversely proportional to a size of the range of the filtering window corresponding to the gradient less than the gradient threshold; According to the range of the filtering window, a preset edge-preserving smoothing filtering algorithm is used to filter the first area corresponding to the semantic edge area in the disparity map and the second area corresponding to the non-semantic edge area in the disparity map.

2. The method according to claim 1, characterized in that Performing semantic edge detection on the first image to determine a semantic edge region and a non-semantic edge region in the first image includes: Performing semantic edge detection on the first image using a semantic edge detector based on machine learning to obtain a confidence level of each pixel belonging to a semantic edge; The confidence of each pixel belonging to the semantic edge is compared with the confidence threshold respectively to determine the semantic edge area and the non-semantic edge area in the first image.

3. The method according to claim 2, characterized in that The semantic edge detector based on machine learning is used to perform semantic edge detection on the first image, and the confidence of each pixel belonging to the semantic edge is obtained, including: Reducing the size of the first image to a preset size to obtain a reduced first image; The reduced first image is input into a semantic edge detector based on machine learning to obtain the confidence that each pixel belongs to a semantic edge.

4. The method according to any one of claims 1 to 3, characterized in that: The first image captured by the first camera is an image captured by the first camera in portrait mode, and the second image captured by the second camera is an image captured by the second camera in portrait mode.

5. An image blurring method, characterized in that: The method comprises: Processing the disparity map to obtain a processed disparity map, wherein the disparity map is obtained by stereo matching a first image captured by a first camera and a second image captured by a second camera, and the processed disparity map is obtained by processing the disparity map according to the method according to any one of claims 1 to 4; The first image is blurred using the processed disparity map to obtain a blurred effect rendered image.

6. An image processing device, characterized in that: The device comprises: A generating unit is configured to perform stereo matching on a first image captured by the first camera and a second image captured by the second camera to obtain a disparity map; A detection unit configured to perform semantic edge detection on the first image to determine a semantic edge region and a non-semantic edge region in the first image; The processing unit is configured to perform edge-preserving smoothing filtering on the disparity map based on the gradient of pixels in the first image, semantic edge regions, and non-semantic edge regions.

7. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 5.

8. A storage medium, when instructions in the storage medium are executed by an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 5.

Images