Background blurring method and device, electronic equipment and storage medium

Abstract

This application discloses a background blurring method, apparatus, electronic device, and storage medium. The method includes: determining a background region in an image to be processed; determining the blurring radius of each pixel based on the depth-of-field value of each pixel in the background region; determining a radius interval based on the blurring radii of all pixels; dividing the radius interval to obtain at least one sub-interval, wherein any two sub-intervals in the at least one sub-interval do not intersect, and the union of the at least one sub-interval is the radius interval; determining a sub-interval corresponding to each pixel in the at least one sub-interval based on the blurring radius of each pixel; determining the blurring pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurring radius of each pixel; and reconstructing the image based on the blurring pixel value of each pixel to obtain a blurred background. Using this method, fully automatic, all-scene, and high-quality digital blurring can be achieved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a background blurring method, apparatus, electronic device, and storage medium. Background Technology

[0002] Due to the size limitations of mobile phones, the focal length and image distance of mobile phone cameras are relatively small. For non-close-up shots of objects, it is usually difficult to create an optical background blur effect similar to that of an SLR camera. Furthermore, most mainstream mobile phones on the market do not have built-in global depth sensors and cannot effectively estimate the depth of mid-range and distant objects through multi-view depth estimation (due to limitations in accuracy and performance). Therefore, many photos and videos taken directly by mobile phones lack the artistic effect of background blur and can only achieve digital blurring through post-processing of the camera and image / video editing stages.

[0003] This digital blurring is based on monocular depth estimation and uses physical formulas to simulate optical blurring effects in photographs.

[0004] Currently, monocular digital bokeh technology has been applied in some mobile phone models and image editing software. Some are based on image segmentation, while others are based on single-image depth estimation. However, image segmentation-based bokeh technology only distinguishes between foreground and background, without considering depth variations in the background. It blurs all areas of the background to the same degree, resulting in unnatural and poor-quality final results. While single-image depth estimation-based bokeh technology can consider depth variations in the background and blur different areas of the background to different degrees based on depth, it is only used when the image has a specific foreground, such as a person or a pre-defined object, or only supports semi-automatic bokeh (requiring manual selection of the focal plane). Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this application provides a background blurring method, apparatus, electronic device, and storage medium that can achieve fully automatic, full-scene, and high-quality digital blurring.

[0006] In a first aspect, embodiments of this application provide a background blurring method, the method comprising:

[0007] Identify the background region in the image to be processed;

[0008] The blur radius of each pixel is determined based on the depth value of each pixel in the background region.

[0009] Determine the radius range based on the blur radius of all pixels;

[0010] Divide the radius interval to obtain at least one subinterval, wherein any two subintervals in the at least one subinterval do not intersect, and the union of the at least one subinterval is the radius interval;

[0011] Based on the blur radius of each pixel, determine the sub-interval corresponding to each pixel in at least one sub-interval;

[0012] The blurred pixel value of each pixel is determined based on the sub-interval corresponding to each pixel and the blurred radius of each pixel.

[0013] Based on the blurred pixel value of each pixel, the image is reconstructed to obtain the blurred background.

[0014] In one possible implementation, the blurred pixel value of each pixel is determined based on the sub-interval corresponding to each pixel and the blurred radius of each pixel, including:

[0015] The endpoint value of the left endpoint of the sub-interval corresponding to each pixel is used as the first blur radius. Each pixel is then blurred to obtain the first blurred pixel value.

[0016] The endpoint value of the right endpoint of the sub-interval corresponding to each pixel is used as the second blur radius. Each pixel is then blurred to obtain the second blur pixel value.

[0017] The stacking coefficient is determined based on the endpoint values ​​of the two endpoints of the sub-interval corresponding to each pixel and the blur radius of each pixel.

[0018] The first and second blurred pixel values ​​are superimposed based on the superposition coefficient to obtain the blurred pixel value of each pixel.

[0019] In one possible implementation, the superposition coefficient is expressed by formula (1):

[0020]

[0021] Where m is the stacking coefficient, R is the blur radius of each pixel, and a l The endpoint value of the left endpoint of the sub-interval corresponding to each pixel, a r The endpoint value of the right endpoint of the sub-interval corresponding to each pixel.

[0022] In one possible implementation, the blurred pixel value of each pixel is represented by formula (2):

[0023] o=(1-m)×b l +m×b r ………(2)

[0024] Where 'o' is the blurred pixel value for each pixel, and 'b' is the blurred pixel value for each pixel.l b is the value of the first blurred pixel. r This is the value of the second blurred pixel.

[0025] In one possible implementation, the blur radius of each pixel is expressed by formula (3):

[0026]

[0027] Where s represents the hyperparameter, c represents the depth value of each pixel, D represents the median depth value of all pixels in the foreground region of the image to be processed, and W represents the difference between the maximum and minimum depth values ​​of all pixels in the foreground region.

[0028] In one possible implementation, the radius range is determined based on the blur radius of all pixels, including:

[0029] Determine the maximum and minimum values ​​of the blur radius for all pixels;

[0030] Determine the radius interval by taking the minimum value as the left endpoint of the interval and the maximum value as the right endpoint of the interval.

[0031] In one possible implementation, the radius interval is divided to obtain at least one sub-interval, including:

[0032] The radius interval is divided equally according to a preset number to obtain at least one sub-interval, wherein the number of at least one sub-interval is equal to the preset number.

[0033] Secondly, embodiments of this application provide a background blurring device, comprising:

[0034] The region determination module is used to determine the background region in the image to be processed;

[0035] The parameter determination module is used to determine the blur radius of each pixel based on the depth value of each pixel in the background region, determine the radius interval based on the blur radius of all pixels, and divide the radius interval to obtain at least one sub-interval, wherein any two sub-intervals in the at least one sub-interval do not intersect, and the union of the at least one sub-interval is the radius interval.

[0036] The blurring module is used to determine the sub-interval corresponding to each pixel in at least one sub-interval based on the blurring radius of each pixel, and to determine the blurring pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurring radius of each pixel.

[0037] The reconstruction module is used to reconstruct the image based on the blurred pixel value of each pixel to obtain a blurred background.

[0038] Thirdly, embodiments of this application provide an electronic device, including: a processor connected to a memory for storing a computer program, and the processor for executing the computer program stored in the memory to cause the electronic device to perform the method as described in the first aspect.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that causes a computer to perform the method as described in the first aspect.

[0040] Fifthly, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, and a computer operable to perform the method as described in the first aspect.

[0041] Implementing the embodiments of this application has the following beneficial effects:

[0042] In this embodiment, the blur radius of each pixel is determined based on its depth value, thus defining the range of all blur radii, i.e., the radius interval. The radius interval is then divided into at least one sub-interval, and the corresponding pixels for each sub-interval are identified. Finally, the blur pixel value of each pixel is determined based on its corresponding sub-interval and its blur radius. The image is then reconstructed based on the blur pixel value of each pixel to obtain a blurred background. Therefore, without being limited by specific scenes or themes, the background region in the image to be processed can be automatically identified, and different degrees of blurring can be applied to different regions of the background region based on the depth of field. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 A schematic diagram of the hardware structure of a background blurring device provided for an embodiment of this application;

[0045] Figure 2 A schematic flowchart illustrating a background blurring method provided for an embodiment of this application;

[0046] Figure 3 A flowchart illustrating a method for determining the blurred pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurred radius of each pixel, provided for an embodiment of this application;

[0047] Figure 4 A functional module block diagram of a background blurring device provided for embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the structure of an electronic device provided for an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0051] In this document, the term "implementation" means that a specific feature, result, or characteristic described in connection with an implementation may be included in at least one implementation of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same implementation, nor is it a separate or alternative implementation mutually exclusive with other implementations. It will be explicitly and implicitly understood by those skilled in the art that the implementations described herein can be combined with other implementations.

[0052] See Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of a background blurring device provided in an embodiment of this application. The background blurring device 100 includes at least one processor 101, a communication line 102, a memory 103, and at least one communication interface 104.

[0053] In this embodiment, the processor 101 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0054] Communication line 102 may include a path for transmitting information between the aforementioned components.

[0055] The communication interface 104 can be any transceiver-like device (such as an antenna) used to communicate with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.

[0056] The memory 103 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0057] In this embodiment, the memory 103 can exist independently and be connected to the processor 101 via the communication line 102. Alternatively, the memory 103 can be integrated with the processor 101. The memory 103 provided in this embodiment is typically non-volatile. The memory 103 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 101. The processor 101 executes the computer execution instructions stored in the memory 103 to implement the method provided in the following embodiments of this application.

[0058] In an optional implementation, the computer execution instructions may also be referred to as application code, and this application does not specifically limit this terminology.

[0059] In an optional implementation, processor 101 may include one or more CPUs, for example... Figure 1 CPU0 and CPU1 in the CPU.

[0060] In an optional implementation, the background blurring device 100 may include multiple processors, such as... Figure 1Processors 101 and 107 are shown in the diagram. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0061] In optional embodiments, if the background blurring device 100 is a server, for example, it can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, then the background blurring device 100 may further include an output device 105 and an input device 106. The output device 105 communicates with the processor 101 and can display information in various ways. For example, the output device 105 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 106 communicates with the processor 101 and can receive user input in various ways. For example, the input device 106 can be a mouse, keyboard, touch screen device, or sensing device, etc.

[0062] The background blurring device 100 described above can be a general-purpose device or a special-purpose device. The embodiments of this application do not limit the type of background blurring device 100.

[0063] The following will describe a background blurring method disclosed in this application, using pixels in the image to be processed as the processing object:

[0064] See Figure 2 , Figure 2 This is a flowchart illustrating a background blurring method provided for an embodiment of this application. The background blurring method includes the following steps:

[0065] 201: Identify the background region in the image to be processed.

[0066] In this embodiment, the background region in the image to be processed can be determined using a depth map combined with a foreground mask. For example, an end-to-end neural network, such as a Transformer neural network, can be used to obtain the depth map of the image. Here, the depth map refers to a value assigned to each pixel after neural processing. This value represents the depth information of that pixel, i.e., the depth value, such as absolute depth, relative depth, or relative disparity. Furthermore, the Transformer's structure is superior to that of a CNN neural network. Compared to CNN networks, the Transformer has a larger receptive field, considering both the correlation between local pixels and integrating global information. For depth estimation tasks, the ability to synthesize global information is crucial.

[0067] In this embodiment, the disparity value, which is the reciprocal of the depth value, is used as the depth value in the depth map. Dense depth prediction is employed, meaning that a disparity value is predicted and output for each pixel to obtain the depth map. The disparity value is chosen because the statistical distribution of disparity values ​​in a single image is usually relatively uniform; that is, the number of pixels is relatively evenly distributed within several equal intervals from the minimum to the maximum value. In contrast, the depth value varies greatly from minimum to maximum (usually differing by several orders of magnitude) and is unevenly distributed. Typically, the depth values ​​of most pixels fall near the minimum or maximum value. Therefore, choosing the disparity value as the depth information in the depth map is more favorable for neural networks and can achieve more accurate depth prediction.

[0068] In this embodiment, a lightweight binary classification neural network, such as ResNet18, ResNet50, ResNet101, MobileNet v1 / v2 / v3, can be used to identify whether a subject exists in the image to be processed. If a subject exists, a subject detection / segmentation neural network, such as Mask RCNN, YOLO-V5 / v6 / v7, is used to perform mask segmentation on the subject region identified by the lightweight binary classification neural network, obtaining the subject mask as the foreground mask. When multiple subjects are identified, the mask of each subject can be output separately, and then the masks of each subject can be merged to obtain the foreground mask.

[0069] In this embodiment, although two neural networks are used to obtain the foreground mask, the first neural network, the lightweight binary classification neural network, is lightweight and performs binary classification on the image. The input image can be resized to a relatively small size (typically between 224×224 and 256×256), so the actual time consumption for this part is very small. Furthermore, only a portion of the images require the second step. Specifically, in this embodiment, instead of directly outputting the mask end-to-end, the subject bounding box is detected indirectly and the subject is segmented separately. Although this increases complexity slightly, it yields more accurate and robust results, and provides better interpretability.

[0070] After obtaining the depth map and the foreground mask, the depth values ​​of each pixel corresponding to the foreground mask region can be determined based on the depth map, thus obtaining the depth value range of the foreground mask. For example, the maximum and minimum depth values ​​of each pixel corresponding to the foreground mask region can be determined, and then the minimum value can be used as the left endpoint of the range, and the maximum value as the right endpoint, to determine the depth value range. Specifically, if the maximum depth value of each pixel corresponding to the foreground mask region is 0.9 and the minimum is 0.05, then the depth value range is [0.05, 0.9].

[0071] In this embodiment, after determining the depth-of-field value range, the region consisting of pixels in the image to be processed whose depth-of-field values ​​fall within that range can be considered the foreground region, or focal plane. The remaining regions, or the regions consisting of pixels in the image to be processed whose depth-of-field values ​​do not fall within that range, are the background regions. Therefore, this embodiment can achieve automatic foreground and background segmentation in any scene and subject.

[0072] 202: Determine the blur radius of each pixel based on the depth value of each pixel in the background region.

[0073] In this embodiment, the bokeh radius of each pixel can be expressed by formula (4):

[0074]

[0075] Where s represents a hyperparameter used to control the overall blur level. Generally, the larger s is, the more blurred the background is, and the smaller s is, the less blurred the background is. c represents the depth value of each pixel, D represents the median depth value of all pixels in the foreground region of the image to be processed, and W represents the difference between the maximum and minimum depth values ​​of all pixels in the foreground region.

[0076] In this regard, D can be expressed by formula (5):

[0077]

[0078] W can be expressed by formula (6):

[0079] W=c MAX -c MIN ………(6)

[0080] Among them, c MIN c is the minimum depth value among all pixels in the foreground region. MAX It is the maximum depth value among all pixels in the foreground region.

[0081] 203: Determine the radius range based on the blur radius of all pixels.

[0082] In this embodiment, the maximum and minimum values ​​of the blur radius of all pixels can be determined. Then, the minimum value is used as the left endpoint of the interval, and the maximum value is used as the right endpoint of the interval to determine the radius interval. Specifically, if the maximum value of the blur radius of all pixels is 10 and the minimum value is 0, then the radius interval is [0, 10].

[0083] 204: Divide the radius interval to obtain at least one sub-interval.

[0084] In this embodiment, any two sub-intervals within at least one sub-interval do not intersect, and the union of at least one sub-interval is the radius interval. For example, the radius interval can be divided equally according to a preset number to obtain at least one sub-interval, wherein the number of at least one sub-interval is equal to the preset number. Specifically, assuming the preset number is 10, following the example of the radius interval [0, 10], after dividing it into 10 equal parts, 10 sub-intervals can be obtained: [0, 1], (1, 2], (2, 3], (3, 4], (4, 5], (5, 6], (6, 7], (7, 8], (8, 9], and (9, 10). Of course, the preset number can also be other values; preferably, its range is 5-20.

[0085] 205: Based on the blur radius of each pixel, determine the sub-interval corresponding to each pixel in at least one sub-interval.

[0086] In this embodiment, the sub-interval in which the bokeh radius of each pixel is located can be taken as its corresponding sub-interval. For example, if the bokeh radius of a certain pixel is 6.6, then it is in the sub-interval (6, 7], and its corresponding sub-interval is (6, 7).

[0087] 206: Determine the blurred pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurred radius of each pixel.

[0088] In this embodiment, a method is provided to determine the blurred pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurred radius of each pixel, such as... Figure 3 As shown, the method includes:

[0089] 301: Take the endpoint value of the left endpoint of the sub-interval corresponding to each pixel as the first blur radius, and perform blur processing on each pixel to obtain the first blur pixel value.

[0090] Specifically, using the example of a pixel with a blur radius of 6.6, its corresponding sub-interval is (6, 7], and the first blur radius is the left endpoint value 6 of this sub-interval. In this embodiment, the blurring process can employ Gaussian blur. In short, the first blur radius 6 is used as the Gaussian blur radius, and the pixel is Gaussian blurred using a Gaussian kernel with a radius of 6 to obtain the first blurred pixel value.

[0091] 302: Use the right endpoint value of the sub-interval corresponding to each pixel as the second blur radius, and perform blur processing on each pixel to obtain the second blur pixel value.

[0092] Similarly, using the example of a pixel with a blur radius of 6.6, its corresponding sub-interval is (6, 7], and the second blur radius is the right endpoint value 7 of this sub-interval. This blurring process is similar to the blurring process in step 301, and will not be described again here.

[0093] 303: Determine the stacking coefficient based on the endpoint values ​​of the two endpoints of the sub-interval corresponding to each pixel and the blur radius of each pixel.

[0094] In this embodiment, the superposition coefficient can be expressed by the following formula (7):

[0095]

[0096] Where m is the stacking coefficient, R is the blur radius of each pixel, and a l The endpoint value of the left endpoint of the sub-interval corresponding to each pixel, a r The endpoint value of the right endpoint of the sub-interval corresponding to each pixel.

[0097] Specifically, using the example of a pixel with a blur radius of 6.6, the corresponding stacking coefficient is:

[0098]

[0099] 304: The first and second blurred pixel values ​​are superimposed according to the superposition coefficient to obtain the blurred pixel value of each pixel.

[0100] In this embodiment, the blurred pixel value of each pixel can be represented by formula (8):

[0101] o=(1-m)×b l +m×b r ………(8)

[0102] Where 'o' is the blurred pixel value for each pixel, and 'b' is the blurred pixel value for each pixel. l b is the value of the first blurred pixel. r This is the value of the second blurred pixel.

[0103] 207: Based on the blurred pixel value of each pixel, the image is reconstructed to obtain the blurred background.

[0104] In this embodiment, the blurred pixel value of each pixel in the background region can be used as the final pixel value, while the pixel value of each pixel in the foreground region remains unchanged. Then, the image is reconstructed to obtain an image with blurred background.

[0105] The following will describe a background blurring method disclosed in this application, taking the entire image to be processed as the processing object:

[0106] First, a Transformer-based end-to-end neural network is used to perform dense depth estimation on the image to be processed, resulting in a depth map. Disparity values ​​are used as depth values ​​in the depth map output to improve the accuracy of the neural network prediction.

[0107] Then, the image to be processed is input into lightweight binary classification neural networks such as ResNet18, ResNet50, ResNet101, and MobileNet v1 / v2 / v3 for subject recognition. If the subject is identified, the image to be processed with the identified subject is input into subject detection / segmentation networks such as Mask RCNN, YOLO-V5 / v6 / v7 to obtain a foreground mask. Based on the region of the foreground mask corresponding to the depth map, the depth value range of the foreground mask is obtained, and thus the background region of the image to be processed is obtained. Specifically, pixels belonging to the depth value range belong to the foreground region, and those not belonging to this range belong to the background region.

[0108] Then, the Gaussian blur radius R corresponding to each pixel in the background area is calculated to obtain the radius range. The background image is then divided into n equal layers based on the radius range, where n is typically between 5 and 20. The layers are arranged in ascending order of depth value as R0, R1, ..., Rn. For example, with an R value radius range of [0, 10], it can be divided into 10 layers: [0, 1], (1, 2], (2, 3], (3, 4], (4, 5], (5, 6], (6, 7], (7, 8], (8, 9], and (9, 10], where R0 is the set of pixels with R values ​​between [0, 1].

[0109] Then, the endpoint values ​​of each of the n layers are used as Gaussian blur radii to blur the image, resulting in (n+1) blurred images. For the example above, 11 endpoint values ​​(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10) are used as Gaussian blur radii to blur the image, resulting in 11 images I0, I1, I2, ..., I11 with different degrees of blur.

[0110] Then, for each pixel in the layer, the final blurred image of each layer is synthesized using the two adjacent blur images. Specifically, for layer R5, the corresponding R value range is (5, 6], so its two adjacent blur images are I5 ​​and I6, which are the images obtained by Gaussian blurring the image to be processed using the two endpoints of the range (5, 6). After determining the two adjacent blur images, the superposition coefficient of each pixel in layer R5 is calculated and denoted as m5. The two adjacent blur images I5 and I6 are synthesized using formula (9) to obtain the final blurred image O5 of layer R5.

[0111] O5=(1-m5)×I5+m5×I6………(9)

[0112] Using a similar method, the final blurred images O0, O1, ..., O9 of other layers can be obtained.

[0113] Finally, by merging the final blurred images of the 10 layers O0, O1, ..., O9 onto the same image, the final blurred image O of the image to be processed can be obtained.

[0114] In summary, the background blurring method provided by this invention determines the blurring radius of each pixel based on its depth-of-field value, thereby determining the range of all blurring radii, i.e., the radius interval. Then, the radius interval is divided to obtain at least one sub-interval, and the corresponding pixels for each sub-interval are determined. Finally, based on the sub-interval corresponding to each pixel and the blurring radius of each pixel, the blurring pixel value of each pixel is determined. Subsequently, based on the blurring pixel value of each pixel, image reconstruction is performed to obtain a blurred background. Therefore, it can automatically identify the background region in the image to be processed without being limited by specific scenes or themes, and automatically blur different regions of the background region to different degrees based on the depth of field.

[0115] See Figure 4 , Figure 4 This is a block diagram illustrating the functional modules of a background blurring device provided for an embodiment of this application. For example... Figure 4 As shown, the background blurring device 400 includes:

[0116] The region determination module 401 is used to determine the background region in the image to be processed;

[0117] The parameter determination module 402 is used to determine the blur radius of each pixel based on the depth value of each pixel in the background region, determine the radius interval based on the blur radius of all pixels, and divide the radius interval to obtain at least one sub-interval, wherein any two sub-intervals in the at least one sub-interval do not intersect, and the union of the at least one sub-interval is the radius interval.

[0118] The blurring module 403 is used to determine the sub-interval corresponding to each pixel in at least one sub-interval based on the blurring radius of each pixel, and to determine the blurring pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurring radius of each pixel.

[0119] The reconstruction module 404 is used to reconstruct the image based on the blurred pixel value of each pixel to obtain a blurred background.

[0120] In an embodiment of the present invention, in determining the blurred pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurred radius of each pixel, the blurred module 403 is specifically used for:

[0121] The endpoint value of the left endpoint of the sub-interval corresponding to each pixel is used as the first blur radius. Each pixel is then blurred to obtain the first blurred pixel value.

[0122] The endpoint value of the right endpoint of the sub-interval corresponding to each pixel is used as the second blur radius. Each pixel is then blurred to obtain the second blur pixel value.

[0123] The stacking coefficient is determined based on the endpoint values ​​of the two endpoints of the sub-interval corresponding to each pixel and the blur radius of each pixel.

[0124] The first and second blurred pixel values ​​are superimposed based on the superposition coefficient to obtain the blurred pixel value of each pixel.

[0125] In an embodiment of the present invention, the superposition coefficient can be expressed by formula (10):

[0126]

[0127] Where m is the stacking coefficient, R is the blur radius of each pixel, and a l The endpoint value of the left endpoint of the sub-interval corresponding to each pixel, a r The endpoint value of the right endpoint of the sub-interval corresponding to each pixel.

[0128] In an embodiment of the present invention, the blurred pixel value of each pixel can be represented by formula (11):

[0129] o=(1-m)×b l +m×b r ………(11)

[0130] Where 'o' is the blurred pixel value for each pixel, and 'b' is the blurred pixel value for each pixel. l b is the value of the first blurred pixel. r This is the value of the second blurred pixel.

[0131] In an embodiment of the present invention, the bokeh radius of each pixel can be expressed by formula (12):

[0132]

[0133] Where s represents the hyperparameter, c represents the depth value of each pixel, D represents the median depth value of all pixels in the foreground region of the image to be processed, and W represents the difference between the maximum and minimum depth values ​​of all pixels in the foreground region.

[0134] In an embodiment of the present invention, the parameter determination module 402, in determining the radius range based on the bokeh radius of all pixels, is specifically used for:

[0135] Determine the maximum and minimum values ​​of the blur radius for all pixels;

[0136] Determine the radius interval by taking the minimum value as the left endpoint of the interval and the maximum value as the right endpoint of the interval.

[0137] In an embodiment of the present invention, in dividing the radius interval to obtain at least one sub-interval, the parameter determination module 402 is specifically used for:

[0138] The radius interval is divided equally according to a preset number to obtain at least one sub-interval, wherein the number of at least one sub-interval is equal to the preset number.

[0139] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided for an embodiment of this application. For example... Figure 5 As shown, the electronic device 500 includes a transceiver 501, a processor 502, and a memory 503. These are connected via a bus 504. The memory 503 stores computer programs and data, and can transfer data stored in the memory 503 to the processor 502.

[0140] Processor 502 is used to read the computer program in memory 503 and perform the following operations:

[0141] Identify the background region in the image to be processed;

[0142] The blur radius of each pixel is determined based on the depth value of each pixel in the background region.

[0143] Determine the radius range based on the blur radius of all pixels;

[0144] Divide the radius interval to obtain at least one subinterval, wherein any two subintervals in the at least one subinterval do not intersect, and the union of the at least one subinterval is the radius interval;

[0145] Based on the blur radius of each pixel, determine the sub-interval corresponding to each pixel in at least one sub-interval;

[0146] The blurred pixel value of each pixel is determined based on the sub-interval corresponding to each pixel and the blurred radius of each pixel.

[0147] Based on the blurred pixel value of each pixel, the image is reconstructed to obtain the blurred background.

[0148] In an embodiment of the present invention, in determining the blurred pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurred radius of each pixel, the processor 502 is specifically configured to perform the following operations:

[0149] The endpoint value of the left endpoint of the sub-interval corresponding to each pixel is used as the first blur radius. Each pixel is then blurred to obtain the first blurred pixel value.

[0150] The endpoint value of the right endpoint of the sub-interval corresponding to each pixel is used as the second blur radius. Each pixel is then blurred to obtain the second blur pixel value.

[0151] The stacking coefficient is determined based on the endpoint values ​​of the two endpoints of the sub-interval corresponding to each pixel and the blur radius of each pixel.

[0152] The first and second blurred pixel values ​​are superimposed based on the superposition coefficient to obtain the blurred pixel value of each pixel.

[0153] In an embodiment of the present invention, the superposition coefficient can be expressed by formula (13):

[0154]

[0155] Where m is the stacking coefficient, R is the blur radius of each pixel, and a l The endpoint value of the left endpoint of the sub-interval corresponding to each pixel, a r The endpoint value of the right endpoint of the sub-interval corresponding to each pixel.

[0156] In an embodiment of the present invention, the blurred pixel value of each pixel can be represented by formula (14):

[0157] o=(1-m)×b l +m×b r ………(14)

[0158] Where 'o' is the blurred pixel value for each pixel, and 'b' is the blurred pixel value for each pixel. l b is the value of the first blurred pixel. r This is the value of the second blurred pixel.

[0159] In an embodiment of the present invention, the bokeh radius of each pixel can be expressed by formula (15):

[0160]

[0161] Where s represents the hyperparameter, c represents the depth value of each pixel, D represents the median depth value of all pixels in the foreground region of the image to be processed, and W represents the difference between the maximum and minimum depth values ​​of all pixels in the foreground region.

[0162] In an embodiment of the present invention, in determining the radius range based on the bokeh radius of all pixels, the processor 502 is specifically configured to perform the following operations:

[0163] Determine the maximum and minimum values ​​of the blur radius for all pixels;

[0164] Determine the radius interval by taking the minimum value as the left endpoint of the interval and the maximum value as the right endpoint of the interval.

[0165] In an embodiment of the present invention, in dividing a radius interval to obtain at least one sub-interval, the processor 502 is specifically configured to perform the following operations:

[0166] The radius interval is divided equally according to a preset number to obtain at least one sub-interval, wherein the number of at least one sub-interval is equal to the preset number.

[0167] It should be understood that the background blurring device in this application may include smartphones (such as Android phones, iOS phones, Windows Phones, etc.), tablet computers, PDAs, laptops, mobile internet devices (MIDs), robots, or wearable devices, etc. The above-mentioned background blurring devices are merely examples and not exhaustive, and include, but are not limited to, the background blurring devices described above. In practical applications, the above-mentioned background blurring devices may also include: intelligent in-vehicle terminals, computer equipment, etc.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software combined with a hardware platform. Based on this understanding, all or part of the technical solution of the present invention that contributes to the background art can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0169] Therefore, embodiments of this application also provide a computer-readable storage medium storing a computer program that is executed by a processor to implement some or all of the steps of any of the background blurring methods described in the above method embodiments. For example, the storage medium may include a hard disk, floppy disk, optical disk, magnetic tape, magnetic disk, USB flash drive, flash memory, etc.

[0170] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the background blurring methods described in the above method embodiments.

[0171] It should be noted that, for the sake of simplicity, the aforementioned methods are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are optional, and the actions and modules involved are not necessarily essential to this application.

[0172] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0176] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0177] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0178] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A background blurring method, characterized in that, The method includes: Identify the background region in the image to be processed; The blur radius of each pixel is determined based on the depth value of each pixel in the background region, the median depth value of all pixels in the foreground region of the image to be processed, and the difference between the maximum and minimum depth values ​​of all pixels in the foreground region. Determine the radius range based on the blur radius of all the pixels; The radius interval is divided to obtain at least one sub-interval, wherein any two sub-intervals in the at least one sub-interval do not intersect, and the union of the at least one sub-interval is the radius interval; Based on the blur radius of each pixel, a sub-interval corresponding to each pixel is determined in the at least one sub-interval; The blurred pixel value of each pixel is determined based on the sub-interval corresponding to each pixel and the blurred radius of each pixel; Based on the blurred pixel value of each pixel, the image is reconstructed to obtain a blurred background; The blur radius of each pixel is determined by the following formula: Where s represents the hyperparameter, c represents the depth value of each pixel, D represents the median depth value of all pixels in the foreground region of the image to be processed, and W represents the difference between the maximum and minimum depth values ​​of all pixels in the foreground region.

2. The method according to claim 1, characterized in that, The step of determining the blurred pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurred radius of each pixel includes: The endpoint value of the left endpoint of the sub-interval corresponding to each pixel is used as the first blur radius to perform blurring processing on each pixel to obtain the first blur pixel value; The endpoint value of the right endpoint of the sub-interval corresponding to each pixel is used as the second blur radius to perform blurring processing on each pixel, thereby obtaining the second blur pixel value; The superposition coefficient is determined based on the endpoint values ​​of the two endpoints of the sub-interval corresponding to each pixel and the blur radius of each pixel; The first blurred pixel value and the second blurred pixel value are superimposed according to the superposition coefficient to obtain the blurred pixel value of each pixel.

3. The method according to claim 2, characterized in that, The superposition coefficient is determined by the following formula: Where m is the superposition coefficient, R is the blur radius of each pixel, and a l a is the endpoint value of the left endpoint of the sub-interval corresponding to each pixel. r The endpoint value is the right endpoint of the sub-interval corresponding to each pixel.

4. The method according to claim 2 or 3, characterized in that, The blurred pixel value of each pixel is determined by the following formula: Where o is the blurred pixel value of each pixel, b l b is the value of the first blurred pixel. r This is the value of the second blurred pixel.

5. The method according to any one of claims 1-3, characterized in that, The step of determining the radius range based on the blur radius of all pixels includes: Determine the maximum and minimum values ​​of the blur radius for all the pixels; The minimum value is used as the left endpoint of the interval, and the maximum value is used as the right endpoint of the interval to determine the radius interval.

6. The method according to any one of claims 1-3, characterized in that, The process of dividing the radius interval to obtain at least one sub-interval includes: The radius interval is divided equally according to a preset number to obtain at least one sub-interval, wherein the number of the at least one sub-interval is equal to the preset number.

7. A background blurring device, characterized in that, The device includes: The region determination module is used to determine the background region in the image to be processed; The parameter determination module is used to determine the blur radius of each pixel based on the depth value of each pixel in the background region, the median depth value of all pixels in the foreground region of the image to be processed, and the difference between the maximum and minimum depth values ​​of all pixels in the foreground region. Based on the blur radius of all pixels, a radius interval is determined, and the radius interval is divided to obtain at least one sub-interval. Any two sub-intervals in the at least one sub-interval do not intersect, and the union of the at least one sub-interval is the radius interval. The blur radius of each pixel is determined by the following formula: Where s represents the hyperparameter, c represents the depth value of each pixel, D represents the median of the depth values ​​of all pixels in the foreground region of the image to be processed, and W represents the difference between the maximum and minimum depth values ​​of all pixels in the foreground region. A blurring module is used to determine the sub-interval corresponding to each pixel in the at least one sub-interval based on the blurring radius of each pixel, and to determine the blurring pixel value of each pixel based on the sub-interval corresponding to each pixel and the blurring radius of each pixel. The reconstruction module is used to reconstruct the image based on the blurred pixel value of each pixel to obtain a blurred background.

8. An electronic device, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the one or more programs include instructions for performing the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1-6.

Images