A method, device, storage medium and electronic device for blurring a target object

Abstract

The embodiments of the present application provide a method, device, storage medium and electronic device for blurring a target object. The method for blurring a target object includes: acquiring an image to be blurred; obtaining a segmentation map and a background segmentation map of the target object based on the image; performing light spot detection on the background segmentation map to obtain a light spot detection map, and obtaining a light spot weight map based on the light spot detection map; obtaining a background light spot blur rendering map based on the light spot weight map; fusing the background light spot blur rendering map with the segmentation map of the target object to obtain a target object light spot blur rendering map. The method for blurring a target object in some embodiments of the present application can make the blurring effect more natural, the light spot clearer, and the simulated light spot effect close to the shooting effect of a SLR.

Description

Technical Field

[0001] The present application relates to the field of image blurring. Specifically, embodiments of the present application relate to a method, device, storage medium, and electronic device for blurring a target object. Background Art

[0002] With the update and iteration of electronic devices (for example, mobile electronic devices such as mobile phones), users have an increasing demand for the camera function of electronic devices and higher requirements for the quality of the captured images. For example, when users use the portrait blur shooting mode on their mobile phones to shoot images, they require that the captured images have a high-quality blur effect.

[0003] In the related art, the portrait video or image blurring technology based on mobile terminals (e.g., mobile phones and other electronic devices) uses computer vision algorithms to render the captured images, so that the rendered images can achieve better portrait blurring shooting effects. However, the portrait blurring effect (or blurring effect on other items or objects) on mobile phones and other electronic devices currently has many technical defects, such as the unsightly portrait blurring effect obtained by using computer algorithms, and the blurred and unclear light spots on the blurred images.

[0004] Therefore, how to improve the blurring effect of electronic devices on images has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a target object blurring method, device, storage medium and electronic device. Through the target object blurring method of some embodiments of the present application, the blurring effect of the target object image output by the electronic device can be made more natural and the light spot clearer, so that the simulated light spot effect after blurring is close to the shooting effect of a SLR.

[0006] In a first aspect, some embodiments of the present application provide a method for blurring a target object, the method comprising: acquiring an image to be blurred, wherein the image comprises a target object; obtaining a segmentation map and a background segmentation map of the target object based on the image; performing light spot detection on the background segmentation map to obtain a light spot detection map, and obtaining a light spot weight map based on the light spot detection map, wherein the light spot detection map comprises light spots and non-light spots, and the light spot weight map is used to characterize the weight value of any light spot and the weight value of non-light spots, and the weight value of any light spot is greater than the weight values ​​of each non-light spot in a neighborhood corresponding to the any light spot; obtaining a background light spot blurred rendering map based on the light spot weight map; and fusing the background light spot blurred rendering map and the segmentation map of the target object to obtain a target object light spot blurred rendering map.

[0007] Some embodiments of the present application perform spot rendering on the background segmentation map based on the acquired spot weight map, thereby improving the spot display effect of the target object spot blurred rendering map, making the spot on the image clearer, more transparent and brighter, and generating an obvious spot shape with the spot shooting effect of a SLR.

[0008] In some embodiments, performing light spot detection on the background segmentation map to obtain the light spot detection map includes: obtaining a single-channel map based on the maximum value of multiple channels for each pixel in the background segmentation map; performing light spot detection on the single-channel map to obtain the light spot detection map, wherein the pixel values ​​of the pixels on the light spot detection map corresponding to the light spots are the pixel values ​​of the corresponding pixels on the single-channel map, and the pixel values ​​of the pixels on the light spot detection map corresponding to the non-light spots are 0.

[0009] Some embodiments of the present application obtain a single-channel image by selecting the maximum value of each pixel point from multiple channels, and then perform point screening detection of light spots based on the single-channel image, which can improve the accuracy of the obtained candidate light spots.

[0010] In some embodiments, the light spot detection is performed on the single-channel image to obtain the light spot detection image, including: if the attribute value of any pixel point on the single-channel image is greater than or equal to a first threshold, and the attribute value of each pixel point in the neighborhood corresponding to the any pixel point on the single-channel image is less than or equal to a second threshold, then confirming that the any pixel point belongs to the detected light spot, wherein the second threshold is less than the first threshold, and the attribute value includes a pixel value or a normalized pixel value, and the normalized pixel value is the value of each pixel point obtained after normalizing the pixel values ​​of all pixels on the entire image.

[0011] Some embodiments of the present application search for candidate light spots (i.e., the center point of the display light spot pattern) from a single-channel image based on the light spot properties (i.e., the pixel value or normalized pixel value located at the center of the display light spot (i.e., the light spot) is large, while the pixel value or normalized pixel value of the point located at the edge of the display light spot is smaller than the center value), thereby improving the accuracy of the obtained light spot.

[0012] In some embodiments, the light spot detection is performed on the single-channel image to obtain the light spot detection image, including: blurring the pixel value of each pixel on the single-channel image to obtain a blurred single-channel image; if the attribute value of any pixel on the single-channel image is greater than a third threshold, and the attribute value of any pixel on the blurred single-channel image is less than a fourth threshold, then confirming that the any pixel belongs to the detected light spot, wherein the third threshold is greater than the fourth threshold, the attribute value includes a pixel value or a normalized pixel value, and the normalized pixel value is the value of each pixel obtained by normalizing the pixel values ​​of all pixels on the entire image.

[0013] Some embodiments of the present application first blur the single-channel image, and then search for light spots on the light spot detection image based on the weight value (i.e., normalized pixel value) or size attribute of the pixel value of the same pixel point on the blurred image (i.e., blurred single-channel image) and the non-blurred image (i.e., single-channel image), thereby improving the accuracy of the obtained candidate light spots and the speed of data processing.

[0014] In some embodiments, obtaining the light spot weight map according to the light spot detection map includes: performing a thinning operation on the light spot detection map to obtain a thinned light spot detection map; and obtaining the light spot weight map according to the thinned light spot detection map.

[0015] Some embodiments of the present application will thin out the entire light spot detection map obtained to prevent the problem of overexposure and lack of layering in large areas of light spot clusters in subsequent rendering, thereby effectively improving the display effect of the light spots on the blurred image.

[0016] In some embodiments, the thinning operation performed on the light spot detection map includes: performing a thinning operation on a dense area included in the light spot detection map.

[0017] Some embodiments of the present application thin out the areas where light spots are densely clustered on the light spot detection map. On the one hand, this can prevent the problem of overexposure and lack of layering in large areas where light spots are clustered in subsequent rendering, effectively improving the display effect of light spots on the blurred image. On the other hand, it can reduce the amount of calculation and improve the data processing speed.

[0018] In some embodiments, before performing a thinning operation on a dense area included in the light spot detection image, the target object blurring method also includes: dividing the light spot detection image into multiple sub-images; obtaining the coordinates of the center of mass of each sub-image in the multiple sub-images; taking each center of mass as the center on the light spot detection image, solving the total number of non-zero value pixel points included in a neighborhood corresponding to each center; and determining the dense area from the multiple sub-images based on the total number.

[0019] Some embodiments of the present application confirm one or more dense areas included in the light spot detection image according to a centroid-related algorithm, and then only perform a thinning operation on the dense areas, which can improve the processing speed of the thinning operation.

[0020] In some embodiments, obtaining the light spot weight map according to the sparse light spot detection map includes: performing an operation of increasing pixel values ​​or normalized pixel values ​​on at least discrete sparse points on the sparse light spot detection map according to the light spot detection map to obtain an isolated point-enlarged light spot detection map; and obtaining the light spot weight map according to the isolated point-enlarged light spot detection map.

[0021] Some embodiments of the present application perform a point-filling operation on isolated light spots (i.e., discrete sparse points) included in the sparse light spot detection image, i.e., increase the pixel values ​​or normalized pixel values ​​(i.e., weight values) of these points, thereby making the discrete light spots brighter and improving the light spot display effect.

[0022] In some embodiments, the pixel value or normalized pixel value increasing operation is performed on at least the discrete sparse points on the sparse light spot detection map according to the light spot detection map, including: if it is confirmed that at least one pixel point on the light spot detection map belongs to a discrete sparse point, then the value of the pixel point on the sparse light spot detection map corresponding to the discrete sparse point is increased, and the value of each pixel point in the neighborhood corresponding to the discrete sparse point on the sparse light spot detection map is increased.

[0023] Some embodiments of the present application first identify discrete sparse points (i.e., isolated light spots) through a light spot detection map, and increase the weight values ​​(i.e., normalized pixel values) or pixel values ​​of the pixels corresponding to the discrete sparse points on the sparse light spot detection map and the pixels in their neighborhood. This can improve the layering of the light spot display corresponding to the discrete sparse points, and thereby improve the display effect of the light spots corresponding to the discrete sparse points.

[0024] In some embodiments, for any discrete sparse point, increasing the value of the pixel point corresponding to the discrete sparse point on the sparsened light spot detection map includes: replacing the value of the pixel point corresponding to the any discrete sparse point on the sparsened light spot detection map with a central increase value, wherein the central increase value is the value of the pixel point corresponding to the any discrete sparse point on the light spot detection map, wherein the value of the pixel point includes a pixel value or a normalized pixel value.

[0025] Some embodiments of the present application replace the values ​​of pixel points corresponding to discrete sparse points on the sparse light spot detection image with the values ​​of the same pixel points on the light spot detection image, thereby realizing an operation of increasing the weight values ​​of the discrete sparse points, i.e., performing weight gain on the sparse discrete points, which can effectively avoid the subsequent inability to restore the display effect of the light spots corresponding to the discrete sparse points.

[0026] In some embodiments, increasing the value of each pixel point in the neighborhood corresponding to the discrete sparse point on the sparse light spot detection image includes: replacing the value of each pixel point in the neighborhood corresponding to any discrete sparse point on the sparse light spot detection image with an edge increase value, wherein the edge increase value is a value less than the center increase value and greater than zero.

[0027] Some embodiments of the present application increase the value of each pixel in the neighborhood of a discrete sparse point on a sparse light spot detection image to a value smaller than the value of the pixel corresponding to the discrete sparse point, which can effectively improve the display effect of the light spot corresponding to the discrete sparse point, making the blurred light spot effect brighter, more transparent and sharper.

[0028] In some embodiments, the light spot rendering according to the light spot weight map obtains a background light spot blurred rendering map, including: saturation enhancing the image to obtain a saturation enhanced image; segmenting the saturation enhanced image to obtain a saturation enhanced background segmentation map; background stretching the saturation enhanced background segmentation map to obtain a stretched background segmentation map; performing light spot area brightness gain processing according to the stretched background segmentation map to obtain a background image to be rendered; acquiring a depth estimation map; and obtaining the background light spot blurred rendering map according to the background image to be rendered, the light spot weight map and the depth estimation map.

[0029] Some embodiments of the present application can enhance the effect of the light spot by background stretching and brightness gain processing, so that the blurred light spot display effect is better.

[0030] In a second aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0031] In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement a method as described in any embodiment of the first aspect.

[0032] In a fourth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a target object blurring system provided in an embodiment of the present application;

[0035] Figure 2 One of the flow charts of the target object blurring method provided in the embodiment of the present application;

[0036] Figure 3 The second flowchart of the method for blurring a target object provided in an embodiment of the present application;

[0037] Figure 4 A block diagram of the composition of a target object blurring device provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0040] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0041] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0042] In the related technology, image blurring technology based on mobile terminals (for example, electronic devices such as mobile phones) mainly includes portrait segmentation technology and simulated depth of field rendering technology. The current mainstream method of portrait (i.e., as a specific example of the target object) segmentation technology is mainly based on deep learning models, and the simulated depth of field blurring rendering technology is mainly simulated by methods similar to Gaussian blur. However, there are many effect problems with the target object blurring effect (for example, portrait blurring effect) on mobile terminals (for example, mobile phones), such as unsightly blurring effect, blurred and unclear light spots, and other problems. Some embodiments of the present application can at least solve the problem of blurred light spots when the target object is blurred. That is to say, the light spots displayed on the generated image after blurring the portrait background using the target object blurring method of some embodiments of the present application are clearer, more transparent and brighter.

[0043] Please see Figure 1 , Figure 1A target object blurring system 100 is provided for some embodiments of the present application, wherein the system 100 includes a mobile phone terminal 10 and a server 20, wherein the mobile phone terminal 10 has an image capturing unit 11, and in some embodiments of the present application, an image 21 to be blurred (for example, a portrait image to be blurred) can be captured by the image capturing unit 11, and then the mobile phone terminal 10 sends the image 21 to be blurred to the server 20. The server 20 can perform background blurring operation on the received image to be blurred, obtain a target object blurring rendering image with a good blurring effect, and feed the image back to the mobile phone terminal 10, so that the mobile phone terminal 10 obtains a target object blurring rendering image with a good blurring effect. It should be noted that the camera 11 of the mobile phone terminal 10 is located on the back of the mobile phone, and in other embodiments, the camera can also be located at other positions such as the front of the mobile phone terminal 10, and the embodiment of the present application does not limit the specific position of the image capturing unit 11 on the mobile phone terminal 10.

[0044] The mobile phone terminal 10 may be any type of electronic device with an image capturing function. For example, the mobile phone terminal 10 may also be a laptop computer with an image capturing function, etc., which is not limited in the embodiments of the present application.

[0045] The server 20 is connected to the mobile terminal 10 via a wired network (not shown in the figure) or a wireless network, and the image to be blurred and the blurred light spot rendering image of the target object are transmitted through the connection.

[0046] It should be noted that in some embodiments of the present application, the mobile terminal 10 can also use its own processor to perform the blurring operation on the image to be blurred, and then obtain the blurred rendering of the target object light spot. It can be understood that in these embodiments, the mobile terminal 10 does not output the unblurred image to the user, but the processor of the mobile terminal 10 performs algorithm processing on the initial image taken by the camera (that is, executes the target object blurring method provided in some embodiments of the present application) and finally outputs the target object light spot blurred rendering to the user. It is not difficult to understand that at this time, it is not necessary to independently set up a server 20 to execute the target object blurring method of the embodiment of the present application.

[0047] Combine the following Figure 2 The target object blurring method of some embodiments of the present application executed by the server 20 or by an electronic device such as a mobile phone terminal 10 is exemplified.

[0048] like Figure 2As shown, some embodiments of the present application provide a method for blurring a target object, which method includes: S101, obtaining an image to be blurred, wherein the image includes a target object; S102, obtaining a segmentation map and a background segmentation map of the target object based on the image; S103, performing light spot detection on the background segmentation map to obtain a light spot detection map, and obtaining a light spot weight map based on the light spot detection map, wherein the light spot detection map includes light spots and non-light spots, and the light spot weight map is used to characterize the weight value of any light spot and the weight value of the non-light spot, and the weight value of any light spot is greater than the weight value of each non-light spot in the neighborhood corresponding to the any light spot; S104, obtaining a background light spot blurred rendering map based on the light spot weight map; S105, fusing the background light spot blurred rendering map and the segmentation map of the target object to obtain a target object light spot blurred rendering map.

[0049] That is to say, some embodiments of the present application perform spot rendering on the background segmentation map based on the acquired spot weight map, thereby improving the spot display effect of the target object spot blurred rendering map, making the spot on the image clearer, transparent and brighter, and generating an obvious spot shape with the spot shooting effect of a SLR.

[0050] The following is an exemplary description of the steps involved in the above process.

[0051] The image involved in S101 may be a multi-channel image. For example, the image to be blurred is an RGB three-channel portrait image taken by a mobile phone in a portrait blur shooting mode. For example, the user focuses on the portrait and clicks to take a photo. The mobile phone device obtains the RGB format image I to be blurred. image It should be noted that the image involved in S101 may also be a multi-channel image with more than three channels, and the embodiments of the present application do not limit the number of channels. It is understandable that the target object included in the image in S101 may be a person or other object.

[0052] The target object involved in S101 may include a person or other object. In some embodiments of the present application, S101 captures a portrait image in a portrait blur mode, and S102 is executed to perform image segmentation based on the image of S101 to obtain a portrait segmentation map (a specific example of a target object segmentation map) and a background segmentation map. In some embodiments of the present application, S101 captures a plant image in an object blur mode, and S102 is executed to perform image segmentation on the image of S101 to obtain a plant segmentation map and a background segmentation map.

[0053] In some embodiments of the present application, the process of obtaining the background segmentation map involved in S102 exemplarily includes: using a deep learning model (such as unet, resnet and other frameworks) to segment the captured image to be blurred (or called the original image), obtaining a segmentation mask of the target object, and then obtaining a background object segmentation map based on the segmentation mask of the target object.

[0054] It should be noted that the target object segmentation map and the background object segmentation map involved in S102 are non-binary maps, and the pixel value of each pixel point in the target object area on the target object segmentation map is equal to the pixel value of each pixel point on the image to be blurred obtained by shooting (or equal to the saturation enhanced pixel value of each pixel point after saturation processing is performed on the image to be blurred obtained by shooting), and the pixel value of each pixel point in other areas except the target object on the target object segmentation map is 0. On the background segmentation map, the pixel value of each pixel point included in the background area except the target object is equal to the pixel value of each pixel point on the image to be blurred obtained by shooting, and the pixel value of each pixel point in the area corresponding to the target object is 0. It can be understood that when the image to be blurred is a multi-channel image, the number of channels of the background segmentation map is the same as the number of channels of the image to be blurred.

[0055] The implementation process of S103 is exemplarily described below, and the process exemplarily includes the following three steps.

[0056] It should be noted that the light spot detection map involved in S103 includes light spots and non-light spots, and the light spot weight map is used to characterize the weight value of any light spot and the weight value of the non-light spot, and the weight value of any light spot is greater than the weight values ​​of each non-light spot in the neighborhood corresponding to the any light spot.

[0057] Some embodiments of the present application obtain a single-channel image by selecting the maximum value of each pixel from a multi-channel background segmentation image, and then perform light spot detection based on the single-channel image to obtain a light spot detection image, which can improve the accuracy of the obtained candidate light spots.

[0058] For example, in some embodiments of the present application, S103 exemplarily includes:

[0059] In the first step, a single channel image is obtained based on the maximum value among multiple channels of each pixel in the background segmentation image. That is, for each pixel, the maximum value is selected from multiple channels of the background segmentation image obtained in S102 to obtain a single channel image.

[0060] For example, in some embodiments of the present application, the image to be blurred is an RGB three-channel image, so the background segmentation image obtained after segmenting the image is also an RGB three-channel image, and the value of any pixel on the single-channel image obtained based on the background segmentation image is the maximum value selected from the three channels corresponding to the any pixel. For example, for the RGB three-channel values ​​corresponding to the pixel at the upper left corner (i.e., the coordinate point (0,0)) on the background segmentation image, the value is (16, 27, 37), then the value of the pixel at the upper left corner on the single-channel image is 37, that is, the maximum value is selected from the three channels as the value of the pixel at the upper left corner on the single-channel image. For the RGB three-channel values ​​corresponding to the pixel with coordinates (10, 10) on the background segmentation image, the value is (254, 100, 76), then the value of the pixel on the single-channel image is 254, that is, the maximum value is selected from the three channels as the value of the pixel on the single-channel image.

[0061] In the second step, light spot detection is performed on the single-channel image to obtain a light spot detection image, wherein the pixel values ​​of the pixels corresponding to the light spots on the light spot detection image are the pixel values ​​of the corresponding pixels on the single-channel image, and the pixel values ​​of the pixels corresponding to the non-light spots on the light spot detection image are 0.

[0062] For example, in some embodiments of the present application, light spots can be directly screened out from a single-channel image according to the attribute of the light spot area (i.e., a light spot detection image is obtained), and the attribute of the light spot area is that the value located at the center of the light spot area (i.e., the light spot) on the single-channel image is greater than the value of the pixel points located at other positions of the light spot area. For example, in some embodiments of the present application, the above-mentioned process of performing light spot detection on the single-channel image to obtain a light spot detection image composed of light spots exemplarily includes: confirming that the attribute value of any pixel point on the single-channel image is greater than or equal to a first threshold, and confirming that the attribute value of each pixel point in the neighborhood corresponding to the any pixel point on the single-channel image is less than or equal to a second threshold, then confirming that the any pixel point belongs to the detected light spot, wherein the second threshold value is less than the first threshold value, and the attribute value includes a pixel value or a normalized pixel, and the normalized pixel value is the value of each pixel point obtained after normalizing the pixel values ​​of all pixels on the entire image.

[0063] Specifically, in some embodiments of the present application, the above-mentioned process of performing light spot detection on the single-channel image to obtain a light spot detection image exemplarily includes: if the normalized pixel value (or weight value) of any pixel point on the single-channel image is greater than or equal to a first threshold, and the normalized pixel value (or weight value) of each pixel point in the neighborhood corresponding to the any pixel point on the single-channel image is less than or equal to a second threshold, then confirming that the any pixel point belongs to the detected light spot. In some other embodiments of the present application, the above-mentioned process of performing light spot detection on the single-channel image to obtain a light spot detection image composed of each light spot exemplarily includes: if the pixel value of any pixel point on the single-channel image is greater than or equal to a first threshold, and the pixel value of each pixel point in the neighborhood corresponding to the any pixel point on the single-channel image is less than or equal to a second threshold, then confirming that the any pixel point belongs to the detected light spot.

[0064] It should be noted that the first threshold and the second threshold can be a value or a range of values. When the first threshold and the second threshold are a range of values, the minimum value included in the first threshold must be greater than the maximum value included in the second threshold. When the light spot on the single-channel image is detected by normalizing the pixel value, the first threshold is a decimal less than or equal to 1 or a range of values ​​consisting of multiple decimals less than 1, and the second threshold is also a decimal less than 1 or a range of values ​​consisting of multiple decimals less than 1.

[0065] For example, in some embodiments of the present application, the value of each pixel on the single-channel image may be blurred, and whether the pixel belongs to a light spot is determined based on the size attribute of the value of any pixel on the image before blurring (i.e., the single-channel image) and the image after blurring (i.e., the blurred single-channel image). For example, in some embodiments of the present application, the above-mentioned process of performing light spot detection on the single-channel image to obtain a light spot detection image exemplarily includes: blurring the pixel value of each pixel on the single-channel image to obtain a blurred single-channel image; if the attribute value of any pixel on the single-channel image is greater than a third threshold, and the attribute value of any pixel on the blurred single-channel image is less than a fourth threshold, then it is confirmed that the any pixel belongs to the detected light spot, wherein the third threshold is greater than the fourth threshold, and the attribute value includes a pixel value or a normalized pixel value, and the normalized pixel value is the value of each pixel obtained by normalizing the pixel values ​​of all pixels on the entire image.

[0066] Specifically, in some embodiments of the present application, the pixel value of each pixel point on the single-channel image is blurred to obtain a blurred single-channel image; if the pixel value of any pixel point on the single-channel image is greater than a third threshold, and the pixel value of any pixel point on the blurred single-channel image is less than a fourth threshold, then it is confirmed that the any pixel point belongs to the detected light spot. In other embodiments of the present application, the pixel value of each pixel point on the single-channel image is blurred to obtain a blurred single-channel image; if the normalized pixel value (or weight value) of any pixel point on the single-channel image is greater than a third threshold, and the normalized pixel value of any pixel point on the blurred single-channel image is less than a fourth threshold, then it is confirmed that the any pixel point belongs to the detected light spot.

[0067] It should be noted that the third threshold and the fourth threshold can be a value or a range of values. When the third threshold and the fourth threshold are a range of values, the minimum value included in the third threshold also needs to be greater than the maximum value included in the fourth threshold. When the light spot on the single-channel image is detected by normalizing the pixel value, the third threshold is a decimal less than or equal to 1 or a range of values ​​consisting of multiple decimals less than 1, and the fourth threshold is also a decimal less than 1 or a range of values ​​consisting of multiple decimals less than 1.

[0068] It is not difficult to understand that some embodiments of the present application utilize the properties of the displayed light spot pattern by first blurring the single-channel image, that is, searching for light spots on the single-channel image based on the weight value or pixel value size attribute of the same pixel point on the blurred image (i.e., blurred single-channel image) and the non-blurred image (i.e., single-channel image), thereby improving the accuracy of the obtained candidate light spots and also improving the speed of data processing.

[0069] The following describes how to determine the value of each pixel on the light spot detection image with reference to a specific example.

[0070] In some embodiments of the present application, as in the above example, the value of the upper left corner pixel on a single-channel image is 37. If the light spot on the upper left corner is confirmed to be a light spot after light spot detection is performed on the single-channel image, the value of the upper left corner pixel on the light spot detection image is also 37 (i.e., equal to the value of the same pixel on the single-channel image). If the pixel with pixel coordinates (10,9) is confirmed to be a non-light spot after light spot detection is performed on the channel image, the value corresponding to the pixel on the light spot detection image is 0. It is not difficult to understand that the pixel with a value of 0 on the light spot detection image belongs to the non-light spot detected by the single-channel image, and the pixel with a non-zero value on the light spot detection image belongs to the light spot detected by the single-channel image.

[0071] The third step is to obtain the light spot weight map according to the light spot detection map.

[0072] For example, in some embodiments of the present application, the light spot detection map is normalized to obtain the light spot weight map. In some embodiments of the present application, it is necessary to perform a thinning operation on the light spot detection map, and then obtain the light spot weight map based on the thinned light spot detection map obtained after the thinning operation. In other embodiments of the present application, it is necessary to perform a thinning operation and a point filling operation (the specific meaning of the point filling operation can be referred to the following description) on the light spot detection map in sequence to obtain the light spot detection map after the point filling operation, and then obtain the light spot weight map based on the light spot detection map after the point filling operation.

[0073] The following is an exemplary description of the process of performing a thinning operation on the light spot detection map to obtain a light spot weight map.

[0074] In some embodiments of the present application, the process of obtaining the light spot weight map according to the light spot detection map involved in the third step exemplarily includes:

[0075] In the first sub-step, a thinning operation is performed on the light spot detection map or a dense area included in the light spot detection map to obtain a thinned light spot detection map. It should be noted that the thinning operation will partially set the pixel points in certain areas of the light spot detection map to zero or average the pixel values ​​to a certain extent, so the number of light spots in the local area (dense area) on the light spot detection map will decrease or the maximum value will decrease.

[0076] It can be understood that the above-mentioned first sub-step includes at least the following two types of examples. In some embodiments of the present application, a thinning operation is performed on the light spot detection map as a whole to obtain a sparse light spot detection map. In other embodiments of the present application, a thinning operation is performed on the dense area included in the light spot detection map to obtain a sparse light spot detection map. It is not difficult to understand that if a thinning operation is performed on a dense area on the light spot detection map, it is necessary to first find the existing dense areas on the light spot detection map. For example, when the number of light spots (such as those determined by the above-mentioned light spot determination method) located in a certain area is greater than a set threshold, the area can be confirmed as a dense area, and then the thinning operation is performed on each identified dense area. For example, before performing a thinning operation on a dense area included in the light spot detection image, the target object blurring method also includes: dividing the light spot detection image into multiple sub-images; obtaining the coordinates of the center of mass of each sub-image in the multiple sub-images; taking each center of mass as the center on the light spot detection image, solving the total number of non-zero values ​​included in all pixel points in a neighborhood corresponding to each center; and determining the dense area from the multiple sub-images based on the total number.

[0077] In some embodiments of the present application, the thinning operation includes multiple implementation methods. For example, in some embodiments of the present application, a value can be retained in each set pixel area while the pixel values ​​or normalized pixel values ​​of other pixels in the area are 0 to complete the thinning operation. In other embodiments of the present application, bilinear interpolation can also be used to up-sample and down-sample the original image (i.e., a specific example of the briefing book). For example, the above-mentioned process of performing a thinning operation on the light spot detection map to obtain a sparse light spot detection map exemplarily includes: according to a target sampling rate, down-sampling and up-sampling the light spot detection map using bilinear interpolation to obtain the sparse light spot detection map, wherein the target sampling rate is a fixed value or is dynamically determined by the distribution density of the light spots on the light spot detection map. That is to say, some embodiments of the present application use bilinear interpolation to perform thinning processing on the light spot detection map or the dense area included in the light spot detection map.

[0078] It is not difficult to understand that some embodiments of the present application thin out the areas where light spots are densely clustered on the light spot detection map to prevent the problem of overexposure and lack of layering in large areas of light spots in subsequent rendering, thereby effectively improving the display effect of the light spots.

[0079] The second sub-step is to obtain the light spot weight map according to the sparse light spot detection map.

[0080] It should be noted that the second sub-step includes the following multiple examples. For example, in some embodiments of the present application, the pixel values ​​of each pixel on the sparse light spot detection map are directly normalized to obtain a sparse light spot weight map, that is, the sparse light spot weight map can be used as a light spot weight map (for example, when the light spot detection map includes a dense area and the sparse light spot detection map is subjected to a sparse operation, the values ​​of all pixels on the sparse light spot detection map can be directly normalized to obtain a light spot weight map). In some other embodiments of the present application, it is also necessary to perform a point-filling operation on the obtained sparse light spot detection map (for example, after the light spot detection map is subjected to a sparse operation as a whole, the obtained sparse light spot detection map can be firstly subjected to a point-filling operation) to improve the light spot blurring effect of isolated points, and the weight map obtained after the point-filling operation is used as a light spot weight map. It can be understood that the value of each pixel on the light spot weight map is a weight value, which is a value less than or equal to 1 after the pixel value of each pixel on the image is normalized.

[0081] The following is an exemplary description of the process of performing a point filling operation on a sparse light spot detection image.

[0082] It should be noted that the sparse light spot detection image that needs to be filled in can be an image obtained by thinning the entire light spot detection image, or an image obtained by thinning the dense area included in the light spot detection. For example, since the thinning operation of the entire light spot detection image will have side effects, that is, the original sparse discrete points will also become fewer, and the weight value will also become smaller, it is necessary to fill in these light spots, that is, increase the pixel value of these points or normalize the pixel value (that is, the weight value), so as to make the discrete light spots brighter and improve the light spot display effect.

[0083] For example, in some embodiments of the present application, the process of obtaining the light spot weight map according to the sparse light spot detection map involved in the above-mentioned second sub-step exemplarily includes: performing an operation of increasing the pixel values ​​or normalized pixel values ​​of at least discrete sparse points on the sparse light spot detection map according to the light spot detection map to obtain an isolated point enlarged light spot detection map; obtaining the light spot weight map according to the isolated point enlarged light spot detection map (for example, normalizing the isolated point enlarged light spot detection map to obtain the light spot weight map).

[0084] It is not difficult to understand that in order to increase the values ​​of the pixels corresponding to the discrete sparse points on the sparse light spot detection map, it is necessary to determine which pixels on the sparse light spot detection map belong to the discrete sparse points. For example, some embodiments of the present application first identify discrete sparse points (i.e., isolated light spots) through the light spot detection map, and increase the weight values ​​or pixel values ​​of the pixels corresponding to the discrete sparse points on the sparse light spot detection map and the pixels in their neighborhood, so that the display effect of the light spots corresponding to the discrete sparse points can be improved.

[0085] Specifically, in some embodiments of the present application, the process of increasing the pixel value or normalized pixel value (or weight value) of at least the discrete sparse points on the sparse light spot detection map according to the light spot detection map exemplarily includes the following A), B) and C):

[0086] A) confirming that at least one pixel point on the light spot detection image belongs to a discrete sparse point.

[0087] For example, in some embodiments of the present application, the above step A) exemplarily includes: if the number of light spots included in the target area on the light spot detection map is less than or equal to the light spot number threshold, then confirming that the pixel point located at the center of the target area belongs to a discrete sparse point. In other words, some embodiments of the present application provide a method for obtaining discrete sparse points based on a light spot detection map, so that the obtained discrete sparse points are more accurate.

[0088] B) Increasing the values ​​of the pixels corresponding to the discrete sparse points on the sparse spot detection image. It should be noted that in some embodiments of the present application, the values ​​of the pixels corresponding to the discrete sparse points may be increased to a set value, or the value of the pixels corresponding to the discrete sparse points may be increased to a certain value based on a reference image (for example, increased to a range of half to the same value of the pixels corresponding to the corresponding discrete sparse points on the reference image).

[0089] In order to make the increased value corresponding to the discrete sparse point more realistically reflect the original brightness of the light spot, in some embodiments of the present application, the value of the pixel corresponding to the discrete sparse point on the sparse light spot detection map can be increased according to the value of the pixel on the light spot detection map. For example, in some embodiments of the present application, the above-mentioned process of increasing the value of the pixel corresponding to the discrete sparse point on the sparse light spot detection map exemplarily includes: for any discrete sparse point, the value of the pixel corresponding to any discrete sparse point on the sparse light spot detection map is replaced with the center increase value, wherein the center increase value is the value of the pixel corresponding to any discrete sparse point on the light spot detection map. It can be understood that the value of the pixel corresponding to the discrete sparse point includes the pixel value or the normalized pixel value of the pixel, and the method of obtaining the normalized pixel value is the same in the embodiments of the present application.

[0090] For example, if the pixel with coordinates (20, 25) is confirmed as a discrete sparse point, and the value of the discrete sparse point on the sparse light spot detection map is 120 and the value on the light spot detection map is 200, then the point filling operation is performed, and the value of the pixel on the sparse light spot detection map is replaced with 200. For example, if the pixel with coordinates (40, 25) is confirmed as a discrete sparse point, and the value of the discrete sparse point on the sparse light spot detection map is 0.3 (i.e., the normalized pixel value) and the value on the light spot detection map is 0.8, then the point filling operation is performed, and the value of the pixel on the sparse light spot detection map is updated to 0.8.

[0091] C) Increase the value of each pixel in the neighborhood corresponding to the discrete sparse point on the sparse spot detection image. It should be noted that in some embodiments of the present application, the pixel values ​​of the pixels in the neighborhood corresponding to these discrete sparse points can be increased to a set value, or the pixel values ​​of the pixels corresponding to the discrete sparse points can be increased to a certain value based on a reference image. In order to make the discrete sparse points have a better spot display effect, the value of each pixel in the neighborhood corresponding to the first discrete sparse point (belonging to any discrete sparse point) after the point filling operation can be less than the value of the pixel corresponding to the first discrete sparse point.

[0092] For example, in order to further improve the light spot display effect, some embodiments of the present application increase the value of the pixel points in the neighborhood of the discrete sparse point on the sparse light spot detection map to an increase value less than the corresponding discrete sparse point, which can effectively improve the display effect of the light spot corresponding to the discrete sparse point, making the blurred light spot effect brighter, more transparent and sharper. Specifically, in some embodiments of the present application, the above-mentioned process of increasing the value of each pixel point in the neighborhood corresponding to the discrete sparse point on the sparse light spot detection map exemplarily includes: replacing the pixel value of each pixel point in the neighborhood corresponding to any discrete sparse point on the sparse light spot detection map with an edge increase value, wherein the edge increase value is a value less than the center increase value and greater than zero. For example, the edge increase value is half the size of the center increase value.

[0093] The implementation process of S104 is described below by way of example.

[0094] Some embodiments of the present application can enhance the effect of the light spot by background stretching and brightness gain processing, so that the light spot display effect after blurring is better. For example, in some embodiments of the present application, S104 exemplarily includes: performing saturation enhancement on the captured image to be blurred to obtain a saturation enhanced image; segmenting the saturation enhanced image to obtain a saturation enhanced background segmentation map; performing background stretching on the saturation enhanced background segmentation map to obtain a stretched background segmentation map; performing brightness gain processing on the light spot area according to the stretched background segmentation map to obtain a background image to be rendered; obtaining a depth estimation map; obtaining the background light spot blurred rendering map according to the background image to be rendered, the light spot weight map and the depth estimation map.

[0095] Combine the following Figure 3 The target object blurring method of the present application is illustrated by taking the portrait blurring method executed on a mobile phone as an example.

[0096] Some embodiments of the present application can at least solve the problem of unnatural blur rendering and blurred and unclear light spots in portrait mode on the mobile phone, and simulate the light spot effect close to the shooting effect of SLR. For example, the target object blurring method of some embodiments of the present application includes: first using a deep learning model (for example, unet, resnet framework segmentation model) to perform portrait segmentation on the captured image (obtaining a background segmentation map), using a depth estimation model to obtain a depth of field distribution map (or called a coc distribution map), and pre-processing the background segmentation map, and then using the convolution gather method to perform blur rendering. The beneficial effects achieved by some embodiments of the present application include: the portrait blur effect is more natural, and the blurred light spot effect is brighter, more transparent and sharper.

[0097] like Figure 3 As shown, the portrait blurring method performed by the mobile phone terminal in some embodiments of the present application includes:

[0098] S201, the mobile phone camera captures an image.

[0099] The user focuses on the portrait and clicks to take a photo. The mobile device obtains the RGB format image to be blurred. image , that is, the image to be blurred I is obtained image .

[0100] S202, separating the foreground and background according to the portrait segmentation model to obtain a portrait binary image, that is, according to the image segmentation model of the related technology (that is, the deep learning model), the blurred image is segmented into the portrait (that is, the foreground) and the background to obtain a portrait segmentation mask.

[0101] The deep learning model of the related technology (for example, networks such as unet or resnet) treats the blurred image I image Perform portrait segmentation to obtain a single-channel portrait binary image I mask , where the values ​​of all pixels in the main area (i.e., the area corresponding to the portrait) on the single-channel portrait binary image are 1, and the values ​​of all pixels in the background area except the portrait are 0.

[0102] S203, generating a light spot detection map of the background, that is, generating a light spot detection map of the background according to the portrait binary image.

[0103] a) Utilize I image ×(1-I mask ) Get the background segmentation map I except the portrait bg , the area corresponding to the portrait on the background segmentation map needs to be filled with 0.

[0104] b) Based on the background segmentation map I bg The R, G and B channels are traversed pixel by pixel to find the maximum value of the three channels, that is, the single channel image I is obtained. maxch_bg .

[0105] c) Based on single channel graph I maxch_bg Perform light spot detection, the detailed process is as follows:

[0106] i. For single channel graph I maxch_bg Perform N*N (for example, 3*3) Gaussian blur to obtain a blurred single-channel image I smooth_maxch_bg .

[0107] ii. For single channel graph I maxch_bg and fuzzy single channel image I smooth_maxch_bg Traverse, if the value of the pixel at the (x, y) position meets the following conditions, then the pixel is considered to be a candidate light spot (that is, it belongs to a light spot), and obtain the light spot detection image I composed of all candidate light spots init_spots :

[0108]

[0109] The above if statement is a conditional judgment statement in C language. The function of this judgment statement is: when it is judged that two inequalities are true at the same time, the assignment statement after the semicolon is executed. The above value operation represents the normalization operation of the parameters in the brackets. Specifically, value(I maxch_bg )(x,y) represents the value of the pixel point (x, y) on the normalized image after normalizing the values ​​of all pixels on the single-channel image (or called the normalized pixel value, or also called the weight value). It is not difficult to understand that this value is a value greater than or equal to 0 and less than or equal to 1. value(I maxch_smooth_bg )(x, y) represents the value of the pixel point (x, y) on the blurred single channel image after the values ​​of all pixels on the blurred single channel image are normalized (or called the normalized pixel value, also called the weight value). It is not difficult to understand that this value is a value greater than or equal to 0 and less than or equal to 1. The above if statement needs to judge that the normalized pixel value of the pixel point (x, y) on the single channel image is greater than the set third threshold bokeh_thresh1, and the normalized pixel value of the pixel point on the blurred single channel image is less than the set fourth threshold bokeh_thresh2. In combination with the above statements, it is not difficult to find that these two thresholds belong to two numerical ranges less than 1, and the maximum value included in the fourth threshold is less than the minimum value included in the third threshold. In addition, it can be seen from the above statements that when the pixel point (x, y) satisfies the two conditions of the judgment statement (i.e., the if statement) at the same time, it is determined that the pixel point belongs to a light spot, and for the light spot, the value of the pixel point (x, y) on the single channel image needs to be assigned to the light spot detection image I init_spots It should be noted that all pixels that do not meet the two conditions of the above if statement are determined not to belong to light spots, and the values ​​of these pixels on the light spot detection map are 0. It is not difficult to understand that a map composed of all the determined light spots can be determined through the light spot detection map.

[0110] S204, thinning the dense area.

[0111] d) Light spot detection image I init_spots Or light spot detection image I init_spots The dense area included is subjected to a sparse operation to obtain a sparse light spot detection image. For example, based on the image scaling operation resize operation provided by the open source library opencv (i.e., Open Source Computer Vision Library), the light spot detection image I init_spotsFirst downsample by 4 times (a specific example of the target sampling rate) and then upsample by 4 times. In this way, you can simply and directly use the bilinear interpolation provided by resize to make screening points and thin out the areas where light spots are densely gathered to prevent overexposure and lack of layering in subsequent rendering of large areas where light spots are clustered.

[0112] The process of obtaining a dense area is described below by way of example.

[0113] Judgment I init_spots The density of light spots in the image includes: dividing the image to be blurred into 4 small images of the same size according to half of the width and height; solving the centroid coordinates of each small image, that is, the center of the pixel concentration in the region. The specific operation is to solve the zero-order moment M of each small image. 00 and the first-order moment M 10 、M 01 ; The centroid coordinates are obtained based on the following formula: Taking each centroid coordinate point as the center, calculate the number of non-zero values ​​sum_spots in the range of 100*100.

[0114] Find the sampling threshold of the dense area of ​​the four small images, thresh = sum_spots / 10000. If at least one of the thresh is greater than 10%, it is judged as I init_spots There are dense areas, which will cause light spots to form clusters and overexposure, and other effects problems. init_spots Or the dense area included in the graph is thinned out.

[0115] S205, performing point filling processing on the discrete light spots to obtain a light spot weight map.

[0116] e) Since the operation of step d on dense points will reduce the number of original sparse discrete points and the weight value will also become smaller, it is necessary to perform a point-filling operation on these light spots. The point-filling operation is to perform a point-filling operation on the sparse light spot detection image I' obtained in step d. init_spots Do the following:

[0117] i. Light spot detection image I init_spots Perform pixel traversal. If the number of light spots (or light spots) in the neighborhood of the traversed pixel point (x, y) with a radius of M (for example, M is 3) pixels does not exceed L (for example, L is 8), then the pixel point is considered to be a discrete sparse point and needs to be detected in the sparse light spot detection map I' init_spots The position (x, y) and the surrounding pixel points are filled in. It should be noted that filling in points means increasing the values ​​of these pixel points on the sparse light spot detection image. The specific method is as follows.

[0118] ii. Update sparse light spot detection map I'init_spots The value of the discrete sparse point (x, y) on the image (the type of the value includes a weight value or a pixel value), for example, the light spot detection image I init_spots The value of the pixel corresponding to the discrete sparse point is assigned to the same pixel on the sparse light spot detection map, in order to perform weight gain on the sparse discrete points.

[0119] iii. Update sparse light spot detection map I' init_spots The values ​​of the points around the discrete sparse point (x, y) on the image are assigned the same weight within a neighborhood with a radius of P (for example, the value of P is 1) from the discrete sparse point. That is, the weight values ​​of multiple (for example, when the value of P is 1, the value is 8) pixel points around the pixel point (x, y) corresponding to the discrete sparse point on the sparse light spot detection map are increased, in order to make the discrete light spot brighter.

[0120] f) Based on step e, the light spot weight map I can be obtained spots The purpose of making this figure is to enhance the weight factor of the light spot area when performing gather convolution blur, which is different from the ordinary blur effect of non-light spot.

[0121] S206, performing gamma stretching on the background.

[0122] g) Perform appropriate saturation enhancement on the original image (i.e., the image to be blurred) (i.e., perform Figure 3 S230) to obtain the saturation enhancement image I' image (ie, the color-enhanced image of S230 is obtained). The purpose of obtaining this image is to increase the gorgeousness of the blurred background portrait and the diversity of the light spot color.

[0123] h) Using the saturation enhancement map and formula I' image ×(1-I mask ) to obtain the saturation enhanced background segmentation map I' except the portrait bg , the portrait area in the image is filled with 0.

[0124] i) Saturation enhanced background segmentation map I' bg Perform gamma transformation (for example, gamma = 1.6) to nonlinearly stretch the background brightness to obtain a stretched background segmentation map I' gamma_bg .

[0125] S207, performing brightness gain processing on the light spot area on the background to obtain the image to be rendered. That is, performing brightness gain processing on the light spot area according to the stretched background segmentation map to obtain the background image to be rendered.

[0126] j) Perform the following spot area brightness gain operation on the background:

[0127] i. Change the original picture I image Convert to YUV format image, where the single channel Y image is named I gray .

[0128] ii. will I gray Normalized to [0,1], we get I light_weight , then +1 as a whole, the value range is [1, 2].

[0129] iii. According to I light_weight *I' gamma_bg Get the image I after brightness gain image4render , that is, the background image I to be rendered is obtained image4render .

[0130] S208, executing a light spot rendering method to obtain a background light spot blurred rendering image. That is, executing a light spot rendering method according to the light spot weight map and the image to be rendered to obtain a background light spot blurred rendering image.

[0131] It should be noted that in order to execute the spot rendering method in S208, a depth of field image needs to be obtained. The process exemplarily includes the following operations: image Perform depth estimation and use deep learning methods (e.g., mono depth, midas, etc.) to obtain a depth map I depth When the mobile phone camera is fixed in focus, it will return the coordinate position (x0, y0) of the portrait in the image. The depth of field image I is obtained by the following formula coc ;

[0132]

[0133] Among them, coc represents the depth of field, that is, the size of the spot radius required for rendering.

[0134] The corresponding S208 exemplarily includes: spots ,I image4render and I coc As input, it is sent to the light spot rendering method based on gather convolution, and gamma transformation is performed again (for example, gamma=1 / 1.6), and finally a background light spot blurred rendering image is obtained.

[0135] S209, merging the portrait after saturation gain with the background light spot blurred rendering image to obtain a portrait light spot blurred rendering image.

[0136] It should be noted that the a)-f) sub-steps included in the above S203, S204 and S205, the g)-j) sub-steps included in the above S206 and S207, and the sub-step of obtaining the depth of field image belong to three sub-threads that can be processed in parallel. In summary, the above-mentioned portrait blurring method includes: the user points to the portrait and clicks to shoot, and the mobile device background obtains the required original image; through spot detection, dense area sparse processing and discrete area point filling operations, a spot weight map is obtained; by performing gamma transformation based on the original image, background spot area brightness enhancement and other operations, the image to be rendered is obtained; the depth of the original image is estimated to obtain the radius size map required for spot rendering; the above three data are input into the spot rendering method to obtain the background spot blur rendering image; the background spot blur rendering image and the portrait are merged to obtain the final portrait spot blur rendering image. By optimizing the process of obtaining the light spot detection image by the light spot detection method, the brightness, edge sharpness and aesthetics of the light spot can be effectively improved; by weighting the light spot area, the brightness, edge reasoning degree and aesthetics of the light spot can be effectively improved.

[0137] Please refer to Figure 4 , Figure 4 The target object blurring device provided in the embodiment of the present application is shown. It should be understood that the device is similar to the above Figure 2 Corresponding to the method embodiment, each step involved in the above method embodiment can be executed. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or solidified in the operating system of the device. The target object blurring device includes: an image acquisition module 410 to be blurred, an image segmentation module 420, a light spot weight map acquisition module 430, a background light spot blur rendering image acquisition module 440, and a target object light spot blur rendering image acquisition module 450.

[0138] The image to be blurred acquisition module 410 is configured to acquire the image to be blurred, wherein the image includes a target object.

[0139] The image segmentation module 420 is configured to obtain a target object segmentation map and a background segmentation map based on the image.

[0140] The light spot weight map acquisition module 430 is configured to perform light spot detection on the background segmentation map to obtain a light spot detection map, and to obtain a light spot weight map based on the light spot detection map, wherein the light spot detection map includes light spots and non-light spots, and the light spot weight map is used to characterize the weight value of any light spot and the weight value of the non-light spot, and the weight value of any light spot is greater than the weight values ​​of each non-light spot in the neighborhood corresponding to the any light spot.

[0141] The background light spot blurred rendering image acquisition module 440 is configured to obtain the background light spot blurred rendering image based on the light spot weight map.

[0142] The target object light spot blurred rendering image acquisition module 450 is configured to fuse the background light spot blurred rendering image with the target object segmented image to obtain a target object light spot blurred rendering image.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0144] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the target object blurring method provided in the above embodiments can be implemented.

[0145] Some embodiments of the present application provide a computer program product, which includes a computer program. When the computer program is executed by a processor, the target object blurring method provided in the above embodiments can be implemented.

[0146] like Figure 5 As shown, some embodiments of the present application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program to implement the method described in any embodiment of the target object blurring method described above.

[0147] Processor 520 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 520 can be a microprocessor.

[0148] The memory 510 may be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 520 of the present disclosure embodiment may be used to execute the instructions in the memory 510 to implement Figure 2 or Figure 3 The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0149] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0151] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0154] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for blurring a target object, characterized in that: The target object blurring method comprises: Acquire an image to be blurred, wherein the image includes a target object; Obtaining a segmentation map of the target object and a background segmentation map based on the image; Performing light spot detection on the background segmentation map to obtain a light spot detection map, and obtaining a light spot weight map according to the light spot detection map, wherein the light spot detection map includes light spots and non-light spots, and the light spot weight map is used to characterize the weight value of any light spot and the weight value of the non-light spot, and the weight value of any light spot is greater than the weight values ​​of each non-light spot in a neighborhood corresponding to the any light spot; Obtaining a background light spot blurred rendering image based on the light spot weight map; The background light spot blurred rendering image and the target object segmentation image are merged to obtain a target object light spot blurred rendering image; The background light spot blur rendering image is obtained based on the light spot weight map, including: Performing saturation enhancement on the image to obtain a saturation enhanced image; Segmenting the saturation enhanced image to obtain a saturation enhanced background segmentation map; Performing background stretching on the saturation enhanced background segmentation map to obtain a stretched background segmentation map; Performing spot area brightness gain processing according to the stretched background segmentation map to obtain a background image to be rendered; Get the depth estimation map; The background light spot blurred rendering image is obtained according to the background image to be rendered, the light spot weight map and the depth estimation map.

2. The method for blurring a target object according to claim 1, wherein: The performing light spot detection on the background segmentation map to obtain a light spot detection map includes: Obtaining a single channel image based on the maximum value of multiple channels of each pixel in the background segmentation image; Performing light spot detection on the single-channel image to obtain the light spot detection image, wherein the pixel values ​​of the pixels corresponding to the light spots on the light spot detection image are the pixel values ​​of the corresponding pixels on the single-channel image, and the pixel values ​​of the pixels corresponding to the non-light spots on the light spot detection image are 0.

3. The method for blurring a target object as claimed in claim 2, characterized in that: The performing light spot detection on the single channel image to obtain the light spot detection image includes: If the attribute value of any pixel point on the single-channel image is greater than or equal to a first threshold, and the attribute value of each pixel point in the neighborhood corresponding to the any pixel point on the single-channel image is less than or equal to a second threshold, then it is confirmed that the any pixel point belongs to the detected light spot, wherein the second threshold is less than the first threshold, and the attribute value includes a pixel value or a normalized pixel value, and the normalized pixel value is the value of each pixel point obtained after normalizing the pixel values ​​of all pixels on the entire image.

4. The method for blurring a target object as claimed in claim 2, characterized in that: The performing light spot detection on the single channel image to obtain the light spot detection image includes: Performing fuzzy processing on the pixel value of each pixel point on the single-channel image to obtain a fuzzy single-channel image; If the attribute value of any pixel point on the single-channel image is greater than the third threshold, and the attribute value of any pixel point on the blurred single-channel image is less than the fourth threshold, it is confirmed that the any pixel point belongs to the detected light spot, wherein the third threshold is greater than the fourth threshold, and the attribute value includes a pixel value or a normalized pixel value, and the normalized pixel value is the value of each pixel point obtained after normalizing the pixel values ​​of all pixels on the entire image.

5. The method for blurring a target object according to any one of claims 1 to 4, characterized in that: The step of obtaining the light spot weight map according to the light spot detection map includes: Performing a thinning operation on the light spot detection image to obtain a thinned light spot detection image; The light spot weight map is obtained according to the thinned light spot detection map.

6. The method for blurring a target object as claimed in claim 5, characterized in that: The performing a thinning operation on the light spot detection map includes: performing a thinning operation on a dense area included in the light spot detection map.

7. The method for blurring a target object as claimed in claim 6, characterized in that: Before performing a thinning operation on a dense area included in the light spot detection image, the target object blurring method further includes: Dividing the light spot detection image into a plurality of sub-images; Obtaining coordinates of the centroid of each subgraph in the multiple subgraphs; Taking each centroid as the center on the light spot detection map, the total number of non-zero value pixel points included in the neighborhood corresponding to each center is solved; The dense region is determined from the plurality of sub-graphs according to the total number.

8. The method for blurring a target object according to any one of claims 6 to 7, characterized in that: The step of obtaining the light spot weight map according to the sparse light spot detection map includes: According to the light spot detection map, at least a discrete sparse point on the sparse light spot detection map is increased in pixel value or normalized pixel value to obtain an isolated point increased light spot detection map; The light spot weight map is obtained by enlarging the light spot detection map according to the isolated points.

9. The method for blurring a target object as claimed in claim 8, characterized in that: The step of performing an operation of increasing pixel values ​​or normalized pixel values ​​of at least discrete sparse points on the sparse light spot detection map according to the light spot detection map includes: If it is confirmed that at least one pixel point on the light spot detection image belongs to a discrete sparse point, the value of the pixel point corresponding to the discrete sparse point on the sparse light spot detection image is increased, and the value of each pixel point in the neighborhood corresponding to the discrete sparse point on the sparse light spot detection image is increased, wherein the value of the pixel point includes a pixel value or a normalized pixel value.

10. The method for blurring a target object as claimed in claim 9, characterized in that: For any discrete sparse point, increasing the value of the pixel point corresponding to the discrete sparse point on the sparse light spot detection map includes: replacing the value of the pixel point corresponding to the any discrete sparse point on the sparse light spot detection map with a central increase value, wherein the central increase value is the value of the pixel point corresponding to the any discrete sparse point on the light spot detection map, wherein the value of the pixel point includes a pixel value or a normalized pixel value.

11. The method for blurring a target object according to claim 10, wherein: The step of increasing the value of each pixel point in the neighborhood corresponding to the discrete sparse point on the sparse spot detection image includes: The value of each pixel point in the neighborhood corresponding to any discrete sparse point on the sparse light spot detection image is replaced by an edge increase value, wherein the edge increase value is a value less than the center increase value and greater than zero.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 11 can be implemented.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method described in any one of claims 1 to 11 can be implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program can implement the method according to any one of claims 1 to 11 when executed by a processor.

Images