Image depth-of-field reconstruction method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an image depth-of-field reconstruction method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a to-be-processed image frame, performing foreground prediction on the to-be-processed image frame to obtain a foreground probability graph, and segmenting the to-be-processed image frame based on the foreground probability graph to obtain an original foreground graph and an original background graph; performing enhancement processing on the original foreground image to obtain a foreground enhancement image; performing depth detection on the to-be-processed image frame to obtain a depth estimation image, and performing layered fuzzy processing on the original background image according to the depth estimation image and the foreground probability image to obtain a background blurred image; and based on the foreground probability graph, fusing the foreground enhancement graph and the background blurring graph to obtain a depth-of-field reconstruction graph. According to the scheme, layered fuzzy processing is carried out on the original background image in combination with the depth information, a chartlet type split feeling caused by unified fuzzy processing on the background is avoided, a soft depth-of-field effect is achieved, meanwhile, clear details of a foreground are reserved, and the video impression of audiences is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to an image depth reconstruction method, apparatus, device, storage medium, and program product. Background Art

[0002] At present, with the continuous development of live broadcast technology, depth of field reconstruction technology has a very wide range of application scenarios in the field of live broadcast. Taking online live broadcast as an example, the live broadcast screen can present scenes such as the host chatting with the audience or performing singing and dancing. By using depth of field reconstruction technology, the foreground characters can be highlighted and the cluttered background can be weakened, which can significantly reduce the interference caused by the background during live broadcast, thereby improving the quality of interaction. At the same time, it can also reduce the video transmission bit rate on the transcoding link side to save costs.

[0003] In related technologies, depth of field reconstruction technology uses foreground and background segmentation and simple blurring of the background, which easily leads to obvious segmentation marks between the foreground and background and creates a texture-like sense of fragmentation. At the same time, for continuous video images, when the foreground characters are moving, the edges of the foreground and background will produce temporal jitter, affecting the audience's video viewing experience and needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, storage medium and program product for reconstructing depth of field of an image, which solves the problem that the foreground and background are segmented and the background is simply blurred in the related technology, which easily leads to obvious segmentation marks of the foreground and background, resulting in a textured sense of fragmentation. At the same time, when the video is played, time domain jitter will be generated at the edge of the foreground and background, affecting the audience's video viewing experience. The original background image is blurred in layers in combination with depth information to increase the layering and realism of the picture, avoid uniform blurring of the background to produce a textured sense of fragmentation, have a soft depth of field effect, and at the same time, retain the clear details of the foreground, thereby improving the audience's video viewing experience.

[0005] In a first aspect, an embodiment of the present application provides a method for reconstructing image depth of field, the method comprising:

[0006] Acquire an image frame to be processed, perform foreground prediction on the image frame to be processed to obtain a foreground probability map, and segment the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map;

[0007] Performing enhancement processing on the original foreground image to obtain a foreground enhanced image;

[0008] Performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map;

[0009] The foreground enhancement map and the background blur map are fused based on the foreground probability map to obtain a depth of field reconstruction map.

[0010] In a second aspect, an embodiment of the present application further provides an image depth of field reconstruction device, the device comprising:

[0011] An acquisition module, configured to acquire an image frame to be processed;

[0012] An image segmentation module is configured to perform foreground prediction on the image frame to be processed to obtain a foreground probability map, and segment the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map;

[0013] a foreground enhancement module, configured to perform enhancement processing on the original foreground image to obtain a foreground enhanced image;

[0014] A background blur module is configured to perform depth detection on the image frame to be processed to obtain a depth estimation map, and perform layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map;

[0015] The depth of field reconstruction module is configured to fuse the foreground enhancement image with the background blur image based on the foreground probability map to obtain a depth of field reconstruction image.

[0016] In a third aspect, an embodiment of the present application further provides an image depth reconstruction device, the device comprising:

[0017] one or more processors;

[0018] a storage device configured to store one or more programs,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the image depth of field reconstruction method described in the embodiment of the present application.

[0020] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are configured to execute the image depth of field reconstruction method described in the embodiment of the present application.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the image depth of field reconstruction method described in the embodiment of the present application.

[0022] In an embodiment of the present application, by acquiring an image frame to be processed, foreground prediction is performed on the image frame to be processed to obtain a foreground probability map, and based on the foreground probability map, the image frame to be processed is segmented to obtain an original foreground image and an original background image; the original foreground image is enhanced to obtain a foreground enhancement image; depth detection is performed on the image frame to be processed to obtain a depth estimation image, and based on the depth estimation image and the foreground probability map, the original background image is layered blurred to obtain a background blur image; based on the foreground probability map, the foreground enhancement image and the background blur image are fused to obtain a depth of field reconstruction image. In the above scheme, by performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain the original foreground map and the original background map, the foreground part and the background part of the image frame to be processed can be effectively separated, while retaining the boundary part between the foreground and the background to a certain extent; by enhancing the original foreground map to obtain a foreground enhancement map, the details of the foreground area can be enhanced to improve the contrast; by performing layered blurring processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map, the depth information can be combined to achieve different levels of blurring effects of the original background map, increase the layering and realism of the picture, and avoid uniform blurring of the background to produce a texture-like sense of fragmentation; by fusing the foreground enhancement map with the background blur map to obtain a depth of field reconstruction map, the clear details of the foreground are retained, and a soft depth of field effect is provided, thereby improving the audience's video viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of an image depth reconstruction method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a process of performing depth of field reconstruction using an image depth of field reconstruction method provided in an embodiment of the present application;

[0025] Figure 3 A flowchart of an image depth reconstruction method including a process of determining a depth interval and a Gaussian convolution kernel corresponding to layered blur processing provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of an image depth reconstruction method including a process of performing layered blurring on an original background image provided in an embodiment of the present application;

[0027] Figure 5 A flowchart of an image depth reconstruction method including a process of performing foreground prediction on an image frame to be processed to obtain a foreground probability map provided by an embodiment of the present application;

[0028] Figure 6 A flowchart of an image depth reconstruction method including a process of performing depth detection on an image frame to be processed to obtain a depth estimation map provided in an embodiment of the present application;

[0029] Figure 7 A flowchart of an image depth reconstruction method including a process of enhancing an original foreground image provided in an embodiment of the present application;

[0030] Figure 8 A structural block diagram of an image depth reconstruction device provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of an image depth reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0033] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0034] The image depth reconstruction method provided in the embodiment of the present application can segment the image frame to be processed to obtain the original foreground image and the original background image, then enhance the original foreground image to obtain a foreground enhancement image, and detect the depth estimation image obtained by the image frame to be processed to perform layered blur processing on the original background image to obtain a background blur image, and finally fuse the foreground enhancement image and the background blur image to obtain a depth of field reconstructed image, which not only retains the clear details of the foreground, but also has a soft depth of field effect. Relevant application scenarios may include: video conferencing, video calls, indoor live broadcasts, outdoor live broadcasts, and short videos. The several application scenarios listed above are only exemplary and explanatory. In actual applications, the image depth of field reconstruction method can also be used in depth estimation tasks in other scenarios, and the embodiments of the present application are not limited to this. The image depth of field reconstruction method, device, equipment, storage medium and program product provided in the embodiments of the present application are intended to solve the problem that in the related technology, the foreground and background are segmented and the background is simply blurred, which easily leads to obvious segmentation marks between the foreground and the background, resulting in a texture-like sense of separation. At the same time, when the video is played, time domain jitter will be generated at the edge of the foreground and background, affecting the audience's video viewing experience.

[0035] In the image depth of field reconstruction method provided in the embodiment of the present application, the executor of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, and can also be servers and other devices, which are not limited in the embodiment of the present application.

[0036] Figure 1 This is a flow chart of an image depth reconstruction method provided in an embodiment of the present application. Figure 1 As shown, the following steps are included:

[0037] Step S101 , obtaining an image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map.

[0038] Among them, the image frame to be processed can be a video image that needs to be reconstructed in depth of field in application scenarios such as live video and short video, which is not limited in this application. The foreground prediction of the image frame to be processed is intended to separate the moving or changing objects or characters in the foreground area. The foreground probability map is the same size as the image frame to be processed, wherein the pixel value corresponding to each pixel is between 0 and 1, representing the probability that the pixel corresponding to the same position in the image frame to be processed belongs to the foreground. If the pixel value corresponding to the pixel is 1, it means that the pixel belongs completely to the foreground, if the pixel value corresponding to the pixel is 0, it means that the pixel belongs completely to the background, if the pixel value corresponding to the pixel is between 0 and 1, it means that the pixel belongs to the transition area between the foreground and the background. Optionally, the foreground prediction of the image frame to be processed can be performed by RVM (Robust Video Matting, Robust Video Matting), SAM (Segment Anything Model, All-purpose Segmentation Model) and other methods, which are not limited in this application. Therefore, based on the foreground probability map, the image frame to be processed can be reasonably segmented to obtain the original foreground image and the original background image.

[0039] In one embodiment, a segmentation threshold between 0 and 1 can be determined by using algorithms based on histogram analysis, maximum inter-class variance method, etc., and the foreground probability map is converted into a binary map based on the segmentation threshold. For example, a pixel value greater than the segmentation threshold is 1, and a pixel value less than or equal to the segmentation threshold is 0. At this time, 1 indicates that the pixel point belongs to the foreground area, and the foreground area includes a part of the area at the junction of the foreground and background, and 0 indicates that the pixel point belongs to the background area, and the background area also includes a part of the foreground and background areas. Then, the binary map can be extracted to obtain the original foreground map and the original background map, respectively. The original foreground map and the original background map both retain the transition area between the foreground and the background to a certain extent. In one embodiment, the specific process of segmenting the image frame to be processed based on the foreground probability map to obtain the original foreground map and the original background map is as follows: multiplying the pixel values ​​of the corresponding pixels of the foreground probability map and the image frame to be processed to obtain the original foreground map; subtracting the pixel values ​​of the corresponding pixels of the image frame to be processed from the original foreground map to obtain the original background map. Among them, since the pixel points with a pixel value of 0 in the foreground probability map completely belong to the background area, multiplying the pixel values ​​of the corresponding pixels of the foreground probability map and the image frame to be processed is equivalent to removing the pixel points that completely belong to the background area, and only retaining the foreground area and the transition area that is biased towards the foreground. Subtracting the pixel values ​​of the corresponding pixels of the video frame to be processed from the original foreground map is equivalent to removing the pixel points that completely belong to the foreground area, and only retaining the background area and the transition area that is biased towards the background. Therefore, by reasonably segmenting the original foreground map and the original background map, the transition information between the foreground and the background can be effectively retained, which can be beneficial for subsequent image processing to take into account the transition area between the foreground and the background, and is beneficial to improving the layering and realism of the depth of field reconstruction map.

[0040] Step S102: Enhance the original foreground image to obtain a foreground enhanced image.

[0041] Among them, taking live video as an example, the foreground area is usually the anchor character itself, and it is also the area of ​​interest that the audience pays attention to. Therefore, in order to improve the audience's viewing experience, the original foreground image can be enhanced to improve the clarity of the foreground details and edges, so that the characters or objects in the foreground are more vivid and vivid in the picture. In one embodiment, the enhancement processing of the original foreground image can use Laplace operator, Gaussian-Laplacian operator, Sobel operator, etc. for edge detection sharpening, which is not limited in this application. Thereby, the contrast and detail performance of the original foreground image are enhanced, and the texture and contour in the foreground are highlighted, thereby improving the overall image quality.

[0042] Step S103: performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map.

[0043] Among them, the depth of field reconstruction can achieve the blurring effect by blurring the background, but if the background is blurred with a uniform intensity, it is easy to cause the transition between the background and the foreground to appear unnatural, and there will be an obvious sense of pasting. Therefore, it is necessary to combine the depth information of the video frame to be processed and perform layered blurring on the background area. The depth estimation map may be a record of the depth information of each pixel in the image frame to be processed, and the depth information indicates the relative distance between the object surface corresponding to the pixel and the image acquisition device. For example, the pixel value in the depth estimation map may be between 0 and 1, 0 indicates the minimum depth, and 1 indicates the maximum depth. Optionally, the depth estimation can use open source deep learning models such as Depth Anything, DINOv2, MiDaS, and ZoeDepth for depth detection to obtain a depth estimation map corresponding to the image frame to be processed, which is not limited in this application. Furthermore, according to the depth information provided by the depth estimation map, the blurring intensity corresponding to different depth levels in the original background map is set accordingly, and background blurring with a depth effect can be achieved. Among them, for the background area closer to the image acquisition device, a weaker blur processing intensity can be set accordingly, and for the background area farther from the image acquisition device, a stronger blur processing intensity can be set accordingly, so that the background blur effect will gradually increase with the change of depth, forming a natural sense of hierarchy, and achieving better depth of field reconstruction quality. Optionally, the blur method used for blurring the original background image may include Gaussian blur, mean blur, median blur, bilateral blur, box blur, bokeh blur, etc., which are not limited in this application. In one embodiment, the foreground probability map can be multiplied by the pixel values ​​of the corresponding pixels of the depth estimation map to obtain a foreground depth estimation map, and the depth estimation map can be subtracted from the pixel values ​​of the corresponding pixels of the foreground depth estimation map to obtain a background depth estimation map. In one embodiment, the foreground probability map can be calculated by probability inversion to obtain a background probability map, and the background probability map can be multiplied by the pixel values ​​of the corresponding pixels of the depth estimation map to obtain a background depth estimation map. In one embodiment, the number of depth layers to be divided into by the original background image can be pre-set, and the corresponding blur intensity can be pre-set for each depth layer, wherein the greater the depth, the higher the blur intensity. Next, the total depth range obtained by the depth estimation map is divided evenly by length or frequency to match the number of depth layers, and the depth range corresponding to each depth layer is determined, wherein the range lengths of the multiple depth ranges obtained by the length even division are the same, and the number of pixels distributed in each depth range in the multiple depth ranges obtained by the frequency even division is the same. Finally, the original background image is divided into multiple depth regions corresponding to different depth ranges, and corresponding blur processing is performed on the pixels in different depth regions in the original background image.

[0044] Step S104: Based on the foreground probability map, the foreground enhancement map and the background blur map are merged to obtain a depth of field reconstruction map.

[0045] Among them, in the foreground probability map, if the pixel value is 1, it means that the pixel is located in the foreground area, if the pixel value is 0, it means that the pixel is located in the background area, and if the pixel value is between 0 and 1, it means that the pixel is located in the transition area between the foreground and the background. The larger the value, the greater the probability that the corresponding pixel belongs to the foreground area, and the smaller the value, the greater the probability that the corresponding pixel belongs to the background area. In one embodiment, the target foreground area and the target background area can be extracted from the foreground enhancement image and the background blur image respectively based on the foreground probability map, and a fixed fusion ratio is set to perform weighted fusion on the corresponding transition areas in the foreground enhancement image and the background blur image to obtain the target transition area. Finally, the target foreground area, the target background area and the target transition area are merged to obtain a depth of field reconstruction image. In one embodiment, the relevant formula for fusing the foreground enhancement image with the background blur image is as follows:

[0046] I final =I sharped ×alpha+B blured ×(1-alpha),

[0047] Among them, I final represents the depth of field reconstruction map, I sharped represents the foreground enhancement map, B blured represents the background blur map, and alpha represents the foreground probability map. Therefore, the area with pixel values ​​of 0 or 1 in the foreground probability map corresponds to the background area and the foreground area, and the transition area between the foreground area and the background area uses the pixel values ​​in the foreground probability map as reference weights to perform weighted average calculations on the transition areas corresponding to the foreground enhancement map and the background blur map to determine the final pixel value.

[0048] In one embodiment, Figure 2 A schematic diagram of a process of reconstructing depth of field by using an image depth of field reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, a depth estimation map 102 can be obtained by performing depth detection on the processed video frame 101, and a foreground probability map 103 can be obtained by performing foreground detection on the processed video frame 101. Then, the original background map 104 and the original foreground map 105 can be obtained by performing foreground and background segmentation on the processed video frame 101 based on the foreground probability map 103. Next, the original foreground map can be enhanced to obtain a foreground enhancement map 107, and the original background map can be layered blurred to obtain a background blur map 106. Finally, the foreground enhancement map and the background blur map are fused to obtain a depth of field reconstruction map 108.

[0049] In the above, by acquiring the image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, segmenting the image frame to be processed based on the foreground probability map to obtain the original foreground map and the original background map; enhancing the original foreground map to obtain a foreground enhanced map; performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blurring on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map; and merging the foreground enhancement map with the background blur map based on the foreground probability map to obtain a depth of field reconstruction map. In the above scheme, by performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain the original foreground map and the original background map, the foreground part and the background part of the image frame to be processed can be effectively separated, while retaining the boundary part between the foreground and the background to a certain extent; by enhancing the original foreground map to obtain a foreground enhancement map, the details of the foreground area can be enhanced to improve the contrast; by performing layered blurring processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map, the depth information can be combined to achieve different levels of blurring effects of the original background map, increase the layering and realism of the picture, and avoid uniform blurring of the background to produce a texture-like sense of fragmentation; by fusing the foreground enhancement map with the background blur map to obtain a depth of field reconstruction map, the clear details of the foreground are retained, and a soft depth of field effect is provided, thereby improving the audience's video viewing experience.

[0050] Figure 3 A flowchart of an image depth reconstruction method including a process of determining a depth interval and a Gaussian convolution kernel corresponding to layered blur processing provided by an embodiment of the present application. Figure 3 As shown, the following steps are included:

[0051] Step S201 , obtaining an image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map.

[0052] Step S202: Enhance the original foreground image to obtain a foreground enhanced image.

[0053] Step S203: perform depth detection on the image frame to be processed to obtain a depth estimation map, perform probability inversion calculation on the foreground probability map to obtain a background probability map, and multiply the pixel values ​​of corresponding pixels of the background probability map and the depth estimation map to obtain a background depth estimation map.

[0054] The probability inversion calculation of the foreground probability map may be performed by subtracting the pixel values ​​of corresponding pixels of the standard reference map whose pixel values ​​are all 1 from the foreground probability map to obtain the background probability map. The background depth estimation map may be a map that records the depth information of each pixel point in the background area and the transition area of ​​the image frame to be processed.

[0055] Step S204: performing extreme value statistics on the background depth estimation map to obtain a maximum depth value and a minimum depth value.

[0056] Among them, the maximum value statistics can be to compare the pixel values ​​corresponding to the pixel points in the background depth estimation map, and screen out the maximum pixel value and the minimum pixel value, and determine them as the maximum depth value and the minimum depth value, so as to obtain the depth distribution range corresponding to the background depth estimation map.

[0057] Step S205: Based on the maximum depth value and the minimum depth value, determine a plurality of depth intervals corresponding to a preset number of intervals, and assign a corresponding Gaussian convolution kernel to each depth interval.

[0058] Among them, the number of preset intervals can be the preset number of depth layers for dividing the original background image, and each depth interval represents the depth range of the corresponding depth layer. In one embodiment, the depth range between the maximum depth value and the minimum depth value can be divided into equal lengths according to the preset number of intervals. For example, the preset number of intervals is 4, which can be divided into 4 depth intervals of equal length. In one embodiment, the depth range between the maximum depth value and the minimum depth value can be divided into increasing or decreasing lengths according to the preset number of intervals. For example, the preset number of intervals is 4, which can be divided into 4 depth intervals with a length ratio of 1:2:3:4. For another example, the preset number of intervals is 3, which can be divided into 3 depth intervals with a length ratio of 4:4:3. For example, the maximum depth value is 0.8, the minimum depth value is 0.2, and the number of preset intervals is 4. The depth range [0.2, 0.8] can be divided into equal lengths to obtain 4 depth intervals, namely [0.2, 0.35], [0.35, 0.5], [0.5, 0.65], and [0.65, 0.8], corresponding to 4 different depth layers. Gaussian blur is used in the embodiment of the present application. In order to achieve the depth of field effect, the larger the depth value of the pixel area, the stronger the corresponding blur intensity, and the smaller the depth value of the pixel area, the weaker the corresponding blur intensity. Therefore, each depth interval needs to be assigned a corresponding Gaussian convolution kernel. Gaussian blur is an image processing method based on a Gaussian function. The image is convolved by a set Gaussian convolution kernel so that each pixel value in the image is affected by the surrounding pixel values ​​to some extent, thereby achieving a blur effect. The formula of the two-dimensional Gaussian function is as follows:

[0059]

[0060] Where x and y represent the horizontal and vertical coordinates of the pixel in the image, respectively, and σ represents the standard deviation. Based on the specified standard deviation σ and convolution kernel size, the weight of the corresponding Gaussian convolution kernel can be calculated. For example, if the standard deviation σ is set to 1 and the convolution kernel size is 3×3, the Gaussian convolution kernel can be calculated as follows:

[0061]

[0062] The intensity of Gaussian blur is determined by the standard deviation of the Gaussian function and the size of the Gaussian convolution kernel. The larger the standard deviation, the greater the influence of the blur weight of each pixel on the neighboring pixels with a greater distance, and the higher the blur intensity. The larger the size of the Gaussian convolution kernel, the more pixels are involved in the blur calculation, and the stronger the blur effect. For example, the four depth intervals from small to large are represented by d1, d2, d3, and d4, respectively. Among them, d1 corresponds to the smallest depth range, and d4 corresponds to the largest depth range. The depth interval corresponds to the set Gaussian convolution kernel standard deviation σ and convolution kernel size kernel. size as follows:

[0063] d1:σ=1.0,kernel size =(3,3),

[0064] d2:σ=2.0,kernel size =(5,5),

[0065] d1:σ=3.0,kernel size =(7,7),

[0066] d1:σ=4.0,kernel size =(9,9).

[0067] The specific division of the aforementioned depth intervals and the specific setting and allocation of the Gaussian convolution kernel are described as examples. Developers can make adaptive adjustments based on the depth of field effect requirements and computing resource limitations of actual application scenarios, and this application does not limit them.

[0068] Step S206: performing layered blur processing on the original background image according to the depth interval, the background depth estimation image and the Gaussian convolution kernel to obtain a background blur image.

[0069] Among them, the pixel areas corresponding to different depth intervals in the original background image can be determined based on the depth range of the depth interval and the background depth estimation map, and blurring can be performed based on the Gaussian convolution kernel corresponding to the pixel area, thereby realizing layered blurring of the original background image and achieving a background blur effect.

[0070] Step S207: Based on the foreground probability map, the foreground enhancement map and the background blur map are merged to obtain a depth of field reconstruction map.

[0071] As mentioned above, by dividing the depth interval based on the depth information of the depth estimation map and assigning corresponding Gaussian convolution kernels to different depth intervals, the appropriate blur intensity can be effectively matched according to different depth sizes, so that the background blur effect can be gradually enhanced as the depth increases, forming a natural sense of layering.

[0072] Figure 4 A flowchart of an image depth reconstruction method including a process of performing layered blurring on an original background image is provided in an embodiment of the present application. Figure 4 As shown, the following steps are included:

[0073] Step S301 , obtaining an image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map.

[0074] Step S302: Enhance the original foreground image to obtain a foreground enhanced image.

[0075] Step S303: perform depth detection on the image frame to be processed to obtain a depth estimation map, perform probability inversion calculation on the foreground probability map to obtain a background probability map, and multiply the pixel values ​​of corresponding pixels of the background probability map and the depth estimation map to obtain a background depth estimation map.

[0076] Step S304: Perform extreme value statistics on the background depth estimation map to obtain a maximum depth value and a minimum depth value.

[0077] Step S305: Based on the maximum depth value and the minimum depth value, a plurality of depth intervals corresponding to a preset number of intervals are determined, and a corresponding Gaussian convolution kernel is allocated to each depth interval.

[0078] Step S306: Determine a mask map corresponding to each depth interval based on the depth interval and the background depth estimation map.

[0079] Among them, since the pixel areas corresponding to different depth intervals in the original background image are irregular, in order to facilitate the subsequent layered blurring process, a mask image corresponding to each depth interval can be created. For a certain depth interval, the pixel values ​​in the background depth estimation image that are located in the depth interval can be set to 1, and the pixel values ​​in the background depth estimation image that are not located in the depth interval can be set to 0, thereby obtaining a mask image corresponding to the depth interval.

[0080] Step S307: Based on the Gaussian convolution kernel corresponding to each depth interval, a convolution calculation is performed on the original background image to obtain a background blur image corresponding to each depth interval.

[0081] Among them, based on the mask image corresponding to each depth interval determined above, the original background image can be convolved based on the Gaussian convolution kernels corresponding to different depth intervals, thereby obtaining multiple background blur images corresponding to different blur intensities.

[0082] Step S308: Based on the mask image, the blurred background image is fused to obtain a blurred background image.

[0083] Among them, based on the mask image corresponding to each depth interval, the blurred pixel area with a depth value in the depth interval can be extracted from the corresponding background blur image, and the final background blur image is obtained by superposition and fusion. For example, the number of depth intervals is 4. After multi-layer blur processing, the multiple background blur images are fused to obtain the background blur image. The relevant formula is as follows:

[0084] B blured =D1×mask1+D2×mask2+D3×mask3+D4×mask4,

[0085] Among them, B blured represents the background blur map, D1, D2, D3 and D4 represent the background blur map, and mask1, mask2, mask3 and mask4 represent the mask map.

[0086] Step S309: Based on the foreground probability map, the foreground enhancement map and the background blur map are merged to obtain a depth of field reconstruction map.

[0087] As described above, by blurring the original background image based on the Gaussian convolution kernel corresponding to each depth interval, multiple background blur images corresponding to the blur effects of different depth levels can be obtained. By fusing the background blur images based on the mask image to obtain the background blur image, the multi-level blur results of the background area can be effectively fused, making the transition between the background and foreground in the depth of field reconstruction image more natural, thereby enhancing the spatial sense and layering of the image.

[0088] Figure 5 A flowchart of an image depth reconstruction method including a process of performing foreground prediction on an image frame to be processed to obtain a foreground probability map is provided in an embodiment of the present application. Figure 5 As shown, the following steps are included:

[0089] Step S401: Obtain an image frame to be processed.

[0090] Step S402: convert the image frame to be processed into a preset format and downsample it to obtain a model input image.

[0091] Among them, since the image formats corresponding to the image frames to be processed from different sources may be different, in order to facilitate the unified processing of the foreground prediction model set in this embodiment, the image frames to be processed can be converted to a preset format first, and the image format of the input image of the foreground prediction model can be unified. For example, the read image frames to be processed can be uniformly converted to RGB format, which is conducive to the processing of the foreground prediction model. In addition, in order to reduce the computing power pressure related to the deployment of the foreground prediction model, the image frames to be processed can be downsampled and cropped to a uniform size, for example, 512×512, thereby reducing the image resolution so that the foreground prediction model can be processed faster.

[0092] Step S403: input the model input image into the trained foreground prediction model to obtain a probability result graph.

[0093] Among them, the prospect prediction model can be constructed by methods such as RVM (Robust Video Matting) and SAM (Segment Anything Model), which are not limited in this application. Optionally, taking the video live broadcast scene as an example, in order to train the constructed prospect prediction model, an image data set containing a variety of different foregrounds can be pre-established, covering relevant business data and a variety of data collected from the Internet, wherein 90% of the images can contain the whole body of a person or part of the body of a person. For example, the established image data set includes 20,000 images, of which 10,000 images can be divided into a training set, 5,000 images can be divided into a verification set, and 5,000 images can be divided into a test set. Then, the image data set is annotated with foregrounds to obtain data pairs of the original image and the corresponding foreground image. Then, the constructed prospect prediction model is tuned and trained based on the training set to improve the model's prediction ability for the business data set, the training accuracy of the model is monitored in real time during the training process based on the verification set, and the prospect prediction models using different training methods are uniformly tested for accuracy based on the test set, the prediction effects of different prospect prediction models are evaluated, and the final prospect prediction model is selected. Based on the trained foreground prediction model, the model input image can be fully feature extracted and then a low-resolution probability result map can be output, in which the pixel values ​​are between 0 and 1.

[0094] Step S404: upsample the probability result map to obtain a foreground probability map.

[0095] After obtaining the probability result map output by the foreground prediction model, the probability result map needs to be upsampled to restore it to the same image size as the image frame to be processed to obtain the final foreground probability map. Optionally, the probability result map can be upsampled using bilinear interpolation.

[0096] Step S405 : segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground image and an original background image.

[0097] Step S406: Enhance the original foreground image to obtain a foreground enhanced image.

[0098] Step S407: performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map.

[0099] Step S408: Based on the foreground probability map, the foreground enhancement map and the background blur map are merged to obtain a depth of field reconstruction map.

[0100] As described above, by performing foreground prediction based on the trained foreground prediction model to obtain a foreground probability map, the foreground area in the image frame to be processed can be accurately identified, which is conducive to the subsequent accurate foreground and background segmentation of the image frame to be processed and ensures the depth of field effect of the reconstructed image.

[0101] Figure 6 A flowchart of an image depth reconstruction method including a process of performing depth detection on an image frame to be processed to obtain a depth estimation map is provided in an embodiment of the present application. Figure 6 As shown, the following steps are included:

[0102] Step S501 : obtaining an image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map.

[0103] Step S502: Enhance the original foreground image to obtain a foreground enhanced image.

[0104] Step S503: down-sample the image frame to be processed.

[0105] Among them, by downsampling the image frame to be processed, the size of the image frame to be processed can be scaled to the size required by the model, which is convenient for image processing of the model.

[0106] Step S504: input the downsampled image frame to be processed into the trained lightweight depth estimation model to obtain a depth result map.

[0107] Among them, in order to meet the lightweight deployment requirements of mobile terminals, a lightweight depth estimation model is set in this embodiment. The lightweight depth estimation model can be constructed using a neural network framework of a joint encoder and decoder. The encoder part can be used to extract features of the image frame to be processed, and the decoder part can be used to reconstruct and restore the image based on the extracted features. In one embodiment, the encoder part of the lightweight depth estimation model can be constructed using a lightweight network MobileNetV3-small, and the decoder part can be constructed using a semantic segmentation method SINet, so that the lightweight depth estimation model can use the confidence score of the low-resolution image to suppress the noise of the high-resolution image while retaining the global information of the high-resolution image, thereby reconstructing a higher quality depth result image. Optionally, in order to train the lightweight depth estimation model, an image sequence data set with rich scenes can be constructed. For example, 10,000 videos can be collected, including business data and videos collected from the Internet, and 5 consecutive frames are extracted from each video as an image sequence, a total of 50,000 images. Then, the 50,000 images are depth-detected using the deep learning model Depth Anything to obtain a depth map, and the obtained depth map is set as the reference true value of the training result, thereby obtaining a training data pair of the original image-depth map. In one embodiment, two stages can be used to train the lightweight depth estimation model. In the first stage, a single-frame image can be used for model training. For example, 80% of the images are randomly selected from the image sequence data set as the training set and 20% of the images are used as the verification set. The encoder part can be trained using preloaded parameters, and the learning rate can be set to 0.001; in the second stage, multiple frames of images can be used for model training, and the model parameters trained in the first stage are used for preloading, and the learning rate is set to 0.0001. After two stages of training, the loss index and accuracy index of the verification set are monitored in real time during training, and the lightweight depth estimation model with the best accuracy index after loss convergence is finally determined. Based on the trained lightweight depth estimation model, the depth detection of the downsampled image frame to be processed can be performed to obtain a depth result map. The pixel value in the depth result map can be a relative depth value between 0 and 1, 0 represents the minimum depth, and 1 represents the maximum depth.

[0108] Step S505: upsample the depth result map to obtain a depth estimation map.

[0109] Among them, after obtaining the depth result map output by the lightweight depth estimation model, the depth result map needs to be upsampled and restored to the same image size as the image frame to be processed to obtain the final depth estimation map.

[0110] Step S506: According to the depth estimation map and the foreground probability map, the original background map is subjected to layered blurring processing to obtain a background blur map.

[0111] Step S507: Based on the foreground probability map, the foreground enhancement map and the background blur map are merged to obtain a depth of field reconstruction map.

[0112] As mentioned above, by performing depth detection based on the trained lightweight depth estimation model to obtain a depth estimation map, the relative depth value of each pixel in the video frame to be processed can be accurately estimated, and the relative distance information between the object surface corresponding to each pixel and the image acquisition device can be provided, providing basic data support for subsequent blur processing, while meeting the lightweight deployment requirements of mobile terminals.

[0113] Figure 7 The present invention provides a flowchart of a method for reconstructing depth of field of an image including a process of enhancing an original foreground image. Figure 7 As shown, the following steps are included:

[0114] Step S601: Acquire an image frame to be processed, perform foreground prediction on the image frame to be processed to obtain a foreground probability map, and segment the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map.

[0115] Step S602: Filter the original foreground image, and extract high-frequency components from the filtered original foreground image to obtain a high-frequency feature image.

[0116] Among them, taking the video live broadcast scene as an example, since many links in the video live broadcast system, such as video acquisition, compression encoding, network transmission, decoding and playback, may introduce noise, the original foreground image can be subjected to noise reduction processing before the high-frequency component of the original foreground image is extracted to avoid noise enhancement. Optionally, mean filtering, median filtering, bilateral filtering, etc. can be used, which is not limited in this application. In one embodiment, a bilateral filtering method can be used for noise reduction processing, wherein the core idea of ​​bilateral filtering is to perform a weighted average calculation on each pixel in the image based on spatial distance and pixel similarity. Spatial distance measures the spatial distribution and structural information of the image at the pixel level, reflecting the image's edge, texture, shape and other features; pixel similarity uses the pixel's value domain information for comparison, reflecting the differences in the image's grayscale level, contrast, brightness and other characteristics. The greater the difference, the greater the possibility that it is noise. Therefore, bilateral filtering is a filter that can both retain edge details and remove noise, and the relevant formula is as follows:

[0117]

[0118] Among them, I filtered (x, y) represents the pixel value after filtering, I(x, y) represents the pixel value in the original image, and W p(x, y) represents the normalized weight, which is used to normalize the filtering result, Ω represents the fixed-size neighborhood window of the filter, and K spatial (i, j) represents the spatial domain kernel function, which is used to calculate the spatial distance weight between pixels, K range (I(x,y),I(x+i,y+j)) represents the pixel value domain kernel function, which is used to calculate the similarity weights between pixel values.

[0119] Among them, the Laplace operator, Gaussian-Laplacian operator, Sobel operator, etc. can be used to extract the high-frequency component of the filtered original foreground image, which is not limited in this application. In one embodiment, the second-order derivative calculation of the pixels in the image can be performed based on the Laplace operator to extract the high-frequency component of the detail layer in the image. For example, the Laplace operator can be implemented based on the following formula:

[0120] F deitail =I fiutered *kermel,

[0121]

[0122] Among them, F deitail represents the high-frequency feature map, I filtered Represents the original foreground image after filtering, and kernel represents the convolution kernel.

[0123] Step S603: multiply the high-frequency feature map by a preset strength coefficient to obtain a high-frequency enhancement map.

[0124] Among them, multiplying the high-frequency feature map with the preset strength coefficient can be used to amplify the detail information of the image, and by controlling the value of the preset strength coefficient, the sharpness of the image can be controlled. For example, the relevant formula for calculating the high-frequency enhancement map is as follows:

[0125] F ed =α·F deitail ,

[0126] Among them, F ed Represents the high frequency enhancement map, F deitail represents a high-frequency feature map, α represents a preset intensity coefficient, and for example, α=0.3.

[0127] Step S604: Add pixel values ​​of corresponding pixels of the high-frequency enhanced image and the original foreground image to obtain a foreground enhanced image.

[0128] For example, the relevant formula for calculating the foreground enhancement map is as follows:

[0129] I sharped d=I+F ed ,

[0130] Among them, I is the original foreground image, F ed is the high frequency enhancement image, I sharped Enhance the foreground image.

[0131] Step S605: perform depth detection on the image frame to be processed to obtain a depth estimation map, and perform layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map.

[0132] Step S606: Based on the foreground probability map, the foreground enhancement map and the background blur map are merged to obtain a depth of field reconstruction map.

[0133] As described above, by enhancing the original foreground image, the clarity of the foreground details and edges can be effectively improved, making the characters or objects in the foreground area more vivid and lively in the picture, thereby improving the overall texture.

[0134] Figure 8 This is a structural block diagram of an image depth reconstruction device provided in an embodiment of the present application. The device is configured to execute the image depth reconstruction method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Figure 8 As shown, the device comprises:

[0135] An acquisition module 201 is configured to acquire an image frame to be processed;

[0136] The image segmentation module 202 is configured to perform foreground prediction on the image frame to be processed to obtain a foreground probability map, and segment the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map;

[0137] The foreground enhancement module 203 is configured to perform enhancement processing on the original foreground image to obtain a foreground enhanced image;

[0138] Background blur module 204, configured to perform depth detection on the image frame to be processed to obtain a depth estimation map, and perform layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map;

[0139] The depth of field reconstruction module 205 is configured to fuse the foreground enhancement image with the background blur image based on the foreground probability map to obtain a depth of field reconstruction image.

[0140] In the above, by acquiring the image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, segmenting the image frame to be processed based on the foreground probability map to obtain the original foreground map and the original background map; enhancing the original foreground map to obtain a foreground enhanced map; performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blurring on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map; and merging the foreground enhancement map with the background blur map based on the foreground probability map to obtain a depth of field reconstruction map. In the above scheme, by performing foreground prediction on the image frame to be processed to obtain a foreground probability map, and segmenting the image frame to be processed based on the foreground probability map to obtain the original foreground map and the original background map, the foreground part and the background part of the image frame to be processed can be effectively separated, while retaining the boundary part between the foreground and the background to a certain extent; by enhancing the original foreground map to obtain a foreground enhancement map, the details of the foreground area can be enhanced to improve the contrast; by performing layered blurring processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map, the depth information can be combined to achieve different levels of blurring effects of the original background map, increase the layering and realism of the picture, and avoid uniform blurring of the background to produce a texture-like sense of fragmentation; by fusing the foreground enhancement map with the background blur map to obtain a depth of field reconstruction map, the clear details of the foreground are retained, and a soft depth of field effect is provided, thereby improving the audience's video viewing experience.

[0141] In a possible embodiment, the background blur module 204 is further configured to:

[0142] The foreground probability map is inverted to obtain a background probability map, and the background probability map is multiplied by the pixel values ​​of the corresponding pixels in the depth estimation map to obtain a background depth estimation map;

[0143] Perform extreme value statistics on the background depth estimation map to obtain the maximum depth value and the minimum depth value;

[0144] Based on the maximum depth value and the minimum depth value, a plurality of depth intervals corresponding to a preset number of intervals are determined, and a corresponding Gaussian convolution kernel is assigned to each depth interval;

[0145] According to the depth interval, the background depth estimation map and the Gaussian convolution kernel, the original background image is layered blurred to obtain a background blur image.

[0146] In a possible embodiment, the background blur module 204 is further configured to:

[0147] Based on the depth intervals and the background depth estimation map, determine a mask map corresponding to each depth interval;

[0148] Based on the Gaussian convolution kernel corresponding to each depth interval, the original background image is convolved to obtain the background blur image corresponding to each depth interval;

[0149] The blurred background image is fused based on the mask image to obtain the blurred background image.

[0150] In a possible embodiment, performing foreground prediction on the image frame to be processed to obtain a foreground probability map includes:

[0151] Perform preset format conversion and down-sampling on the image frame to be processed to obtain a model input image;

[0152] Input the model input image into the trained prospect prediction model to obtain a probability result graph;

[0153] The probability result map is upsampled to obtain the foreground probability map.

[0154] In a possible embodiment, the background blur module 204 is further configured to:

[0155] Down-sampling the image frame to be processed;

[0156] The downsampled image frame to be processed is input into the trained lightweight depth estimation model to obtain a depth result map;

[0157] The depth result map is upsampled to obtain the depth estimation map.

[0158] In a possible embodiment, the foreground enhancement module 203 is further configured to:

[0159] Filtering the original foreground image, and extracting high-frequency components from the filtered original foreground image to obtain a high-frequency feature image;

[0160] Multiplying the high-frequency feature map with a preset strength coefficient to obtain a high-frequency enhancement map;

[0161] The pixel values ​​of corresponding pixels in the high-frequency enhancement image and the original foreground image are added to obtain the foreground enhancement image.

[0162] In a possible embodiment, the depth of field reconstruction module 205 is further configured to:

[0163] Multiply the pixel values ​​of the corresponding pixels of the foreground probability map and the image frame to be processed to obtain the original foreground map;

[0164] The pixel values ​​of the corresponding pixels of the image frame to be processed and the original foreground image are subtracted to obtain the original background image.

[0165] Fig. 9 A schematic diagram of the structure of an image depth reconstruction device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device includes a processor 301, a memory 302, an input device 303 and an output device 304; the number of processors 301 in the device can be one or more. Fig. 9 A processor 301 is taken as an example; the processor 301, memory 302, input device 303 and output device 304 in the device can be connected by a bus or other means. Fig. 9 The example of connecting through a bus is taken. The memory 302, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image depth of field reconstruction method in the embodiment of the present application. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 302, that is, implements the above-mentioned image depth of field reconstruction method. The input device 303 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 304 may include a display device such as a display screen.

[0166] An embodiment of the present application also provides a non-volatile storage medium containing computer executable instructions, which, when executed by a computer processor, is configured to execute an image depth reconstruction method described in the above embodiment, which includes: acquiring an image frame to be processed, performing foreground prediction on the image frame to be processed to obtain a foreground probability map, segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground image and an original background image; performing enhancement processing on the original foreground image to obtain a foreground enhancement image; performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blur processing on the original background image according to the depth estimation map and the foreground probability map to obtain a background blur map; and fusing the foreground enhancement map with the background blur map based on the foreground probability map to obtain a depth reconstruction map.

[0167] It is worth noting that in the embodiment of the above-mentioned image depth of field reconstruction device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.

[0168] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is configured to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the image depth reconstruction method described in the embodiment of the present application. The program product may be implemented in any combination of one or more readable media.

Claims

1. A method for reconstructing image depth of field, characterized in that: include: Acquire an image frame to be processed, perform foreground prediction on the image frame to be processed to obtain a foreground probability map, and segment the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map; Performing enhancement processing on the original foreground image to obtain a foreground enhanced image; Performing depth detection on the image frame to be processed to obtain a depth estimation map, and performing layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map; The foreground enhancement map and the background blur map are fused based on the foreground probability map to obtain a depth of field reconstruction map.

2. The image depth reconstruction method according to claim 1, characterized in that: The step of performing layered blurring processing on the original background image to obtain a background blur image according to the depth estimation image and the foreground probability image includes: Performing probability inversion calculation on the foreground probability map to obtain a background probability map, and multiplying pixel values ​​of corresponding pixels of the background probability map and the depth estimation map to obtain a background depth estimation map; Performing extreme value statistics on the background depth estimation map to obtain a maximum depth value and a minimum depth value; Based on the maximum depth value and the minimum depth value, determining a plurality of depth intervals corresponding to a preset number of intervals, and assigning a corresponding Gaussian convolution kernel to each of the depth intervals; According to the depth interval, the background depth estimation map and the Gaussian convolution kernel, the original background image is subjected to layered blurring processing to obtain a background blurred image.

3. The image depth reconstruction method according to claim 2, characterized in that: The step of performing layered blurring processing on the original background image to obtain a background blur image according to the depth interval, the background depth estimation image and the Gaussian convolution kernel includes: Based on the depth intervals and the background depth estimation map, determining a mask map corresponding to each of the depth intervals; Based on the Gaussian convolution kernel corresponding to each of the depth intervals, convolution calculation is performed on the original background image to obtain a background blur image corresponding to each of the depth intervals; The background blur image is fused based on the mask image to obtain a background blur image.

4. The image depth reconstruction method according to claim 1, characterized in that: The step of performing foreground prediction on the image frame to be processed to obtain a foreground probability map includes: Performing preset format conversion and downsampling on the image frame to be processed to obtain a model input image; Inputting the model input image into the trained prospect prediction model to obtain a probability result graph; The probability result map is upsampled to obtain a foreground probability map.

5. The image depth reconstruction method according to claim 1, characterized in that: The performing depth detection on the image frame to be processed to obtain a depth estimation map includes: Down-sampling the image frame to be processed; Inputting the downsampled image frame to be processed into the trained lightweight depth estimation model to obtain a depth result map; The depth result map is upsampled to obtain a depth estimation map.

6. The image depth reconstruction method according to claim 1, characterized in that: The step of performing enhancement processing on the original foreground image to obtain a foreground enhanced image includes: Filtering the original foreground image, and extracting high-frequency components from the filtered original foreground image to obtain a high-frequency feature image; Multiplying the high-frequency feature map by a preset strength coefficient to obtain a high-frequency enhancement map; The pixel values ​​of corresponding pixels of the high-frequency enhancement image and the original foreground image are added to obtain a foreground enhancement image.

7. The image depth reconstruction method according to claim 1, characterized in that: The step of segmenting the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map includes: Multiplying the foreground probability map by the pixel values ​​of corresponding pixels of the image frame to be processed to obtain an original foreground map; The pixel values ​​of corresponding pixels of the image frame to be processed and the original foreground image are subtracted to obtain the original background image.

8. An image depth reconstruction device, characterized in that: include: An acquisition module, configured to acquire an image frame to be processed; An image segmentation module is configured to perform foreground prediction on the image frame to be processed to obtain a foreground probability map, and segment the image frame to be processed based on the foreground probability map to obtain an original foreground map and an original background map; a foreground enhancement module, configured to perform enhancement processing on the original foreground image to obtain a foreground enhanced image; A background blur module is configured to perform depth detection on the image frame to be processed to obtain a depth estimation map, and perform layered blur processing on the original background map according to the depth estimation map and the foreground probability map to obtain a background blur map; The depth of field reconstruction module is configured to fuse the foreground enhancement image with the background blur image based on the foreground probability map to obtain a depth of field reconstruction image.

9. An image depth reconstruction device, the device comprising: one or more processors; A storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image depth of field reconstruction method described in any one of claims 1-7.

10. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the image depth reconstruction method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image depth reconstruction method according to any one of claims 1 to 7 is implemented.

Images