Background blur method, device, computer equipment and storage medium

Abstract

The present application relates to a background blur method, device, computer equipment and storage medium, and relates to the field of image processing technology. The method comprises: determining the depth map corresponding to each video frame in the original video; for each video frame, performing optical flow estimation on the video frame and the adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames; wherein the adjacent video frames are adjacent to the video frames, and the optical flow map reflects the movement of pixels in the video frame and the adjacent video frames; smoothing the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map; performing background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame to obtain a background blur image corresponding to each video frame; and obtaining a target video according to the background blur images corresponding to each video frame. The use of this method can effectively solve the limitations of background blur.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a background blurring method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the continuous development of computer technology and multimedia technology, image background blurring methods are widely used in image processing. The purpose of blurring an image is to achieve a greater depth of field, for example, to enhance the image's aesthetics or safety.

[0003] Currently, background blurring methods can usually only blur the background of a single-frame image based on its depth information, and cannot be used in scenarios where background blurring is performed on a video. Therefore, there are significant limitations. Summary of the Invention

[0004] Based on this, it is necessary to provide a background blur method, device, computer equipment and storage medium that can blur the background of a video to address the above technical problems, so as to effectively solve the limitations of background blur.

[0005] In a first aspect, the present application provides a background blurring method. The method comprises:

[0006] Determine the depth map corresponding to each video frame in the original video;

[0007] For each video frame, optical flow estimation is performed on the video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames; wherein the adjacent video frames are adjacent to the video frame, and the optical flow map reflects the motion of pixels in the video frame and the adjacent video frames;

[0008] Smoothing the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map;

[0009] Perform background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame to obtain a background blur image corresponding to each video frame;

[0010] The target video is obtained according to the background blur images corresponding to each video frame.

[0011] In a second aspect, the present application further provides a background blurring device. The device comprises:

[0012] An image determination unit, configured to determine a depth map corresponding to each video frame in the original video;

[0013] an optical flow estimation unit, configured to perform optical flow estimation on each video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames; wherein the adjacent video frames are adjacent to the video frames, and the optical flow map reflects the motion of pixels in the video frame and the adjacent video frames;

[0014] An image smoothing unit is used to smooth the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map;

[0015] A background blur unit is used to perform background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame to obtain a background blur image corresponding to each video frame;

[0016] The video determination unit is used to obtain a target video according to the background blur images corresponding to each video frame.

[0017] In some embodiments, the image determination unit is also used to perform depth estimation on each video frame in the original video to obtain an initial depth map corresponding to each video frame; reconstruct a three-dimensional scene model based on each initial depth map and preset image acquisition parameters; determine the plane area in each video frame; perform planar correction on the three-dimensional scene model based on the plane area in each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame; and determine the depth map corresponding to each video frame based on each corrected three-dimensional scene model.

[0018] In some embodiments, the image determination unit is further configured to map the planar region in each video frame to a three-dimensional space based on the image acquisition parameters to obtain regional point cloud data corresponding to each video frame;

[0019] In three-dimensional space, the point cloud data of the three-dimensional scene model is used to perform point cloud registration on the regional point cloud data corresponding to each video frame to obtain the registered point cloud data corresponding to each video frame; the registered point cloud data corresponding to each video frame is merged into the point cloud data of the three-dimensional scene model to obtain the fused point cloud data corresponding to each video frame; the three-dimensional scene model is reconstructed according to the fused point cloud data corresponding to each video frame to obtain the corrected three-dimensional scene model corresponding to each video frame.

[0020] In some embodiments, the image determination unit is further configured to map each corrected three-dimensional scene model to a two-dimensional space according to the image acquisition parameters to obtain a depth map corresponding to each video frame.

[0021] In some embodiments, the image smoothing unit is further configured to determine, based on the optical flow map, corresponding pixel positions of each pixel of the video frame in adjacent video frames;

[0022] Mapping each pixel of the video frame to the adjacent video frame according to the corresponding pixel position of each pixel in the adjacent video frame to obtain an optical flow map;

[0023] The depth map is weighted and smoothed according to the optical flow map to obtain a depth smoothing map.

[0024] In some embodiments, the depth information in the depth smoothing map includes the pixel depth value of each pixel in the depth smoothing map, and the background blur unit is further used to determine, for each depth smoothing map, a blur weight factor corresponding to each pixel in the depth smoothing map based on the difference between the pixel depth value of each pixel in the depth smoothing map and the average depth value; wherein the average depth value is the average value of the pixel depth values ​​of each pixel in a specified focus area of ​​the depth smoothing map, and the blur weight factor is used to indicate the degree of blur of each pixel; for each video frame, the video frame is subjected to background blur processing according to the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame to obtain a background blur image corresponding to the video frame.

[0025] In some embodiments, the background blurring unit is further configured to determine, for each video frame, a blur kernel size corresponding to each pixel in the video frame according to a blurring weight factor corresponding to each pixel in the depth smoothing map of each video frame;

[0026] The video frames are blurred to different degrees according to the blur kernel size corresponding to each pixel in the video frame to obtain the background blurred image corresponding to the video frame.

[0027] In a third aspect, the present application further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned video segmentation method when executing the computer program.

[0028] In a fourth aspect, the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned video segmentation method are implemented.

[0029] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in the above-mentioned video segmentation method.

[0030] The above-mentioned background blurring method, device, computer equipment, storage medium and computer program product determine the depth map corresponding to each video frame in the original video; for each video frame, perform optical flow estimation on the video frame and adjacent video frames adjacent to the video frame to obtain an optical flow map reflecting the motion of each pixel in the video frame and the adjacent video frames, which can increase the stability of the background blurring of each video frame; smooth the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map, so that the blurring effect between each video frame is smoother; based on the depth information in the depth smoothing map of each video frame, perform background blurring on the video frame to obtain a background blurring image corresponding to each video frame; based on the background blurring image corresponding to each video frame, obtain the target video. Since the depth map of each video frame has undergone optical flow and smoothing processing in the above process, the target video obtained after background blurring based on each depth map is more coherent and stable. That is, the background blurring of the present application can not only blur the background of the video, effectively solve the limitations of background blurring, but also improve the stability of the generated target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic flow chart of a background blurring method in some embodiments;

[0032] Figure 2 Schematic diagram of the process of background blurring method in other embodiments;

[0033] Figure 3 Schematic diagram of the process of background blurring method in other embodiments;

[0034] Figure 4 is a structural block diagram of a background blur device in some embodiments;

[0035] Figure 5 Internal structure diagrams of some computer devices provided in embodiments of the present application;

[0036] Figure 6 Internal structure diagrams of other computer devices provided in embodiments of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0038] In some embodiments, as Figure 1As shown, a video segmentation method is provided. This embodiment uses the method applied to a computer device as an example. It is understood that the computer device can be a server or a terminal. The method can be implemented by the server or terminal alone, or through interaction between the server and the terminal. In this embodiment, the method includes the following steps:

[0039] Step 102: Determine the depth map corresponding to each video frame in the original video.

[0040] The original video can be split into multiple video frames, and each video frame is a single image picture that is the smallest unit in the original video, which is usually represented in the form of an image.

[0041] A depth map, also known as a distance map, refers to an image that uses the depth or distance values ​​of each point in the scene captured by an image acquisition device as pixel values. It directly reflects the geometric shape of the visible surface in the scene.

[0042] Specifically, the computer device can directly obtain the depth maps corresponding to each predetermined video frame in the original video, and can also perform depth estimation on each video frame in the original video to obtain the depth maps corresponding to each video frame. Each depth map carries different depth information, i.e., depth information.

[0043] Step 104 : For each video frame, perform optical flow estimation on the video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames.

[0044] The adjacent video frame is adjacent to the video frame, and the adjacent video frame may be the previous frame of the video frame or the next frame of the video frame.

[0045] Optical flow refers to the speed of pattern movement in a time-varying image. When an object moves, the brightness pattern of its corresponding points in the image also moves. Optical flow captures changes in an image and, because it contains information about the object's motion, is used to determine its motion.

[0046] Optical flow estimation refers to estimating the optical flow between two frames based on two consecutive frames of images.

[0047] The optical flow map reflects the movement of pixels in a video frame and adjacent video frames. That is, the optical flow map can reflect the distance that a pixel in a video frame needs to move in at least one direction, such as vertically or horizontally, to reach the corresponding pixel position in an adjacent video frame. It can be understood that if the video frame is the t-th frame, the previous frame of the video frame is the t-1-th frame, and the next frame of the video frame is the t+1 frame, then the optical flow map between the t-th frame and the t-1-th frame can reflect how much horizontal and vertical movement the pixels of the t-1-th frame need to take to reach the corresponding pixel position of the t-th frame, and the optical flow map between the t-th frame and the t+1-th frame can reflect how much horizontal and vertical movement the pixels of the t+1-th frame need to take to reach the corresponding pixel position of the t-th frame.

[0048] Specifically, for each video frame, the computer device may perform optical flow estimation on the video frame and the frame preceding the video frame to obtain a reverse optical flow map from the video frame to the frame preceding the video frame. Alternatively, for each video frame, the computer device may also perform optical flow estimation on the video frame and the frame following the video frame to obtain a forward optical flow map from the video frame to the frame following the video frame. In other words, the computer device may perform optical flow estimation to obtain at least one of a reverse optical flow map and a forward optical flow map.

[0049] It can be understood that when the video frame is the first frame, since the video frame has no previous frame, the optical flow estimation is usually performed on the video frame and the next frame to obtain the forward optical flow map from the video frame to the next frame. When the video frame is the last frame, since the video frame has no next frame, the optical flow estimation is usually performed on the video frame and the previous frame to obtain the reverse optical flow map from the video frame to the previous frame.

[0050] Step 106 : Smoothing the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map.

[0051] Specifically, for each video frame, the computer device may map the reverse optical flow map between the video frame and the previous frame back to the video frame to obtain an updated video frame, and smooth the depth map corresponding to the video frame based on the updated video frame to obtain a depth smoothing map. Alternatively, for each video frame, the computer device may also map the forward optical flow map between the video frame and the next frame back to the video frame to obtain an updated video frame, and smooth the depth map corresponding to the video frame based on the updated video frame to obtain a depth smoothing map.

[0052] Step 108 : performing background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame to obtain a background blur image corresponding to each video frame.

[0053] Among them, background blur is to make the depth of field shallower and focus on the subject.

[0054] Specifically, the computer device determines the degree of background blurring for each pixel in each video frame based on the depth information in the depth smoothing map of each video frame, and performs different degrees of blurring on each pixel in each video frame according to the degree of background blurring for each pixel in each video frame to obtain a background blur image corresponding to each video frame.

[0055] Step 110 : Obtain a target video based on the background blur images corresponding to each video frame.

[0056] The target video refers to the video obtained by blurring the background of the original video.

[0057] It can be understood that each video frame records its timing information in the original video, and the timing information is used to indicate the image of the corresponding video frame at a specific time point in the original video.

[0058] Specifically, the computer device arranges the background blur images corresponding to each video frame in the time sequence of the original video according to the timing information corresponding to each video frame to form a complete target video.

[0059] The above-mentioned background blurring method determines the depth map corresponding to each video frame in the original video; for each video frame, optical flow estimation is performed on the video frame and the adjacent video frames adjacent to the video frame to obtain an optical flow map reflecting the motion of each pixel in the video frame and the adjacent video frames, which can increase the stability of the background blurring of each video frame; the depth map corresponding to the video frame is smoothed according to the optical flow map to obtain a depth smoothing map, so that the blurring effect between each video frame is smoother; according to the depth information in the depth smoothing map of each video frame, the video frame is subjected to background blurring processing to obtain a background blurring image corresponding to each video frame; according to the background blurring image corresponding to each video frame, the target video is obtained. Since the depth map of each video frame has undergone optical flow and smoothing processing in the above process, the target video obtained after background blurring based on each depth map is more coherent and stable. That is, the background blurring of the present application can not only blur the background of the video, effectively solve the limitations of background blurring, but also improve the stability of the generated target video.

[0060] In some embodiments, step 102 specifically includes but is not limited to: performing depth estimation on each video frame in the original video to obtain an initial depth map corresponding to each video frame; reconstructing a three-dimensional scene model based on each initial depth map and preset image acquisition parameters; determining a plane area in each video frame; performing planar correction on the three-dimensional scene model based on the plane area in each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame; and determining a depth map corresponding to each video frame based on each corrected three-dimensional scene model.

[0061] Depth estimation is to estimate the depth of an object in a video frame, that is, to estimate the distance of each pixel in the video frame relative to the image acquisition device.

[0062] The image acquisition parameters are the internal and external parameters of the image acquisition device. Since the internal and external parameters in this application are only intermediate variables and do not have a decisive impact on the final result, any input can be set to a fixed focal length of the image acquisition device, and then the center coordinates of the imaging plane are set according to the resolution of the input video frame. The two parameters of focal length and center coordinates are used to form the internal and external parameter matrix of the image acquisition device, or the internal parameter matrix for three-dimensional reconstruction.

[0063] The plane area refers to a large area in a video frame, such as at least one of a desktop, a ground, or a wall in the video frame.

[0064] Specifically, first, the computer device performs depth estimation on each video frame in the original video to obtain an initial depth map corresponding to each video frame. Secondly, the computer device performs three-dimensional reconstruction based on each initial depth map and preset image acquisition parameters, that is, maps each initial depth map to a three-dimensional scene to obtain a reconstructed three-dimensional scene model. Then, the computer device performs plane area detection on each video frame to obtain the plane area in each video frame. In addition, the computer device also performs plane correction on the three-dimensional scene model based on the detected plane areas in each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame. Finally, the computer device determines the depth map corresponding to each video frame based on each corrected three-dimensional scene model. The present application performs depth estimation on each video frame, and further corrects the initial depth map based on the depth estimation, and performs background blur processing based on the depth map obtained after correction, which can improve the background blur effect.

[0065] In some embodiments, the Figure 2 The 3D reconstruction and depth correction module reconstructs the 3D scene for each video frame and corrects the 3D scene model to determine the depth map corresponding to each video frame according to each corrected 3D scene model.

[0066] In some embodiments, as Figure 2 As shown, the computer device can perform depth estimation on each continuous video frame or a single image through the depth estimation module of the present application. The depth estimation module includes a gating module and a depth estimation model.

[0067] In some embodiments, when a computer device performs depth estimation on each consecutive video frame, since the depth estimation module has a gating module that can perform loop processing, the depth estimation module can implicitly store information of previous video frames under the input of each consecutive video frame to improve the stability and consistency of each initial depth map output after depth estimation of each video frame.

[0068] It can be understood that the gating module is a module designed specifically for video input. The module implicitly stores some information of the previous video frame and provides guidance for the blurring result of the next frame. If the gating module is not turned off in the scenario of background blurring for a single image, continuous calls to this method may cause some blurring information of the previous image to remain on the next image. However, since the two images themselves may not be related, this will cause problems with the blurring result of the next image. Based on the above situation, this application considers that when performing depth estimation on a single image, it is necessary to turn off the gating module and directly perform depth estimation on the single image through the depth estimation model to obtain its corresponding initial depth map.

[0069] In some embodiments, the process of training the depth estimation module may include the following steps: first, determining the initial depth estimation model obtained by training in the first training stage using the three-dimensional sample map. Secondly, determining the intermediate depth estimation model to be trained, which is obtained by adding a gating module on the basis of the initial depth estimation model. Then, fixing the model parameters obtained in the first training stage in the intermediate depth estimation model, and using a time-series image sequence to perform iterative training on the intermediate depth estimation model in the second training stage to obtain a basic depth estimation model. Finally, using a learning rate less than a preset threshold, the basic depth estimation model obtained by training in the second training stage is fine-tuned in the third training stage to obtain the final depth estimation model. It can be understood that the depth estimation model is a neural network model for depth estimation obtained by training through deep learning technology.

[0070] Specifically, first, a neural network model is established and trained using a large number of color images or depth images corresponding to color images collected by a 3D camera until the neural network model converges, thereby obtaining an initial depth estimation model for the first training phase. It should be noted that in this phase of training, the images can be input into the neural network model in a random order for model training.

[0071] Secondly, a gating module is attached to the initial depth estimation model to obtain the intermediate depth estimation model to be trained.

[0072] Next, the model parameters of the intermediate depth estimation model obtained in the first training phase are fixed, and the intermediate depth estimation model is trained until convergence, thereby obtaining the basic depth estimation model of the second training phase. It should be noted that in this phase of training, each image can be continuously input into the intermediate depth estimation model for model training in a time sequence, that is, in the time order in which each image was acquired.

[0073] Finally, all model parameters of the basic depth estimation model are released, and the basic depth estimation model is fine-tuned using a learning rate less than a preset threshold. During this stage of training, the images can be alternately input into the basic depth estimation model in a random order of a batch and a continuous order of a batch to train the model, resulting in a trained depth estimation model. The trained depth estimation model can then be used to directly perform depth estimation on consecutive video frames or single images.

[0074] It should be noted that this application needs to take into account both video input and single image input scenarios. For video input, a continuous sequence of video frames is required to train the depth estimation model. However, if all training data is continuous, the depth estimation model is likely to become dependent on the data of the previous few frames, resulting in poor depth estimation when used for single image input. Therefore, training in a form that combines the two input methods can enhance the performance of the depth estimation model.

[0075] In some embodiments, the process of reconstructing a 3D scene model includes: if the video frame corresponding to the initial depth map is the first frame in the original video, directly reconstructing the video frame according to customized image acquisition parameters to obtain a 3D scene model. If the video frame corresponding to the initial depth map is not the first frame in the original video, merging the frame into the 3D scene using a multi-frame reconstruction method to obtain a 3D scene model for the corresponding frame.

[0076] It should be noted that, when the video frame corresponding to the initial depth map is the first frame in the original video, the matching cubes algorithm (Marching Cubes) can be used to reconstruct it in three dimensions. Specifically, the initial depth map can be first converted into point cloud data through the internal and external parameter matrix or the internal parameter matrix of the image acquisition device, and then the entire point cloud space is divided into several small cubes, and the point cloud isosurface is calculated in each cube through the form of the cube to generate small facets belonging to the cube, and finally all the small facets are combined to obtain the reconstructed three-dimensional scene model. When the video frame corresponding to the initial depth map is not the first frame in the original video, it can be converted into point cloud data using custom image acquisition parameters, and then it can be reconstructed in three dimensions using the matching cubes algorithm to obtain a new three-dimensional scene model.

[0077] In some embodiments, as Figure 2 As shown, the computer device can perform plane area detection on each video frame through the large plane detection module of the present application to obtain the plane area in each video frame. It can be understood that the large plane detection module detects the plane area through the plane detection model.

[0078] In some embodiments, the process of training a plane detection model may include establishing a to-be-trained model for plane detection and training the to-be-trained model using a large number of labeled color images with plane segmentation masks until convergence, thereby obtaining a trained plane detection model. Labeling an image refers to marking the locations of plane regions in an image using segmentation, i.e., marking which pixels in the image belong to a plane and which do not. It is understood that the plane detection model is a neural network model for plane detection obtained through deep learning techniques.

[0079] In some embodiments, the step of "performing planar correction on the three-dimensional scene model according to the planar areas in each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame" specifically includes but is not limited to: mapping the planar area in each video frame to three-dimensional space according to image acquisition parameters to obtain regional point cloud data corresponding to each video frame; in three-dimensional space, using the point cloud data of the three-dimensional scene model to perform point cloud registration on the regional point cloud data corresponding to each video frame to obtain registered point cloud data corresponding to each video frame; merging the registered point cloud data corresponding to each video frame into the point cloud data of the three-dimensional scene model to obtain fused point cloud data corresponding to each video frame; reconstructing the three-dimensional scene model according to the fused point cloud data corresponding to each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame.

[0080] Among them, point cloud registration refers to finding a suitable rotation angle and translation coefficient so that the regional point cloud data can overlap with the point cloud data of the three-dimensional scene to the greatest extent after rotation and translation.

[0081] The registered point cloud data refers to the point cloud data obtained by updating the original regional point cloud data corresponding to each video frame during the point cloud registration process.

[0082] Specifically, first, based on the pinhole camera model and on the premise that the depth map and image acquisition parameters corresponding to the video frame are known, the computer device maps each pixel point of the planar area in each video frame in the two-dimensional space to the corresponding point cloud data in the three-dimensional space, so as to obtain the regional point cloud data corresponding to each video frame. Secondly, the computer device discretizes the three-dimensional scene model and converts it into point cloud data. In the three-dimensional space, the computer device performs point cloud registration on the regional point cloud data corresponding to each video frame and the individual point cloud data in the corresponding three-dimensional scene model to obtain the registered point cloud data corresponding to each video frame. Then, the computer device merges the registered point cloud data corresponding to each video frame into the point cloud data of the corresponding three-dimensional scene to obtain the fused point cloud data corresponding to each video frame. Finally, the computer device reconstructs the three-dimensional scene model based on the fused point cloud data corresponding to each video frame. The reconstructed three-dimensional scene model is the corrected three-dimensional scene model corresponding to each video frame. It should be noted that since the registered point cloud data is converted from a planar area, it is usually flatter than the point cloud data of the three-dimensional scene. Therefore, the planar correction of the three-dimensional scene model can be achieved through the registered point cloud data. The background of each video frame is blurred based on the depth map obtained based on the corrected three-dimensional scene model, which can further improve the background blur effect.

[0083] In some embodiments, the step of "determining the depth map corresponding to each video frame based on each corrected three-dimensional scene model" specifically includes but is not limited to: mapping each corrected three-dimensional scene model to a two-dimensional space according to image acquisition parameters to obtain a depth map corresponding to each video frame.

[0084] Specifically, the computer device reverse maps each corrected three-dimensional scene model back to the two-dimensional space based on the same image acquisition parameters used to reconstruct the three-dimensional scene, and obtains a depth map corresponding to each video frame, which is further corrected on the basis of the initial depth map. The background of each video frame is blurred by the corrected depth map, which can make the background blur effect more natural.

[0085] In some embodiments, step 106 specifically includes but is not limited to: determining the corresponding pixel positions of each pixel of the video frame in the adjacent video frames according to the optical flow map; mapping each pixel of the video frame to the adjacent video frames according to the corresponding pixel positions of each pixel of the video frame in the adjacent video frames to obtain an optical flow map; and performing weighted smoothing on the depth map according to the optical flow map to obtain a depth smoothing map.

[0086] Specifically, for each video frame, the computer device obtains the movement of all pixels in the video frame to the pixels in the adjacent video frame based on the optical flow map, and further determines the corresponding pixel positions of each pixel of the video frame in the adjacent video frame. Then, the computer moves each pixel of the video frame to the corresponding pixel position in the adjacent video frame based on the corresponding pixel position of each pixel of the video frame in the adjacent video frame. The above pixel movement process is the mapping process of each pixel of the video frame, and an optical flow map is obtained after the mapping is completed. Finally, the computer device matches each pixel in the optical flow map with the depth map corresponding to the video frame to achieve frame-to-frame smoothing operations on the multi-frame depth map, and obtains a smoothed depth smoothing map, so that the target video obtained by blurring the background of each video frame based on the depth smoothing map is smoother and more natural.

[0087] In some embodiments, the Figure 2 The optical flow smoothing module shown performs optical flow estimation and weighted smoothing on each video frame to obtain a smoothed depth smoothing map.

[0088] In some embodiments, the depth information in the depth smoothing map includes the pixel depth value of each pixel in the depth smoothing map. Step 108 specifically includes but is not limited to: for each depth smoothing map, determining the blur weight factor corresponding to each pixel in the depth smoothing map according to the difference between the pixel depth value of each pixel in the depth smoothing map and the average depth value; for each video frame, performing background blur processing on the video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame to obtain a background blur image corresponding to the video frame.

[0089] The average depth value is the average value of the pixel depth values ​​of each pixel in the specified focus area of ​​the depth smoothing map, and the blur weight factor is used to indicate the blur degree of each pixel.

[0090] The designated focus area refers to a region in a video frame that the user specifies in advance as needing to be focused on, and this region is the designated focus area.

[0091] Specifically, for each depth smoothing map, the computer device calculates the difference between the pixel depth value of each pixel in the depth smoothing map and the average depth value, that is, the difference between the pixel depth value of each pixel and the average depth value, and then normalizes it to the range [0, 1] to obtain the blur weight factor corresponding to each pixel for determining the degree of blur. For each video frame, the computer device performs different degrees of background blur processing on each pixel in the video frame based on the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame, and obtains the background blur image corresponding to the video frame.

[0092] In some embodiments, for each video frame, the difference A between the pixel depth value of each pixel in the depth smoothing map and the average depth value can be calculated using formula (1):

[0093]

[0094] in, This is the depth smoothing map corresponding to the t-th frame in the original video, specifically referring to each pixel in the depth smoothing map. d refers to the average of all depth values ​​in the specified focus area in the t-th frame, and abs refers to the absolute value operation.

[0095] It should be noted that the value obtained after normalizing the result calculated by formula (1) is the blur weight factor used to judge the degree of blur, which can be represented by a blur weight factor graph. Through the blur factor graph, each pixel in the video frame can be subjected to different degrees of background blur processing to obtain a background blur image corresponding to the video frame.

[0096] In some embodiments, the Figure 2 The adaptive blur module performs background blur on each depth smoothing image to obtain multiple background blur images.

[0097] In some embodiments, the step of "for each video frame, performing background blur processing on the video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame to obtain a background blur image corresponding to the video frame" specifically includes but is not limited to: for each video frame, determining the blur kernel size corresponding to each pixel in the video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of each video frame; performing different degrees of blur processing on the video frame according to the blur kernel size corresponding to each pixel in the video frame to obtain a background blur image corresponding to the video frame.

[0098] A blur kernel is a type of convolution kernel. It's essentially a matrix. Convolving a sharp image with the blur kernel causes the image to become blurred, hence the name blur kernel. As you can see, different blur kernel sizes result in varying degrees of blurring.

[0099] Specifically, for each video frame, the computer device determines the blur kernel size corresponding to each pixel in the video frame based on the blur weight factor corresponding to each pixel in the depth smoothing map of each video frame. Then, the computer device sets different blur kernel sizes for each pixel in the video frame or image regions of different depths based on the blur kernel size corresponding to each pixel in the video frame, thereby blurring the video frame to varying degrees, resulting in a natural-looking background blur image with diverse depths of field corresponding to the video frame. The aforementioned image region generally refers to a certain area in the video frame.

[0100] In some embodiments, since the depth value of each pixel in a video frame may not be the same, the blur weight factor calculated from the depth value may not be the same for each pixel. Therefore, a different blur kernel size can be set for each pixel. In practical applications, to further improve processing efficiency, it is possible to consider merging pixels with similar blur kernel sizes in a video frame into a single region, thereby implementing regional segmentation. Different blur kernel sizes can be set for each region, allowing for different background blur processing based on the region.

[0101] In some embodiments, as Figure 3 As shown, the background blurring method of the present application specifically includes but is not limited to the following steps:

[0102] Step 302 : perform depth estimation on each video frame in the original video to obtain an initial depth map corresponding to each video frame.

[0103] Step 304: reconstruct a three-dimensional scene model according to the initial depth maps and preset image acquisition parameters.

[0104] Step 306: Determine the planar area in each video frame.

[0105] Step 308 : Map the planar area in each video frame to a three-dimensional space according to the image acquisition parameters to obtain regional point cloud data corresponding to each video frame.

[0106] Step 310 : In three-dimensional space, point cloud data of the three-dimensional scene model are used to perform point cloud registration on the regional point cloud data corresponding to each video frame to obtain registered point cloud data corresponding to each video frame.

[0107] In step 312 , the registered point cloud data corresponding to each video frame is merged into the point cloud data of the three-dimensional scene model to obtain fused point cloud data corresponding to each video frame.

[0108] Step 314 : reconstructing a three-dimensional scene model based on the fused point cloud data corresponding to each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame.

[0109] In step 316 , each corrected three-dimensional scene model is mapped to a two-dimensional space according to the image acquisition parameters to obtain a depth map corresponding to each video frame.

[0110] Step 318 : For each video frame, perform optical flow estimation on the video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames.

[0111] Step 320 : Determine the corresponding pixel positions of each pixel of the video frame in adjacent video frames according to the optical flow map.

[0112] In step 322 , each pixel of the video frame is mapped to an adjacent video frame according to the corresponding pixel position of each pixel of the video frame in the adjacent video frame to obtain an optical flow map.

[0113] Step 324 : Perform weighted smoothing on the depth map according to the optical flow map to obtain a depth smoothing map.

[0114] Step 326 : For each depth smoothing map, determine a blurring weight factor corresponding to each pixel in the depth smoothing map according to the difference between the pixel depth value of each pixel in the depth smoothing map and the average depth value.

[0115] Step 328 : For each video frame, determine the blur kernel size corresponding to each pixel in the video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of each video frame.

[0116] In step 330 , blurring is performed on the video frame to different degrees using the blur kernel size corresponding to each pixel in the video frame to obtain a background blurred image corresponding to the video frame.

[0117] Step 332: Obtain a target video based on the background blur images corresponding to each video frame.

[0118] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0119] Based on the same inventive concept, the present application also provides a background blur device for implementing the aforementioned background blur method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more background blur device embodiments provided below can be found in the aforementioned limitations of the background blur method and will not be further elaborated here.

[0120] In one embodiment, Figure 4 As shown, a background blurring device is provided, comprising: an image determination unit 402, an optical flow estimation unit 404, an image smoothing unit 406, a background blurring unit 408, and a video determination unit 410, wherein:

[0121] An image determination unit 402 is configured to determine a depth map corresponding to each video frame in the original video;

[0122] an optical flow estimation unit 404 configured to perform optical flow estimation on each video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames; wherein the adjacent video frames are adjacent to the video frames, and the optical flow map reflects the motion of pixels in the video frame and the adjacent video frames;

[0123] An image smoothing unit 406 is configured to smooth the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map;

[0124] A background blur unit 408 is configured to perform background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame to obtain a background blur image corresponding to each video frame;

[0125] The video determination unit 410 is configured to obtain a target video according to the background blur images corresponding to the respective video frames.

[0126] The above-mentioned background blur device determines the depth map corresponding to each video frame in the original video; for each video frame, optical flow estimation is performed on the video frame and the adjacent video frames adjacent to the video frame to obtain an optical flow map reflecting the movement of each pixel in the video frame and the adjacent video frames, which can increase the stability of the background blur of each video frame; the depth map corresponding to the video frame is smoothed according to the optical flow map to obtain a depth smoothing map, so that the blur effect between each video frame is smoother; based on the depth information in the depth smoothing map of each video frame, the video frame is subjected to background blur processing to obtain a background blur image corresponding to each video frame; based on the background blur image corresponding to each video frame, a target video is obtained. Since the depth map of each video frame has undergone optical flow and smoothing processing in the above process, the target video obtained after background blur based on each depth map is more coherent and stable. That is, the background blur of the present application can not only blur the background of the video, but also improve the stability of the generated target video.

[0127] In some embodiments, the image determination unit 402 is further configured to perform depth estimation on each video frame in the original video to obtain an initial depth map corresponding to each video frame; reconstruct a 3D scene model based on each initial depth map and preset image acquisition parameters; determine a planar area in each video frame; and perform planar correction on the 3D scene model based on the planar area in each video frame to obtain a corrected 3D scene model corresponding to each video frame.

[0128] Determine the depth map corresponding to each video frame based on the corrected three-dimensional scene models.

[0129] In some embodiments, the image determination unit 402 is further used to map the planar area in each video frame to three-dimensional space based on image acquisition parameters to obtain regional point cloud data corresponding to each video frame; in three-dimensional space, use the point cloud data of the three-dimensional scene model to perform point cloud registration on the regional point cloud data corresponding to each video frame to obtain registered point cloud data corresponding to each video frame; merge the registered point cloud data corresponding to each video frame into the point cloud data of the three-dimensional scene model to obtain fused point cloud data corresponding to each video frame; reconstruct the three-dimensional scene model based on the fused point cloud data corresponding to each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame.

[0130] In some embodiments, the image determination unit 402 is further configured to map each corrected three-dimensional scene model to a two-dimensional space according to the image acquisition parameters to obtain a depth map corresponding to each video frame.

[0131] In some embodiments, the image smoothing unit 406 is further used to determine the corresponding pixel positions of each pixel of the video frame in the adjacent video frames based on the optical flow map; map each pixel of the video frame to the adjacent video frames based on the corresponding pixel positions of each pixel of the video frame in the adjacent video frames to obtain an optical flow mapping map; and perform weighted smoothing on the depth map based on the optical flow mapping map to obtain a depth smoothing map.

[0132] In some embodiments, the depth information in the depth smoothing map includes the pixel depth value of each pixel in the depth smoothing map, and the background blur unit 408 is further used to determine, for each depth smoothing map, a blur weight factor corresponding to each pixel in the depth smoothing map according to the difference between the pixel depth value of each pixel in the depth smoothing map and the average depth value; wherein the average depth value is the average value of the pixel depth values ​​of each pixel in the specified focus area of ​​the depth smoothing map, and the blur weight factor is used to indicate the degree of blur of each pixel; for each video frame, the background blur processing is performed on the video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame to obtain a background blur image corresponding to the video frame.

[0133] In some embodiments, the background blur unit 408 is further configured to determine, for each video frame, a blur kernel size corresponding to each pixel in the video frame according to a blur weight factor corresponding to each pixel in the depth smoothing map of each video frame;

[0134] The video frames are blurred to different degrees according to the blur kernel size corresponding to each pixel in the video frame to obtain the background blurred image corresponding to the video frame.

[0135] Each module in the background blurring device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0136] In some embodiments, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to background blur. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a background blur method is implemented.

[0137] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a background blurring method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse. Those skilled in the art will understand that Figure 5 and Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0139] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0140] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A background blurring method, characterized in that: include: Determine the depth map corresponding to each video frame in the original video; For each video frame, performing optical flow estimation on the video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames; wherein the adjacent video frames are adjacent to the video frame, and the optical flow map reflects the motion of pixels in the video frame and the adjacent video frames; Smoothing the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map; Performing background blur processing on the video frames according to the depth information in the depth smoothing map of each video frame to obtain background blur images corresponding to each video frame; A target video is obtained according to the background blurred images corresponding to the respective video frames.

2. The method according to claim 1, characterized in that Determining the depth map corresponding to each video frame in the original video includes: Performing depth estimation on each video frame in the original video to obtain an initial depth map corresponding to each video frame; Reconstructing a three-dimensional scene model according to each of the initial depth maps and preset image acquisition parameters; Determining a plane area in each of the video frames; Performing plane correction on the three-dimensional scene model according to the plane regions in each video frame to obtain a corrected three-dimensional scene model corresponding to each video frame; Determine a depth map corresponding to each of the video frames based on the corrected three-dimensional scene models.

3. The method according to claim 2, characterized in that The performing plane correction on the three-dimensional scene model according to the plane regions in each video frame to obtain the corrected three-dimensional scene model corresponding to each video frame includes: Mapping the planar area in each of the video frames to a three-dimensional space according to the image acquisition parameters to obtain regional point cloud data corresponding to each of the video frames; In three-dimensional space, using the point cloud data of the three-dimensional scene model, perform point cloud registration on the regional point cloud data corresponding to each video frame to obtain registered point cloud data corresponding to each video frame; Merging the registered point cloud data corresponding to each of the video frames into the point cloud data of the three-dimensional scene model to obtain fused point cloud data corresponding to each of the video frames; A three-dimensional scene model is reconstructed based on the fused point cloud data corresponding to each of the video frames to obtain a corrected three-dimensional scene model corresponding to each of the video frames.

4. The method according to claim 2, characterized in that Determining the depth map corresponding to each of the video frames according to each of the corrected three-dimensional scene models includes: The corrected three-dimensional scene models are mapped to a two-dimensional space according to the image acquisition parameters to obtain a depth map corresponding to each of the video frames.

5. The method according to claim 1, characterized in that The step of smoothing the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map includes: Determining, according to the optical flow map, corresponding pixel positions of each pixel of the video frame in the adjacent video frames; Mapping each pixel of the video frame to the adjacent video frames according to the corresponding pixel positions of each pixel of the video frame in the adjacent video frames to obtain an optical flow map; The depth map is weighted and smoothed according to the optical flow map to obtain a depth smoothing map.

6. The method according to any one of claims 1 to 5, characterized in that The depth information in the depth smoothing map includes a pixel depth value of each pixel in the depth smoothing map; The performing background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame to obtain a background blur image corresponding to each video frame, including: For each of the depth smoothing maps, determining a blur weight factor corresponding to each pixel in the depth smoothing map based on a difference between a pixel depth value of each pixel in the depth smoothing map and an average depth value; wherein the average depth value is an average of the pixel depth values ​​of each pixel in a specified focus area of ​​the depth smoothing map, and the blur weight factor is used to indicate a degree of blurring of each pixel; For each video frame, background blur processing is performed on the video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame to obtain a background blur image corresponding to the video frame.

7. The method according to claim 6, characterized in that The method of performing background blur processing on each video frame according to the blur weight factor corresponding to each pixel in the depth smoothing map of the video frame to obtain a background blur image corresponding to the video frame includes: For each video frame, determining a blur kernel size corresponding to each pixel in the video frame according to a blur weight factor corresponding to each pixel in the depth smoothing map of each video frame; The video frames are blurred to different degrees according to the blur kernel size corresponding to each pixel in the video frame to obtain a background blurred image corresponding to the video frame.

8. A background blur device, characterized in that: include: An image determination unit, configured to determine a depth map corresponding to each video frame in the original video; an optical flow estimation unit, configured to perform optical flow estimation on each video frame and adjacent video frames to obtain an optical flow map between the video frame and the adjacent video frames; wherein the adjacent video frames are adjacent to the video frame, and the optical flow map reflects the motion of pixels in the video frame and the adjacent video frames; an image smoothing unit, configured to smooth the depth map corresponding to the video frame according to the optical flow map to obtain a depth smoothing map; A background blur unit is configured to perform background blur processing on the video frame according to the depth information in the depth smoothing map of each video frame, so as to obtain a background blur image corresponding to each video frame; The video determination unit is used to obtain a target video according to the background blur images corresponding to the video frames.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images