A method, apparatus and device for image background blurring

Abstract

The application discloses an image background blurring method, device and equipment, comprising the following steps: after constructing an initial depth image generation model and an initial blurring image generation model, a sample image set received is used for model training and total loss function value determination, the initial depth image generation model and the initial blurring image generation model are optimized by analyzing the total loss function value, and a target depth image generation model and a target blurring image generation model are output when the total loss function value converges; when a target image is received, a corresponding target sparse depth map is constructed by using a visual mileage calculation method, a corresponding target depth map is constructed according to the target image and the target sparse depth map by using the target depth image generation model, and a corresponding target background blurring image is constructed by using the target blurring image generation model based on the target depth map. In the whole image background blurring processing process, the image quality of the generated background blurring image is better through multi-model fusion.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image background blurring method, apparatus, and device. Background Technology

[0002] In daily life, for devices such as embedded small mobile devices, smartphones, and tablets, the front-facing monocular camera has difficulty acquiring a variety of image perception information, which makes it impossible for these devices to achieve effective background blurring. They usually need to rely on image post-processing technology to achieve more effective background blurring.

[0003] Currently, 3D scenes are becoming increasingly popular, and background blurring technology is gradually being extended to 3D scenes. Many methods based on convolutional neural networks can produce natural and realistic background blurring effects in multi-object scenarios. However, the expansion of most methods in scene applications mainly depends on the number of object categories that the segmentation network can identify. Their diversity of application scenarios is greatly limited, making it difficult to adapt to the complexity of 3D scenes. The image quality of the generated background blurring images still needs to be optimized. Summary of the Invention

[0004] This invention provides an image background blurring method, apparatus, and device, which solves the technical problem that the image quality of the generated background blurring image is not high when existing background blurring techniques are used to blur the background of an image.

[0005] The first aspect of this invention provides an image background blurring method, comprising:

[0006] When a set of sample images is received, an initial depth image generation model and an initial bokeh image generation model are constructed.

[0007] The initial depth image generation model and the initial blurred image generation model are trained using the sample image set, and the total loss function value is determined.

[0008] The initial depth image generation model and the initial bokeh image generation model are optimized based on the total loss function value until the total loss function value converges, and the target depth image generation model and the target bokeh image generation model are output.

[0009] When a target image is received, a corresponding sparse depth map of the target is constructed using visual odometry.

[0010] Based on the target image and the target sparse depth map, a corresponding target depth map is constructed using the target depth image generation model.

[0011] Based on the target depth map, a corresponding target background blur map is constructed using the target blur image generation model.

[0012] Optionally, the step of constructing an initial depth image generation model and an initial bokeh image generation model when a sample image set is received includes:

[0013] When a set of sample images is received, an initial depth image generation model is constructed using an initial sparse autoencoder, an initial depth estimator, and an initial skip connection module.

[0014] An initial blurred image generation model is constructed using an initial local guidance module, an initial generator encoder, an initial normalization module, an initial upsampling module, an initial discriminator, an initial constraint encoder, an initial decoder, and an initial confidence fusion head.

[0015] Optionally, the sample image set includes a left frame sample image, a right frame sample image, an original sample image, and a dense depth sample image; the step of training the initial depth image generation model and the initial blurred image generation model using the sample image set respectively, and determining the total loss function value, includes:

[0016] Based on the left frame sample image and the right frame sample image, the corresponding left sparse depth map and right sparse depth map are constructed using the visual odometry method.

[0017] The left frame sample image, the right frame sample image, the left sparse depth image, and the right sparse depth image are input into the initial depth image generation model, and the corresponding left depth image is output. The corresponding depth loss function value is calculated according to the preset depth loss function model.

[0018] The original sample image, the left depth image, and the dense depth sample image are input into the initial blurred image generation model to construct a background blur prediction image, and the corresponding blur loss function value is calculated according to the preset blur loss function model.

[0019] The total loss function value is determined by summing the depth loss function value and the blurring loss function value.

[0020] Optionally, the step of inputting the left frame sample image, the right frame sample image, the left sparse depth map, and the right sparse depth map into the initial depth image generation model, outputting the corresponding left depth map, and calculating the corresponding depth loss function value according to a preset depth loss function model includes:

[0021] The left sparse depth map and the right sparse depth map are respectively input into the corresponding initial sparse autoencoder, and the left dense depth map and the right dense depth map are output.

[0022] The left frame sample image, the left sparse depth image, and the left dense depth image are sent to the initial depth estimator to generate a left reverse depth image and a right reverse depth image;

[0023] The initial skip connection module extracts features from the left dense depth map and the right dense depth map respectively to generate the corresponding left residual dense depth map and right residual dense depth map.

[0024] The left reverse depth map is added to the left residual dense depth map, and the right reverse depth map is added to the right residual dense depth map, respectively, to output the left disparity map and the right disparity map;

[0025] The left frame sample image and the left disparity image are respectively weighted according to the preset depth image weights, and the left depth image is output.

[0026] Using the left sparse depth map, the right sparse depth map, the left dense depth map, the right dense depth map, the left disparity map, and the right disparity map, the corresponding depth loss function values ​​are calculated according to a preset depth loss function model.

[0027] Optionally, the step of inputting the original sample image, the left depth image, and the dense depth sample image into the initial blurred image generation model to construct a background blurred prediction image, and calculating the corresponding blurred loss function value according to a preset blurred loss function model, includes:

[0028] The initial local guidance module extracts features from the original sample image to generate a local guidance image, and the first sum of the original sample image and the local guidance image is input into the initial generator encoder for downsampling and outputting an image feature map.

[0029] Using the initial constraint encoder, the second summation result of the left depth map and the local guide map is subjected to convolution extraction, average pooling mapping and downsampling to construct a depth feature map;

[0030] The image feature map and the depth feature map are input into the initial normalization module for feature fusion, and a first fused image is output.

[0031] The first fused map is upsampled by the initial upsampling module to generate an upsampled map, and the depth feature map is upsampled by the decoder to construct a local depth map and a local confidence map;

[0032] The upsampled image and the local depth image are input into the initial normalization module, and the fused depth image and the fused confidence image are output.

[0033] Using the initial confidence fusion head, the fused depth map and the local depth map are subjected to image weighting operations according to the confidence image weights corresponding to the fused confidence map and the local confidence map, and a background blur prediction map is constructed.

[0034] The initial discriminator identifies the true probability of the output image by distinguishing the dense depth sample map and the fused depth map, and uses the true probability of the image to calculate the corresponding blurring loss function value according to the preset blurring loss function model.

[0035] Optionally, the step of optimizing the initial depth image generation model and the initial blurred image generation model based on the total loss function value until the total loss function value converges, and then outputting the target depth image generation model and the target blurred image generation model, includes:

[0036] Determine whether the total loss function value has converged;

[0037] If not, the model parameters of the initial depth image generation model and the initial bokeh image generation model, as well as the depth image weights, are updated through backpropagation, and the process jumps to the step of training the initial depth image generation model and the initial bokeh image generation model using the sample images respectively to determine the total loss function value.

[0038] If so, the initial depth image generation model is used as the target depth image generation model, and the initial bokeh image generation model is used as the target bokeh image generation model.

[0039] Optionally, the target depth image generation model includes a target sparse autoencoder, a target depth estimator, and a target skip connection module; the step of constructing a corresponding target depth map based on the target image and the target sparse depth map using the target depth image generation model includes:

[0040] The target sparse depth map is input into the target sparse autoencoder to generate the target dense depth map;

[0041] The target depth estimator is used to estimate the depth of the target image, the target sparse depth, and the target dense depth map to construct a target inverse depth map.

[0042] The target jump connection module extracts features from the target dense depth map and outputs the target residual dense depth map.

[0043] The target inverse depth map and the target residual dense depth map are feature-added to output a target disparity map;

[0044] The target depth map is generated by performing element-wise image weighting operations based on the target image and the target disparity map corresponding to the target depth image weights.

[0045] Optionally, the step of constructing a corresponding target background blur map based on the target depth map using the target blur image generation model includes:

[0046] The target depth map is input into the target generator encoder in the target blurred image generation model for downsampling, and the target image feature map is output.

[0047] The target image feature map is upsampled by the target upsampling module in the target blurring image generation model to construct the corresponding target background blurring image.

[0048] A second aspect of the present invention provides an image background blurring device, comprising:

[0049] The model building module is used to build an initial depth image generation model and an initial bokeh image generation model when a sample image set is received.

[0050] The total loss function value determination module is used to train the initial depth image generation model and the initial blurred image generation model using the sample image set, respectively, and determine the total loss function value.

[0051] The target model output module is used to optimize the initial depth image generation model and the initial bokeh image generation model based on the total loss function value until the total loss function value converges, and output the target depth image generation model and the target bokeh image generation model.

[0052] The target sparse depth map construction module is used to construct the corresponding target sparse depth map using visual odometry when a target image is received.

[0053] The target depth image construction module is used to construct a corresponding target depth map based on the target image and the target sparse depth map through the target depth image generation model;

[0054] The target background blurring image construction module is used to construct a corresponding target background blurring image based on the target depth map and the target blurring image generation model.

[0055] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the image background blurring method as described in any of the first aspects of the present invention.

[0056] As can be seen from the above technical solutions, the present invention has the following advantages:

[0057] This invention, after constructing an initial depth image generation model and an initial blurred image generation model, trains the models using a received sample image set and determines the total loss function value. The initial depth image generation model and the initial blurred image generation model are then optimized by analyzing the total loss function value. When the total loss function value converges, a target depth image generation model and a target blurred image generation model are output. When a target image is received, a corresponding target sparse depth map is constructed using visual odometry. The target depth image generation model then constructs a corresponding target depth map based on the target image and the target sparse depth map. Based on the target depth map, the target blurred image generation model constructs a corresponding target background blurred image. Throughout the entire image background blurring process, multi-model fusion is used to improve the image quality of the generated background blurred image. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the steps of an image background blurring method provided in Embodiment 1 of the present invention;

[0060] Figure 2 This is a flowchart of the steps of an image background blurring method provided in Embodiment 2 of the present invention;

[0061] Figure 3 This is a schematic diagram of the working framework of the depth image generation model provided in Embodiment 2 of the present invention;

[0062] Figure 4 This is a schematic diagram of the working framework of the blurred image generation model provided in Embodiment 2 of the present invention;

[0063] Figure 5 This is a schematic diagram of the working framework of the constraint network of the blurred image generation model provided in Embodiment 2 of the present invention;

[0064] Figure 6 This is a structural block diagram of an image background blurring processing device provided in Embodiment 3. Detailed Implementation

[0065] This invention provides an image background blurring method, apparatus, and device to solve the technical problem that the image quality of the generated background blurring image is not high when existing background blurring techniques are used to blur the background of an image.

[0066] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of an image background blurring method provided in Embodiment 1 of the present invention.

[0068] This invention provides an image background blurring method, comprising:

[0069] Step 101: When the sample image set is received, construct the initial depth image generation model and the initial bokeh image generation model.

[0070] A sample image set refers to a collection of images used for model training.

[0071] The initial depth image generation model refers to the initial model with initial model parameters set to generate a depth image.

[0072] The initial blurred image generation model refers to the initial model set with initial model parameters, which is used to generate a background blurred image.

[0073] In this embodiment of the invention, when a sample image set is received from any demand platform that supports the application of the image background blurring method, an initial depth image generation model and an initial blurring image generation model are constructed according to preset initial model parameters.

[0074] It is understandable that model parameters include the model's weights and biases.

[0075] Step 102: Train the initial depth image generation model and the initial bokeh image generation model using the sample image set, and determine the total loss function value.

[0076] The total loss function value refers to the simulation calculation result of the total loss function model during the training process of the initial depth image generation model and the initial bokeh image generation model.

[0077] The total loss function model is as follows:

[0078]

[0079] in, This is the total loss function value. For the depth loss function value, This represents the value of the virtualization loss function.

[0080] The depth loss function value refers to the simulated calculation result of the depth loss function model during the training process of the initial depth image generation model. The blurring loss function value refers to the simulated calculation result of the blurring loss function model during the training process of the initial blurring image generation model.

[0081] In this embodiment of the invention, the sample image set is input into the initial depth image generation model and the initial bokeh image generation model for training. Based on all the output results of the two models, simulation calculation is performed according to the preset total loss function model to determine the total loss function value.

[0082] Step 103: Optimize the initial depth image generation model and the initial bokeh image generation model based on the total loss function value until the total loss function value converges, and output the target depth image generation model and the target bokeh image generation model.

[0083] A target depth image generation model refers to a target model that is trained and iteratively optimized from an initial depth image generation model to achieve a predetermined goal.

[0084] A target bokeh image generation model refers to a target model that is trained and iteratively optimized from an initial bokeh image generation model to achieve a predetermined target.

[0085] In this embodiment of the invention, based on the feature error of the model in the image processing process reflected by the total loss function value, and with the convergence of the total loss function value as the goal, the parameters involved in the initial depth image generation model and the initial bokeh image generation model are updated and optimized, and the target depth image generation model and the target bokeh image generation model are output.

[0086] Step 104: When the target image is received, the corresponding sparse depth map of the target is constructed using the visual odometry method.

[0087] The target image refers to the RGB image of the target scene acquired using a monocular vision system. It can be understood that an RGB image refers to a three-channel color image.

[0088] The target sparse depth map refers to the sparse depth map corresponding to the target image. The sparse depth map is used to represent the depth map containing key pixel information.

[0089] Visual odometry (VO) can estimate sparse depth information of three-dimensional structures from images.

[0090] In this embodiment of the invention, when a target image is received from any demand platform that supports the application of image background blurring methods, feature points are extracted from the target image using visual odometry to construct a corresponding target sparse depth map.

[0091] Step 105: Based on the target image and the target sparse depth map, construct the corresponding target depth map using the target depth image generation model.

[0092] A target depth image refers to a depth map of the target scene corresponding to a target image. The depth map is used to represent the depth distribution of various objects in the current scene.

[0093] In this embodiment of the invention, after the target sparse depth map is determined, the target image and the target sparse depth map are input together into the target depth image generation model. After image processing by the target depth image generation model, the corresponding target depth map is generated.

[0094] Step 106: Based on the target depth map, construct the corresponding target background blur map using the target blur image generation model.

[0095] The target background blur map refers to the background blur map after the background part of the target image has been blurred. The background blur map is used to represent the depth of field of each part of the current scene.

[0096] In this embodiment of the invention, the target depth image is fed to the target bokeh image generation model, and after the background part is bokeh processed, the corresponding target background bokeh image is output.

[0097] In this embodiment of the invention, after constructing an initial depth image generation model and an initial blurred image generation model, the received sample image set is used for model training to determine the total loss function value. The initial depth image generation model and the initial blurred image generation model are optimized by analyzing the total loss function value. When the total loss function value converges, a target depth image generation model and a target blurred image generation model are output. When a target image is received, a corresponding target sparse depth map is constructed using visual odometry. The target depth image generation model constructs a corresponding target depth map based on the target image and the target sparse depth map. Based on the target depth map, the target blurred image generation model constructs a corresponding target background blurred image. Throughout the entire image background blurring process, multi-model fusion improves the real-time performance and effectiveness of depth prediction, resulting in higher quality generated background blurred images.

[0098] Please see Figure 2 , Figure 2 This is a flowchart of the steps of an image background blurring method provided in Embodiment 2 of the present invention.

[0099] This invention provides an image background blurring method, comprising:

[0100] Step 201: When the sample image set is received, construct the initial depth image generation model and the initial bokeh image generation model.

[0101] Optionally, step 201 includes the following sub-steps:

[0102] When a set of sample images is received, an initial depth image generation model is constructed using an initial sparse autoencoder, an initial depth estimator, and an initial skip connection module.

[0103] An initial blurred image generation model is constructed using an initial local guidance module, an initial generator encoder, an initial normalization module, an initial upsampling module, an initial discriminator, an initial constraint encoder, an initial decoder, and an initial confidence fusion head.

[0104] The sparse autoencoder consists of five sparse invariant convolutional layers. A channel attention module is added before the sparse convolutional layers. After the channel attention module, the first part consists of four sparse convolutional layers, each with sixteen filters. The kernel size of the first three convolutional layers decreases layer by layer, and remains unchanged until the third layer. The input resolution of each layer remains the same. The second part consists of sparse convolutional pixel-wise filters.

[0105] The depth estimator PyD-Net works by downsampling features from the input image in a pyramid-like manner. As the number of layers increases, more features are extracted, effectively capturing global information and inferring the depth of the corresponding layer at that resolution. When the lowest resolution level is reached, i.e., the highest layer of the pyramid, upsampling is performed layer by layer, and the upsampled features are concatenated with the features extracted from the input image until the highest resolution is reached.

[0106] The jump connection module includes a three-layer ResBlock.

[0107] The local guidance module uses U-Net as the feature extractor. The normalization module, W-AdaIN, includes a weighted adaptive instance normalization layer.

[0108] The generator encoder and the upsampling module form a symmetrical encoder-decoder structure. The generator encoder includes convolutional layers, average pooling layers, and convolutional downsampling layers, while the upsampling module includes convolutional upsampling layers corresponding to the convolutional downsampling layers of the generator encoder.

[0109] The discriminator includes convolutional layers, normalization layers, non-linear activation layers, dropout layers, and fully connected layers.

[0110] Constraint encoders and decoders, see [link / reference] Figure 5 Based on the ResNet-34 architecture, a symmetrical encoder-decoder structure is formed. The constraint encoder includes a 7x7 convolutional layer, an average pooling layer, and a convolutional downsampling layer. The decoder includes a convolutional upsampling layer corresponding to the convolutional downsampling layer of the constraint encoder. H / 2xW / 2, H / 4xW / 4, H / 8xW / 8, H / 16xW / 16, and H / 32xW / 32 represent the image resolution, where H and W are the height and width of the input image, respectively.

[0111] The confidence fusion head includes convolutional layers, normalization layers, nonlinear activation layers, and transposed convolutional layers.

[0112] In this embodiment of the invention, upon receiving a sample image set from any demand-side platform that supports the application of image background blurring methods, the parameters of the sparse autoencoder, depth estimator, skip connection module, local guidance module, generator encoder, normalization module, upsampling module, discriminator, constraint encoder, decoder, and confidence fusion head are set according to preset initial model parameters. See also... Figure 4 As shown, the initial sparse autoencoder, initial depth estimator and initial skip connection module are used to construct the initial depth image generation model. The initial local guidance module, initial generator encoder, initial normalization module, initial upsampling module and initial discriminator are used to form RDF-GAN (Generative Adversarial Network). The initial constraint encoder and initial decoder are used to form the constraint network. The initial blurred image generation model is constructed based on RDF-GAN, constraint network and initial confidence fusion head.

[0113] Optionally, the sample image set includes a left frame sample image, a right frame sample image, an original sample image, and a dense depth sample image.

[0114] The left and right frame sample images serve as a calibration stereo pair. These are RGB images of the training scene acquired using a binocular vision system. The corresponding original sample image is an RGB image of the same training scene acquired using a monocular vision system. The corresponding dense depth sample image serves as the Ground Truth image. It can be understood that in deep learning, the Ground Truth serves as a reference standard for assessing the classification accuracy of images in supervised training.

[0115] Step 202: Based on the left frame sample image and the right frame sample image, construct the corresponding left sparse depth map and right sparse depth map using the visual odometry method.

[0116] The left sparse depth map and the right sparse depth map refer to the sparse depth maps corresponding to the left frame sample map and the right frame sample map, respectively.

[0117] In this embodiment of the invention, after receiving the left frame sample image and the right frame sample image, the visual odometry method is used to extract the sparse depth information of the feature points of the left frame sample image and the right frame sample image respectively, so as to construct the left sparse depth map corresponding to the left frame sample image and the right sparse depth map corresponding to the right frame sample image.

[0118] Step 203: Input the left frame sample image, the right frame sample image, the left sparse depth map, and the right sparse depth map into the initial depth image generation model, output the corresponding left depth map, and calculate the corresponding depth loss function value according to the preset depth loss function model.

[0119] Optionally, step 203 includes the following sub-steps:

[0120] Input the left sparse depth map and the right sparse depth map into the corresponding initial sparse autoencoder, and output the left dense depth map and the right dense depth map.

[0121] Send the left frame sample map, the left sparse depth map, and the left dense depth map to the initial depth estimator to generate the left reverse depth map and the right reverse depth map;

[0122] Features are extracted from the left and right dense depth maps respectively through the initial skip connection module to generate the corresponding left and right residual dense depth maps.

[0123] The features of the left reverse depth map and the left residual dense depth map, and the right reverse depth map and the right residual dense depth map are added respectively to output the left disparity map and the right disparity map.

[0124] The left frame sample image and the left disparity image are weighted according to the preset depth image weights, and the left depth image is output.

[0125] Using the left sparse depth map, right sparse depth map, left dense depth map, right dense depth map, left disparity map, and right disparity map, the corresponding depth loss function values ​​are calculated according to the preset depth loss function model.

[0126] Furthermore, the depth loss function model is as follows:

[0127]

[0128] in,

[0129]

[0130]

[0131] For the depth loss function value, This is the internal loss value. This is the external loss value. Let α be the consistency loss value. in α is the balance coefficient for internal loss values. out α is the balance coefficient for external loss values. cons The balancing coefficient for the consistency loss value, where N is the total number of samples, i is the i-th sample, j is the j-th image channel, and DD ij For the left or right dense depth map of the j-th image channel of the i-th sample, SD ij For the left or right sparse depth map of the j-th image channel of the i-th sample, d ij Let be the left or right disparity map of the j-th image channel of the i-th sample. This is the left disparity map of the j-th image channel of the i-th sample. In order to combine In the case of the right disparity map of the j-th image channel of the i-th sample.

[0132] The left dense depth map and the right dense depth map refer to the dense depth maps corresponding to the left frame sample map and the right frame sample map, respectively. The dense depth map is used to represent the feature map containing more pixel information.

[0133] The left inverse depth map and the right inverse depth map refer to the inverse depth maps corresponding to the left frame sample map. The inverse depth map is used to represent the feature map containing RGB information and depth space information.

[0134] The left residual dense depth map and the right residual dense depth map refer to the residual dense depth maps corresponding to the left frame sample map and the right frame sample map, respectively. The residual dense depth map is used to represent the feature map containing more basic information and to ensure identity mapping.

[0135] The left disparity map and the right disparity map refer to the disparity maps corresponding to the left frame sample map and the right frame sample map. The disparity map is used to represent the feature map containing more depth spatial information.

[0136] The left depth map refers to the depth map corresponding to the left frame sample map.

[0137] Depth image weights, including the first sub-depth image weights corresponding to the left frame sample image and the second sub-depth image weights corresponding to the left disparity image, refer to the weights of each image in the image blending process.

[0138] See Figure 3 As shown, "L" represents the left frame sample image, "R" represents the left frame sample image, and "SD" represents the left frame sample image. L "" indicates the left sparse depth map, "SD R "" indicates the right sparse depth map, "DD" L "" indicates the left dense depth map, "DD" R "" indicates a right-hand dense depth map.

[0139] In this embodiment of the invention, the left sparse depth map and the right sparse depth map are used to generate a left dense depth map DD through their respective initial sparse autoencoders with shared weights. L And right dense depth map DD R An additional channel is created outside the initial sparse autoencoder to transfer the left sparse depth map SD. L Feed to left dense depth map DD L It is sent together with the left frame sample image L to the initial depth estimator to output the left reverse depth image D. LR And right reverse depth map D RL The images are aligned with the left and right frame sample images of the calibration stereo pair, respectively. To improve network performance, an initial skip connection module is used after depth estimation to align the left dense depth map DD with the corresponding sample images. L And right dense depth map DD R Features are extracted layer by layer and added to the features of the residual channels to generate the corresponding left residual dense depth map DD′. L And right residual dense depth map DD′ R Then, the left-reverse depth map D... LR With left residual dense depth map DD ′L Right-reverse depth map D RL With right residual dense depth map DD′ R The features are summed separately, and the left disparity map d is output. L And right disparity map d R Finally, the left frame sample image L and the left disparity image d are compared. L According to the preset depth image weights, the corresponding pixels are subjected to element-wise image weighting operations to output the left depth map. In the above image processing, according to the depth loss function model, the internal loss value is calculated using the left sparse depth map and the left dense depth map, and the right sparse depth map and the right dense depth map respectively. The external loss value is calculated using the left sparse depth map and the left disparity map, and the right sparse depth map and the right disparity map respectively. At the same time, by using the L1 penalty between the left and right disparity maps and the reconstructed disparity map, the left and right disparities are made consistent to determine the consistency loss value. The corresponding depth loss function value is determined by weighting and summing the balance coefficients corresponding to the internal loss value, the external loss value, and the consistency loss value.

[0140] Step 204: Input the original sample image, left depth image and dense depth sample image into the initial blurred image generation model to construct the background blurred prediction image, and calculate the corresponding blurred loss function value according to the preset blurred loss function model.

[0141] Optionally, step 204 includes the following sub-steps:

[0142] The initial local guidance module extracts features from the original sample image to generate a local guidance image, and the first sum of the original sample image and the local guidance image is input into the initial generator encoder for downsampling and outputting the image feature map.

[0143] An initial constraint encoder is used to perform convolution extraction, average pooling mapping, and downsampling on the second summation result of the left depth map and the local guide map to construct a depth feature map;

[0144] Image feature maps and depth feature maps are input into the initial normalization module for feature fusion, and the first fused image is output.

[0145] An upsampled map is generated by upsampling the first fused map through an initial upsampling module, and a local depth map and a local confidence map are constructed by upsampling the depth feature map through a decoder.

[0146] The upsampled image and the local depth map are input into the initial normalization module, which outputs the fused depth map and the fused confidence map.

[0147] Using the initial confidence fusion head, the fused depth map and the local depth map are weighted according to the confidence image weights corresponding to the fused confidence map and the local confidence map to construct a background blur prediction map;

[0148] The initial discriminator identifies the true probability of the output image from the dense depth sample map and the fused depth map, and calculates the corresponding blurring loss function value according to the preset blurring loss function model based on the true probability of the image.

[0149] The system consists of: a local guide map (representing feature maps containing foreground and semantic information), a first summation result (representing feature maps with more semantic relevance), an image feature map (representing feature maps containing approximate spatial depth and color information), a second summation result (representing feature maps with more local relevance), a depth feature map (representing feature maps containing approximate depth information), a first fusion map (representing feature maps containing approximate spatial depth, color information, and depth information), and an upsampled map (representing feature maps containing detailed spatial depth, color information, and depth information).

[0150] Local depth maps are used to represent feature maps that focus more on local information. Local confidence maps are used to represent feature maps that include local weight information. Fusion depth maps are used to represent feature maps that include deep spatial depth, color information, and depth information. Fusion confidence maps are used to represent feature maps that include overall weight information.

[0151] The confidence image weights, including the first sub-confidence image weights obtained from the local depth map through the local confidence map and the second sub-confidence image weights obtained from the fused depth map through the fused confidence map, refer to the weights of each image in the image mixing prediction process.

[0152] The true probability of an image includes the true probability of the first sub-image corresponding to the fused depth map and the true probability of the second sub-image corresponding to the dense depth sample map.

[0153] Preferably, the virtualization loss function model is as follows:

[0154]

[0155] in,

[0156]

[0157]

[0158]

[0159] To blur the loss function value, The discriminator loss value. This represents the generator loss value. To predict the loss value, β pred The balancing coefficients for predicting loss values, where r is the original sample image, D is the discriminator, G is the generator encoder, M is the constraint encoder, and d... raw For the left depth map, D raw For the set of left depth maps, d gt For dense depth sample maps, D gt It is a collection of dense depth sample maps. The data from the left depth map is processed by the entropy of the discriminator. M(d) is the entropy of the data from the dense depth sample map through the discriminator. raw ) represents the depth feature map, G(M(d) raw ) represents the upsampled image, D(G(M(d) raw Let D(d)|r) be the true probability of the first sub-image of the fused depth images, and D(d) be the true probability of the first sub-image. gt |r) represents the true probability of the second sub-image of the dense depth sample map, L1(G(M(d)) raw )) represents the L1 loss of the upsampled image, λ is the balance coefficient of the L1 loss of the upsampled image, and L1(d) pred d raw ) represents the L1 loss of the background blur prediction map and the left depth map, d pred For the background blur prediction image, d pred (i,j) is the background blur prediction map of the j-th image channel of the i-th sample, d l (i,j) is the local depth map of the j-th image channel of the i-th sample, c l(i,j) is the local confidence map of the j-th image channel of the i-th sample. The first sub-confidence image weight for the local depth map of the j-th image channel of the i-th sample, d f (i,j) is the fusion depth map of the j-th image channel of the i-th sample, c f (i,j) is the fusion confidence map of the j-th image channel of the i-th sample. The second sub-confidence image weight is the fusion depth map of the j-th image channel of the i-th sample.

[0160] In this embodiment of the invention, the input original sample image undergoes feature extraction via an initial local guidance module to generate a local guidance map. This guides the constraint network to focus on local depth correlations, thereby improving the quality of the generated local depth map. The local guidance map is then added to the original sample image in the RDF-GAN, and the first addition result is input into the initial generator encoder. The initial generator encoder learns the spatial mapping of the original sample image to perform downsampling and output an image feature map. The local guidance map is then added to the left depth map to output a second addition result. This second addition result is used in the constraint network. First, a convolutional layer performs preliminary feature extraction, followed by an average pooling layer to increase the receptive field and capture more features. The size of the acquired features is progressively compressed. By performing downsampling, the mapping from the left depth map to the depth space z is learned, resulting in a constructed depth feature map. The decoder then upsamples the depth feature map to construct a local depth map and a local confidence map. The image feature map obtained after learning the spatial mapping is then compared with the depth feature map. Figure 1 The input is fed into the initial normalization module for feature information sharing and fusion. The first fused image is then fed into the initial upsampling module for upsampling, gradually expanding the features and reducing the number of feature channels to generate an upsampled image. The upsampled image and the local depth map are then fed back into the initial normalization module, outputting a fused depth map and a fused confidence map. The local depth map obtains higher confidence in regions where the original depth values ​​are more accurate, while the fused depth map has higher confidence in large missing and noisy regions. Through the initial confidence fusion head, the fused depth map and the local depth map are weighted according to the confidence image weights corresponding to the fused confidence map and the local confidence map to construct a background blur prediction map. Simultaneously, the fused depth map and dense depth samples... Figure 1 The input initial discriminator can identify whether the input fused depth map and dense depth sample map are true by classifying them, and output the corresponding true probability of the image. The true probability of the image is then used to calculate the corresponding blurring loss function value according to the preset blurring loss function model.

[0161] Step 205: Perform a summation operation on the depth loss function value and the blurring loss function value to determine the total loss function value.

[0162] In this embodiment of the invention, the total loss function value during the entire training process can be determined by summing the depth loss function value and the blurring loss function value corresponding to the initial depth image generation model and the initial blurring image generation model, respectively.

[0163] Step 206: Optimize the initial depth image generation model and the initial bokeh image generation model based on the total loss function value until the total loss function value converges, and output the target depth image generation model and the target bokeh image generation model.

[0164] Optionally, step 203 includes the following sub-steps:

[0165] Determine whether the total loss function value has converged;

[0166] If not, then update the model parameters and depth image weights of the initial depth image generation model and the initial bokeh image generation model through backpropagation, and jump to the step of training the initial depth image generation model and the initial bokeh image generation model with sample images respectively to determine the total loss function value;

[0167] If so, the initial depth image generation model is used as the target depth image generation model, and the initial bokeh image generation model is used as the target bokeh image generation model.

[0168] In this embodiment of the invention, the convergence of the total loss function value obtained from training is judged. If the total loss function value has not converged, the feature error reflected by the total loss function value is analyzed, and the model parameters of the initial depth image generation model and the initial bokeh image generation model are updated through backpropagation. At the same time, the depth image weights involved in the initial depth image generation model are adjusted, and the process jumps to the step of training the initial depth image generation model and the initial bokeh image generation model with sample images respectively to determine the total loss function value. This process continues until the total loss function value is determined to have converged. Then, the target depth image generation model and the target bokeh image generation model are output, and the preset depth image weights are used as the target depth image weights.

[0169] Step 207: When the target image is received, the corresponding sparse depth map of the target is constructed using the visual odometry method.

[0170] In this embodiment of the invention, the specific implementation process of step 207 is similar to that of step 104, and will not be repeated here.

[0171] Step 208: Based on the target image and the target sparse depth map, construct the corresponding target depth map using the target depth image generation model.

[0172] Optionally, the target depth image generation model includes a target sparse autoencoder, a target depth estimator, and a target skip connection module; step 203 includes the following sub-steps:

[0173] Input the sparse depth map of the target into the sparse autoencoder of the target to generate a dense depth map of the target;

[0174] A target depth estimator is used to estimate the depth of the target image, the target sparse depth, and the target dense depth map, and to construct a target inverse depth map.

[0175] The target dense depth map is feature extracted by the target jump connection module, and the target residual dense depth map is output.

[0176] The target inverse depth map and the target residual dense depth map are feature-added to output the target disparity map;

[0177] The target depth map is generated by performing element-wise image weighting operations based on the target image and the target disparity map corresponding to the target depth image weights.

[0178] Target dense depth map refers to the dense depth map corresponding to the target image. Target reverse depth map refers to the reverse depth map corresponding to the target image. Target residual dense depth map refers to the residual dense depth map corresponding to the target image. Target disparity map refers to the disparity map corresponding to the target image.

[0179] Understandably, when processing the target image, only one dense depth map, one inverse depth map, one residual dense depth map, and one disparity map are generated.

[0180] In this embodiment of the invention, corresponding to the initial module composition of the initial depth image generation model, the output target depth image generation model includes a corresponding target sparse autoencoder, a target depth estimator, a target skip connection module, and associated target depth image weights, after adjusting the model parameters until the total loss function converges. The target sparse depth map is input into the target sparse autoencoder of the target depth image generation model to generate a target dense depth map. The target depth estimator then performs depth estimation on the target image, target sparse depth, and target dense depth map to construct a target inverse depth map. Features from the target inverse depth map and the target residual dense depth map are summed to determine the target disparity map. Finally, element-wise image weighting operations are performed on the target image and the target disparity map according to the target depth image weights to output the target depth map.

[0181] Step 209: Based on the target depth map, construct the corresponding target background blur map using the target blur image generation model.

[0182] Optionally, step 203 includes the following sub-steps:

[0183] The target depth map is input into the target generator encoder in the target blurring image generation model for downsampling, and the target image feature map is output.

[0184] The target image feature map is upsampled by the target upsampling module in the target blurring image generation model to construct the corresponding target background blurring image.

[0185] The target image feature map refers to the image feature map corresponding to the target image.

[0186] In this embodiment of the invention, corresponding to the initial module composition of the initial blurred image generation model, the output target blurred image generation model, after adjusting the model parameters until the total loss function converges, includes a corresponding target local guidance module, a target generator encoder, a target normalization module, a target upsampling module, a target discriminator, a target constraint encoder, a target decoder, and a target confidence fusion head. The target depth map is input into the target generator encoder in the target blurred image generation model for downsampling, and the output target image feature map is then sent to the target upsampling module in the target blurred image generation model. After performing the upsampling operation, the corresponding target background blurred image is constructed.

[0187] In this embodiment of the invention, after constructing an initial depth image generation model and an initial bokeh image generation model, a visual odometry method is used to construct corresponding left and right sparse depth maps based on the left and right frame sample images. The initial depth image generation model outputs the corresponding left depth map based on the left and right frame sample images, the left sparse depth map, and the right sparse depth map, and calculates the corresponding depth loss function value according to a preset depth loss function model. The original sample image, the left depth map, and the dense depth sample image are input into the initial bokeh image generation model to construct a background bokeh prediction map, and the corresponding bokeh loss function value is calculated according to the preset bokeh loss function model. The blurring loss function value is calculated by summing the depth loss function value and the blurring loss function value to determine the total loss function value. This total loss function value is then used to optimize the initial depth image generation model and the initial blurring image generation model. When the total loss function value converges, the target depth image generation model and the target blurring image generation model are output. When a target image is received, a corresponding target sparse depth map is constructed using visual odometry. The target depth image generation model then constructs a corresponding target depth map based on the target image and the target sparse depth map. Based on the target depth map, the target blurring image generation model constructs a corresponding target background blurring map. Throughout the entire image background blurring process, multi-model fusion improves the real-time performance and effectiveness of depth prediction, resulting in higher quality generated background blurring images.

[0188] Please see Figure 6 , Figure 6This is a structural block diagram of an image background blurring processing device provided in Embodiment 3.

[0189] The model building module 601 is used to build an initial depth image generation model and an initial bokeh image generation model when a sample image set is received;

[0190] The total loss function value determination module 602 is used to train the initial depth image generation model and the initial bokeh image generation model using the sample image set respectively, and determine the total loss function value;

[0191] The target model output module 603 is used to optimize the initial depth image generation model and the initial bokeh image generation model based on the total loss function value until the total loss function value converges, and output the target depth image generation model and the target bokeh image generation model.

[0192] The target sparse depth map construction module 604 is used to construct the corresponding target sparse depth map by using visual odometry when a target image is received.

[0193] The target depth image construction module 605 is used to construct the corresponding target depth map based on the target image and the target sparse depth map through the target depth image generation model.

[0194] The target background blurring image construction module 606 is used to construct the corresponding target background blurring image based on the target depth map and through the target blurring image generation model.

[0195] Optionally, the model building module 601 is specifically used for:

[0196] When a set of sample images is received, an initial depth image generation model is constructed using an initial sparse autoencoder, an initial depth estimator, and an initial skip connection module.

[0197] An initial blurred image generation model is constructed using an initial local guidance module, an initial generator encoder, an initial normalization module, an initial upsampling module, an initial discriminator, an initial constraint encoder, an initial decoder, and an initial confidence fusion head.

[0198] Optionally, the sample image set includes a left frame sample image, a right frame sample image, the original sample image, and a dense depth sample image; the total loss function value determination module 602 includes:

[0199] The sparse depth map construction unit is used to construct the corresponding left sparse depth map and right sparse depth map based on the left frame sample map and the right frame sample map using the visual odometry method.

[0200] The depth loss function value determination unit is used to input the left frame sample image, the right frame sample image, the left sparse depth image and the right sparse depth image into the initial depth image generation model, output the corresponding left depth image, and calculate the corresponding depth loss function value according to the preset depth loss function model.

[0201] The blurring loss function value determination unit is used to input the original sample image, the left depth image, and the dense depth sample image into the initial blurring image generation model, construct the background blurring prediction image, and calculate the corresponding blurring loss function value according to the preset blurring loss function model.

[0202] The total loss function value calculation unit is used to perform a summation operation on the depth loss function value and the blurring loss function value to determine the total loss function value.

[0203] Optionally, the depth loss function value determination unit is specifically used for:

[0204] Input the left sparse depth map and the right sparse depth map into the corresponding initial sparse autoencoder, and output the left dense depth map and the right dense depth map.

[0205] Send the left frame sample map, the left sparse depth map, and the left dense depth map to the initial depth estimator to generate the left reverse depth map and the right reverse depth map;

[0206] Features are extracted from the left and right dense depth maps respectively through the initial skip connection module to generate the corresponding left and right residual dense depth maps.

[0207] The features of the left reverse depth map and the left residual dense depth map, and the right reverse depth map and the right residual dense depth map are added respectively to output the left disparity map and the right disparity map.

[0208] The left frame sample image and the left disparity image are weighted according to the preset depth image weights, and the left depth image is output.

[0209] Using the left sparse depth map, right sparse depth map, left dense depth map, right dense depth map, left disparity map, and right disparity map, the corresponding depth loss function values ​​are calculated according to the preset depth loss function model.

[0210] Optionally, the unit for determining the virtualization loss function value is specifically used for:

[0211] The initial local guidance module extracts features from the original sample image to generate a local guidance image, and the first sum of the original sample image and the local guidance image is input into the initial generator encoder for downsampling and outputting the image feature map.

[0212] An initial constraint encoder is used to perform convolution extraction, average pooling mapping, and downsampling on the second summation result of the left depth map and the local guide map to construct a depth feature map;

[0213] Image feature maps and depth feature maps are input into the initial normalization module for feature fusion, and the first fused image is output.

[0214] An upsampled map is generated by upsampling the first fused map through an initial upsampling module, and a local depth map and a local confidence map are constructed by upsampling the depth feature map through a decoder.

[0215] The upsampled image and the local depth map are input into the initial normalization module, which outputs the fused depth map and the fused confidence map.

[0216] Using the initial confidence fusion head, the fused depth map and the local depth map are weighted according to the confidence image weights corresponding to the fused confidence map and the local confidence map to construct a background blur prediction map;

[0217] The initial discriminator identifies the true probability of the output image from the dense depth sample map and the fused depth map, and calculates the corresponding blurring loss function value according to the preset blurring loss function model based on the true probability of the image.

[0218] Optionally, the target model output module 603 is specifically used for:

[0219] Determine whether the total loss function value has converged;

[0220] If not, then update the model parameters and depth image weights of the initial depth image generation model and the initial bokeh image generation model through backpropagation, and jump to the step of training the initial depth image generation model and the initial bokeh image generation model with sample images respectively to determine the total loss function value;

[0221] If so, the initial depth image generation model is used as the target depth image generation model, and the initial bokeh image generation model is used as the target bokeh image generation model.

[0222] Optionally, the target depth image generation model includes a target sparse autoencoder, a target depth estimator, and a target skip connection module; the target depth image construction module 605 is specifically used for:

[0223] Input the sparse depth map of the target into the sparse autoencoder of the target to generate a dense depth map of the target;

[0224] A target depth estimator is used to estimate the depth of the target image, the target sparse depth, and the target dense depth map, and to construct a target inverse depth map.

[0225] The target dense depth map is feature extracted by the target jump connection module, and the target residual dense depth map is output.

[0226] The target inverse depth map and the target residual dense depth map are feature-added to output the target disparity map;

[0227] The target depth map is generated by performing element-wise image weighting operations based on the target image and the target disparity map corresponding to the target depth image weights.

[0228] Optionally, the target background blurring image construction module 606 is specifically used for:

[0229] The target depth map is input into the target generator encoder in the target blurring image generation model for downsampling, and the target image feature map is output.

[0230] The target image feature map is upsampled by the target upsampling module in the target blurring image generation model to construct the corresponding target background blurring image.

[0231] This invention also provides an electronic device, characterized in that it includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the super-resolution image background blurring method as described in any of the above embodiments.

[0232] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0233] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0234] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0235] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0236] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for blurring the background of an image, characterized in that, include: When a set of sample images is received, an initial depth image generation model and an initial bokeh image generation model are constructed. The initial depth image generation model and the initial blurred image generation model are trained using the sample image set, and the total loss function value is determined. The initial depth image generation model and the initial bokeh image generation model are optimized based on the total loss function value until the total loss function value converges, and the target depth image generation model and the target bokeh image generation model are output. When a target image is received, a corresponding sparse depth map of the target is constructed using visual odometry. Based on the target image and the target sparse depth map, a corresponding target depth map is constructed using the target depth image generation model. Based on the target depth map, a corresponding target background blur map is constructed using the target blur image generation model; The target depth image generation model includes a target sparse autoencoder, a target depth estimator, and a target skip connection module; The step of constructing a corresponding target depth map using the target depth image generation model based on the target image and the target sparse depth map includes: The target sparse depth map is input into the target sparse autoencoder to generate the target dense depth map; The target depth estimator is used to estimate the depth of the target image, the target sparse depth, and the target dense depth map to construct a target inverse depth map. The target jump connection module extracts features from the target dense depth map and outputs the target residual dense depth map. The target inverse depth map and the target residual dense depth map are feature-added to output a target disparity map; The target depth map is generated by performing element-wise image weighting operations based on the target image and the target disparity map corresponding to the target depth image weights.

2. The image background blurring method according to claim 1, characterized in that, The steps of constructing an initial depth image generation model and an initial bokeh image generation model when a sample image set is received include: When a set of sample images is received, an initial depth image generation model is constructed using an initial sparse autoencoder, an initial depth estimator, and an initial skip connection module. An initial blurred image generation model is constructed using an initial local guidance module, an initial generator encoder, an initial normalization module, an initial upsampling module, an initial discriminator, an initial constraint encoder, an initial decoder, and an initial confidence fusion head.

3. The image background blurring method according to claim 2, characterized in that, The sample image set includes a left frame sample image, a right frame sample image, an original sample image, and a dense depth sample image; The step of training the initial depth image generation model and the initial blurred image generation model using the sample image set respectively, and determining the total loss function value, includes: Based on the left frame sample image and the right frame sample image, the corresponding left sparse depth map and right sparse depth map are constructed using the visual odometry method. The left frame sample image, the right frame sample image, the left sparse depth image, and the right sparse depth image are input into the initial depth image generation model, and the corresponding left depth image is output. The corresponding depth loss function value is calculated according to the preset depth loss function model. The original sample image, the left depth image, and the dense depth sample image are input into the initial blurred image generation model to construct a background blur prediction image, and the corresponding blur loss function value is calculated according to the preset blur loss function model. The total loss function value is determined by summing the depth loss function value and the blurring loss function value.

4. The image background blurring method according to claim 3, characterized in that, The steps of inputting the left frame sample image, the right frame sample image, the left sparse depth image, and the right sparse depth image into the initial depth image generation model, outputting the corresponding left depth image, and calculating the corresponding depth loss function value according to the preset depth loss function model include: The left sparse depth map and the right sparse depth map are respectively input into the corresponding initial sparse autoencoder, and the left dense depth map and the right dense depth map are output. The left frame sample image, the left sparse depth image, and the left dense depth image are sent to the initial depth estimator to generate a left reverse depth image and a right reverse depth image; The initial skip connection module extracts features from the left dense depth map and the right dense depth map respectively to generate the corresponding left residual dense depth map and right residual dense depth map. The left reverse depth map is added to the left residual dense depth map, and the right reverse depth map is added to the right residual dense depth map, respectively, to output the left disparity map and the right disparity map; The left frame sample image and the left disparity image are respectively weighted according to the preset depth image weights, and the left depth image is output. Using the left sparse depth map, the right sparse depth map, the left dense depth map, the right dense depth map, the left disparity map, and the right disparity map, the corresponding depth loss function values ​​are calculated according to a preset depth loss function model.

5. The image background blurring method according to claim 3, characterized in that, The steps of inputting the original sample image, the left depth image, and the dense depth sample image into the initial blurred image generation model to construct a background blurred prediction image, and calculating the corresponding blurred loss function value according to the preset blurred loss function model, include: The initial local guidance module extracts features from the original sample image to generate a local guidance image, and the first sum of the original sample image and the local guidance image is input into the initial generator encoder for downsampling and outputting an image feature map. Using the initial constraint encoder, the second summation result of the left depth map and the local guide map is subjected to convolution extraction, average pooling mapping and downsampling to construct a depth feature map; The image feature map and the depth feature map are input into the initial normalization module for feature fusion, and a first fused image is output. The first fused map is upsampled by the initial upsampling module to generate an upsampled map, and the depth feature map is upsampled by the decoder to construct a local depth map and a local confidence map; The upsampled image and the local depth image are input into the initial normalization module, and the fused depth image and the fused confidence image are output. Using the initial confidence fusion head, the fused depth map and the local depth map are subjected to image weighting operations according to the confidence image weights corresponding to the fused confidence map and the local confidence map, and a background blur prediction map is constructed. The initial discriminator identifies the true probability of the output image by distinguishing the dense depth sample map and the fused depth map, and uses the true probability of the image to calculate the corresponding blurring loss function value according to the preset blurring loss function model.

6. The image background blurring method according to claim 4, characterized in that, The step of optimizing the initial depth image generation model and the initial blurred image generation model based on the total loss function value until the total loss function value converges, and then outputting the target depth image generation model and the target blurred image generation model, includes: Determine whether the total loss function value has converged; If not, the model parameters of the initial depth image generation model and the initial bokeh image generation model, as well as the depth image weights, are updated through backpropagation, and the process jumps to the step of training the initial depth image generation model and the initial bokeh image generation model using the sample images respectively to determine the total loss function value. If so, the initial depth image generation model is used as the target depth image generation model, and the initial bokeh image generation model is used as the target bokeh image generation model.

7. The image background blurring method according to claim 1, characterized in that, The step of constructing a corresponding target background blur map based on the target depth map using the target blur image generation model includes: The target depth map is input into the target generator encoder in the target blurred image generation model for downsampling, and the target image feature map is output. The target image feature map is upsampled by the target upsampling module in the target blurring image generation model to construct the corresponding target background blurring image.

8. An image background blurring device, characterized in that, include: The model building module is used to build an initial depth image generation model and an initial bokeh image generation model when a sample image set is received. The total loss function value determination module is used to train the initial depth image generation model and the initial blurred image generation model using the sample image set, respectively, and determine the total loss function value. The target model output module is used to optimize the initial depth image generation model and the initial bokeh image generation model based on the total loss function value until the total loss function value converges, and output the target depth image generation model and the target bokeh image generation model. The target sparse depth map construction module is used to construct the corresponding target sparse depth map using visual odometry when a target image is received. The target depth image construction module is used to construct a corresponding target depth map based on the target image and the target sparse depth map through the target depth image generation model; The target background blurring image construction module is used to construct a corresponding target background blurring image based on the target depth map and through the target blurring image generation model; The target depth image generation model includes a target sparse autoencoder, a target depth estimator, and a target skip connection module; the target depth image construction module is specifically used for: The target sparse depth map is input into the target sparse autoencoder to generate the target dense depth map; The target depth estimator is used to estimate the depth of the target image, the target sparse depth, and the target dense depth map to construct a target inverse depth map. The target jump connection module extracts features from the target dense depth map and outputs the target residual dense depth map. The target inverse depth map and the target residual dense depth map are feature-added to output a target disparity map; The target depth map is generated by performing element-wise image weighting operations based on the target image and the target disparity map corresponding to the target depth image weights.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the image background blurring method as described in any one of claims 1-7.

Images