Dynamic scene full-focus image synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a dynamic scene full-focus image synthesis method and device, electronic equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a plurality of source images under a plurality of focusing distances collected in the same scene region, each source image corresponding to a different focusing distance; separating based on the scene depth information of the plurality of source images to obtain a dynamic area and a static area of the scene area; respectively performing full-focus reconstruction on the dynamic area and the static area based on the plurality of source images to generate a first full-focus image and a second full-focus image; and fusing the first full-focus image and the second full-focus image to generate a target full-focus image. Through the method provided by the invention, the problem that the synthesis quality is reduced due to motion blur and defocus artifacts in a dynamic scene in a traditional method is effectively solved, the detail definition of the final full-focus image is improved, and a better image synthesis effect is obtained while efficient processing is ensured.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for synthesizing dynamic scene full-focus images. Background Technology

[0002] Traditional all-focus image synthesis techniques typically rely on image sequences acquired at multiple fixed focal lengths and then using fusion algorithms to generate a full depth-of-field image. However, these methods face fundamental challenges in dynamic scenes. The choice of focusing distance is often pre-set or uniformly sampled, failing to optimize for the actual depth distribution of the scene. This results in numerous redundant shots or missed key focal points in complex depth-of-field environments, leading to low acquisition efficiency. Consequently, the synthesized all-focus image exhibits motion blur and distortion in dynamic areas and lacks detail in static areas, resulting in a significant overall quality degradation.

[0003] Therefore, developing a synthesis method that can intelligently optimize acquisition strategies and specifically handle dynamic scenes has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for synthesizing dynamic scene full-focus images, in order to overcome the deficiencies existing in the prior art.

[0005] This invention provides a method for synthesizing dynamic scene full-focus images, comprising the following steps: Acquire multiple source images at different focusing distances captured in the same scene area, with each source image corresponding to a different focusing distance; Based on the scene depth information of the multiple source images, the dynamic region and static region of the scene area are obtained; Based on the multiple source images, full-focus reconstruction is performed on the dynamic region and the static region respectively to generate a first full-focus image and a second full-focus image; The first full-focus image and the second full-focus image are fused to generate the target full-focus image.

[0006] According to the present invention, a method for synthesizing a dynamic scene full-focus image, wherein the separation based on the scene depth information of the multiple source images to obtain the dynamic region and static region of the scene region includes: The multiple focusing distances are dynamically optimized based on the scene depth information of the multiple source images to determine the target focusing distance; Based on the target focusing distance, the content in the scene area is separated according to its dynamic and static attributes to obtain the dynamic and static areas of the scene area.

[0007] According to a dynamic scene full-focus image synthesis method provided by the present invention, the step of dynamically optimizing the multiple focus distances based on the scene depth information of the multiple source images to determine the target focus distance includes: Depth estimation is performed on the multiple source images to generate scene depth information; Based on the scene depth information, a depth-pixel density histogram is constructed; Based on the depth-pixel density histogram, the multiple focusing distances are iterated through an optimization algorithm to determine the target focusing distance, so as to minimize the number of selected focusing distances while satisfying the preset pixel coverage.

[0008] According to the present invention, a dynamic scene full-focus image synthesis method is provided, wherein determining the target focus distance by iteratively optimizing the plurality of focus distances based on the depth-pixel density histogram using an optimization algorithm includes: Based on the depth-pixel density histogram, the focusing distance corresponding to the depth value with the highest pixel density is determined as the initial focusing distance; The initial focusing distance is iteratively optimized using the mean-shift clustering algorithm and the adaptive Gaussian kernel function based on the depth-of-field model to obtain the optimized focusing distance; Based on the optimized focusing distance and the corresponding depth of field range, update the depth-pixel density histogram and remove pixels that have been covered by the depth of field range; Repeat the above determination, optimization and update steps until the number of remaining pixels in the updated depth-pixel density histogram is less than the preset threshold, and use all optimized focusing distances as the target focusing distance.

[0009] According to a dynamic scene full-focus image synthesis method provided by the present invention, the step of performing full-focus reconstruction on the dynamic region and the static region based on the multiple source images respectively to generate a first full-focus image and a second full-focus image includes: Based on the multiple source images, a first neural network model is used to perform full-focus reconstruction on the dynamic region to generate the first full-focus image, and a second neural network model is used to perform full-focus reconstruction on the static region to generate the second full-focus image. The first neural network model and the second neural network model have different network architectures.

[0010] The present invention also provides a dynamic scene full-focus image synthesis device, comprising the following modules: The acquisition module is used to acquire multiple source images at multiple focusing distances in the same scene area, with each source image corresponding to a different focusing distance; The decoupling module is used to separate the scene depth information based on the multiple source images to obtain the dynamic region and the static region of the scene area; The reconstruction module is used to perform full-focus reconstruction on the dynamic region and the static region based on the multiple source images respectively, and generate a first full-focus image and a second full-focus image; The fusion module is used to fuse the first full-focus image and the second full-focus image to generate a target full-focus image.

[0011] According to the present invention, a dynamic scene full-focus image synthesis device is provided, wherein the decoupling module is specifically used for: The multiple focusing distances are dynamically optimized based on the scene depth information of the multiple source images to determine the target focusing distance; Based on the target focusing distance, the content in the scene area is separated according to its dynamic and static attributes to obtain the dynamic and static areas of the scene area.

[0012] According to the present invention, a dynamic scene full-focus image synthesis device is provided, the device further comprising a dynamic optimization module, specifically used for: Depth estimation is performed on the multiple source images to generate scene depth information; Based on the scene depth information, a depth-pixel density histogram is constructed; Based on the depth-pixel density histogram, the multiple focusing distances are iterated through an optimization algorithm to determine the target focusing distance, so as to minimize the number of selected focusing distances while satisfying the preset pixel coverage.

[0013] According to the present invention, a dynamic scene full-focus image synthesis device is provided, the device further comprising a dynamic focusing module, specifically used for: Based on the depth-pixel density histogram, the focusing distance corresponding to the depth value with the highest pixel density is determined as the initial focusing distance; The initial focusing distance is iteratively optimized using the mean-shift clustering algorithm and the adaptive Gaussian kernel function based on the depth-of-field model to obtain the optimized focusing distance; Based on the optimized focusing distance and the corresponding depth of field range, update the depth-pixel density histogram and remove pixels that have been covered by the depth of field range; Repeat the above determination, optimization and update steps until the number of remaining pixels in the updated depth-pixel density histogram is less than the preset threshold, and use all optimized focusing distances as the target focusing distance.

[0014] According to the present invention, a dynamic scene full-focus image synthesis device is provided, wherein the reconstruction module is specifically used for: Based on the multiple source images, a first neural network model is used to perform full-focus reconstruction on the dynamic region to generate the first full-focus image, and a second neural network model is used to perform full-focus reconstruction on the static region to generate the second full-focus image. The first neural network model and the second neural network model have different network architectures.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dynamic scene full-focus image synthesis method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic scene full-focus image synthesis method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the dynamic scene full-focus image synthesis method as described above.

[0018] This invention provides a method, apparatus, electronic device, and storage medium for synthesizing dynamic scene full-focus images. It acquires multiple source images at different focusing distances within the same scene region, each source image corresponding to a different focusing distance. Based on the scene depth information of the multiple source images, the dynamic and static regions of the scene region are separated. Full-focus reconstruction is then performed on the dynamic and static regions based on the multiple source images to generate a first full-focus image and a second full-focus image. The first and second full-focus images are then fused to generate a target full-focus image. Therefore, this invention effectively overcomes the degradation in synthesis quality caused by motion blur and defocus artifacts in dynamic scenes using traditional methods by separately processing the dynamic and static regions and performing targeted full-focus reconstruction. This significantly improves the detail clarity and visual consistency of the final full-focus image, making it particularly suitable for complex scenes containing moving objects. It achieves superior image synthesis results while ensuring efficient processing. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1This is a flowchart illustrating the dynamic scene full-focus image synthesis method provided by the present invention.

[0021] Figure 2 This is a flowchart illustrating the dynamic focusing process in the dynamic scene full-focus image synthesis method provided by the present invention.

[0022] Figure 3 This is a comparison diagram of the training and inference processes of the conditional diffusion model provided by this invention.

[0023] Figure 4 This is a system architecture diagram of the dynamic scene full-focus image synthesis method provided by the present invention.

[0024] Figure 5 This is a schematic diagram illustrating the effect of the dynamic scene full-focus image synthesis method provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the dynamic scene full-focus image synthesis device provided by the present invention.

[0026] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The following is combined with Figures 1-7 This invention describes a dynamic scene full-focus image synthesis method, apparatus, electronic device, and storage medium.

[0029] Among the related technologies, there are several methods for synthesizing full-focus images.

[0030] Traditional full-focus fusion methods. Traditional fusion methods are mainly based on signal processing and image feature analysis, and can be further subdivided into transform domain methods and spatial domain methods. The core idea of ​​transform domain methods is to transform the source image from the spatial domain to another mathematical transform domain for processing. Its general process includes three stages: First, the source image is decomposed into a series of coefficients representing different scales or frequencies using some image decomposition algorithm (such as Laplacian pyramid, wavelet transform, shear wavelet transform, etc.); then, specific fusion rules (such as the "select maximum value" rule) are designed to merge these coefficients from different source images to retain the features of the clearest parts of each image; finally, the fused coefficients are reconstructed into the final full-focus image through the corresponding inverse transform. The mainstream of this type of method includes methods based on multi-scale decomposition (MSD), such as pyramid transform and wavelet transform, which can effectively separate image information at different levels, but pyramid transform may produce blurring at the fusion boundary, while wavelet transform lacks translation invariance. Another major category is sparse representation (SR) methods, which use an overcomplete dictionary to represent image patches as linear combinations of sparse atoms and achieve image fusion by fusing sparse coefficients. This method can handle image noise well, but it has high computational complexity and is not ideal for handling details such as edges. Spatial domain methods are more direct. They do not perform domain transformations but operate directly at the pixel or pixel block level of the image. The main process is to first divide the source image into several regions (such as fixed-size blocks or irregular regions), then calculate the sharpness or "activity" of each region, and finally determine which source image each region or pixel in the final fused image comes from based on the sharpness metric. For example, block-based segmentation methods divide the image into multiple fixed-size blocks for independent processing. Although computationally efficient, they are prone to "blocking artifacts" or blurring at block boundaries. Whether it is a transform domain method or a spatial domain method, they all rely on manually designed features and rules to determine and extract focus information. The rule design is complex and difficult to handle complex scene scenarios.

[0031] Deep learning-based full-focus synthesis methods utilize the powerful feature extraction and representation capabilities of neural networks to automatically distinguish between in-focus and out-of-focus areas in an image, thereby generating a high-quality fused image. In practice, these methods employ various advanced neural network architectures. Early or fundamental methods typically use deep convolutional neural networks (CNNs) as their core. The general process involves inputting a set of source images (i.e., a "focus stack") taken at different focal lengths into the network. The CNN, through its multi-layered convolutions and non-linear activations, learns and extracts deep features that distinguish between sharp (focused) and blurry (out-of-focus) pixels. Based on these features, the network generates a pixel-level "decision map" or "weight map" for each source image, where the values ​​represent the sharpness of the corresponding pixel. Finally, guided by the decision map, all source images are fused at the pixel level using a weighted average to obtain the final full-focus image. To further enhance the realism and detail of the fused image, some existing techniques have introduced the framework of generative adversarial networks (GANs), such as FuseGAN and MFIF-GAN. These methods add a discriminator to the basic fusion network (generator). The generator is responsible for producing the fused image, while the discriminator learns to distinguish between the "fake" images generated by the generator and the ideal "real" images. Through adversarial training between the two, the generator is "forced" to learn a more refined fusion strategy to produce fused results with sharper edges and fewer artifacts. In recent years, with the success of the Transformer model in computer vision, some recent methods (such as SwinFusion and TransFusion-Net) have also begun to adopt this architecture. Unlike CNNs, which mainly focus on local pixel neighborhoods, the Transformer, through its core self-attention mechanism, can capture long-distance dependencies between any two pixels in an image. This ability to establish a global receptive field allows it to more accurately understand the focus distribution of the entire image, generating more accurate focus maps than traditional CNNs, thereby further improving fusion performance. Although deep learning methods have made significant progress in full-focus image synthesis tasks, existing methods still face many challenges in dynamic scenes. The motion of objects and changes in illumination in dynamic scenes place higher demands on the accuracy and robustness of image fusion. Meanwhile, most existing methods rely on static image datasets for training, lacking adaptability to dynamic scenes, and some methods are even fundamentally unsuitable for dynamic scenes.

[0032] Based on this, the present invention provides a dynamic scene full-focus image synthesis method to solve at least one of the above problems.

[0033] Figure 1 This is a flowchart illustrating the dynamic scene full-focus image synthesis method provided by the present invention, as shown below. Figure 1As shown, the method includes the following: Step 100: Acquire multiple source images at multiple focusing distances in the same scene area, with each source image corresponding to a different focusing distance.

[0034] Specifically, multiple source images are captured in the same scene area at multiple different focusing distances using image acquisition devices (such as mobile phones, cameras, or microscopes). Each image clearly presents details at different depth levels in the scene due to the difference in focal length, and together they form a "focus stack" that covers the entire depth of field.

[0035] Step 200: Based on the scene depth information of the multiple source images, separate the scene region into dynamic and static regions.

[0036] Specifically, scene depth information is estimated based on these source images, which is a depth map in which each pixel is labeled with its actual physical distance from the camera. Based on this, the scene content is separated into dynamic regions (such as people walking, vehicles moving, etc. whose positions change over time or between frames) and static regions (such as background buildings, stationary furniture, etc., which remain unchanged in the scene).

[0037] Step 300: Based on the multiple source images, perform full-focus reconstruction on the dynamic region and the static region respectively to generate a first full-focus image and a second full-focus image.

[0038] Specifically, full-focus reconstruction is performed separately for the different characteristics of these two types of regions: for dynamic regions, a first type of neural network, such as the conditional diffusion model, which is good at handling temporal consistency and motion blur, is used to generate the first full-focus image; for static regions, a second type of neural network, such as the multi-scale convolutional neural network, which is good at detail enhancement and structural fidelity, is used to generate the second full-focus image. The two types of model architectures are significantly different to adapt to their respective task requirements.

[0039] Step 400: Fuse the first full-focus image and the second full-focus image to generate a target full-focus image.

[0040] Specifically, a lightweight fusion network seamlessly merges two separately optimized full-focus images, resolving potential edge mismatches or brightness differences, ultimately outputting a high-quality target full-focus image that is clear overall, rich in detail, and free of motion artifacts. This method is particularly suitable for multi-focal-length image synthesis applications in dynamic scenarios such as mobile photography, microscopic imaging, and security monitoring.

[0041] The above describes the steps of the dynamic scene full-focus image synthesis method provided by this invention. As can be seen from the above description, the dynamic scene full-focus image synthesis method provided by this invention acquires multiple source images at multiple focusing distances within the same scene region, each source image corresponding to a different focusing distance; separates the scene depth information of the multiple source images to obtain the dynamic region and static region of the scene region; performs full-focus reconstruction on the dynamic region and the static region based on the multiple source images respectively, generating a first full-focus image and a second full-focus image; and fuses the first full-focus image and the second full-focus image to generate a target full-focus image. Therefore, this invention, by processing the dynamic region and static region separately and performing targeted full-focus reconstruction, effectively overcomes the problem of decreased synthesis quality caused by motion blur and defocus artifacts in dynamic scenes using traditional methods, significantly improving the detail clarity and visual consistency of the final full-focus image. It is particularly suitable for complex scenes containing moving objects, achieving superior image synthesis results while ensuring efficient processing.

[0042] Based on the above embodiments, in this embodiment, step 200 separates the scene depth information of the multiple source images to obtain the dynamic and static regions of the scene region, including: Step 210: Dynamically optimize the multiple focusing distances based on the scene depth information of the multiple source images to determine the target focusing distance.

[0043] Step 220: Based on the target focusing distance, separate the content in the scene area according to the dynamic and static attributes to obtain the dynamic area and static area of ​​the scene area.

[0044] It should be noted that in optical theory, the focal plane represents the plane with the sharpest focus. However, in practical applications, the degree of defocus blur of objects within the depth of field (DoF) is lower than the resolution limit of the imaging system, and the human eye can hardly perceive the change in sharpness. Therefore, the depth-priority dynamic focusing proposed in this embodiment is based on the principle of human visual perception, treating objects within the depth of field as sharp, and using this as a basis to verify the effectiveness of the focusing algorithm.

[0045] This embodiment is the core preprocessing step for achieving high-quality full-focus compositing of dynamic scenes. Specifically, it first uses scene depth information (a two-dimensional map representing the actual physical distance between objects and the camera in pixels) recovered from multiple source images (i.e., image sequences taken at different focal lengths in the same scene, also known as focus stacks) as input. Subsequently, it dynamically optimizes the multiple focus distances (i.e., the set of focus distances actually set by the camera lens when acquiring source images). The purpose is to intelligently calculate a set of target focus distances that are the fewest in number but have the most comprehensive coverage (i.e., the key focal length points that are most efficiently covered across the entire depth of field after optimization) based on the actual depth distribution of the current scene. This process is achieved by analyzing the depth-pixel density histogram and iteratively applying clustering algorithms (such as mean shift) to ensure that redundant shooting is minimized while meeting pixel coverage requirements. After determining the target focusing distance, the algorithm further separates the content in the scene region according to its dynamic and static attributes based on the depth of field range and information corresponding to the target focusing distance. Dynamic regions typically refer to objects in the scene whose position or shape changes over time or between frames (such as walking people, drifting clouds, and moving vehicles). These regions may exhibit significant motion blur or consistency differences between images with different focal lengths. Static regions, on the other hand, refer to stable background parts in the scene (such as buildings, roads, and stationary furniture). The separation process can be performed in the latent space using a dynamic / static separation module (such as MSOS). By calculating temporal residuals and combining them with a convolutional network to generate motion masks, the algorithm ultimately outputs accurate binary masks or feature identifiers for dynamic and static regions, laying the foundation for subsequent differentiated full-focus reconstruction.

[0046] The dynamic scene full-focus image synthesis method provided in this embodiment achieves accurate separation and targeted processing of dynamic and static regions by intelligently analyzing scene depth information and dynamically optimizing the focusing distance. It effectively overcomes the problem of reduced synthesis quality caused by motion blur and inaccurate focusing in dynamic scenes in traditional methods, and significantly improves the detail clarity, motion consistency and overall visual quality of the final full-focus image.

[0047] Based on the above embodiments, in this embodiment, step 210 dynamically optimizes the multiple focusing distances based on the scene depth information of the multiple source images to determine the target focusing distance, including: Step 211: Perform depth estimation on the multiple source images to generate scene depth information.

[0048] Step 212: Based on the scene depth information, construct a depth-pixel density histogram.

[0049] Step 213: Based on the depth-pixel density histogram, the multiple focusing distances are iterated through an optimization algorithm to determine the target focusing distance, so as to minimize the number of selected focusing distances while satisfying the preset pixel coverage.

[0050] Step 213 specifically includes: Based on the depth-pixel density histogram, the focusing distance corresponding to the depth value with the highest pixel density is determined as the initial focusing distance; The initial focusing distance is iteratively optimized using the mean-shift clustering algorithm and the adaptive Gaussian kernel function based on the depth-of-field model to obtain the optimized focusing distance; Based on the optimized focusing distance and the corresponding depth of field range, update the depth-pixel density histogram and remove pixels that have been covered by the depth of field range; Repeat the above determination, optimization and update steps until the number of remaining pixels in the updated depth-pixel density histogram is less than the preset threshold, and use all optimized focusing distances as the target focusing distance.

[0051] Specifically, multiple source images (i.e., image sequences with different sharpness distributions taken at different physical focal lengths) acquired in the same scene are processed using depth estimation techniques (such as monocular or stereo matching algorithms based on deep learning) to generate a scene depth information map. This map is a two-dimensional matrix, where each pixel value represents the actual physical distance between the point and the camera lens. Figure 2 This is a flowchart illustrating the dynamic focusing process in the dynamic scene full-focus image synthesis method provided by the present invention, as shown below. Figure 2The diagram illustrates the complete workflow of a dynamic focus distance optimization algorithm based on a depth-pixel density histogram. The process begins with a scene depth map generated by the depth estimation module, which is used to construct a depth-pixel density histogram. The vertical axis, Number of Pixels, represents the number of pixels, and the horizontal axis, Depth Value, represents the depth value, visually displaying the pixel distribution density at different depths in the scene. The algorithm first selects the depth value with the maximum current pixel density from the histogram as the initial focus depth. Then, an iterative optimization method is used. In each round, clustering algorithms such as mean shift are combined with a depth of field (DoF) model to calculate an optimal focus distance (Choose the focal distance for this round and fix the corresponding DoF), determining the depth of field coverage range corresponding to that focal length. Each time a focal length is determined, the histogram is updated by removing pixels from the already covered depth range. This process is repeated (from time 1 to time n) until the remaining pixel count falls below a preset threshold T, at which point the iteration terminates (the no branch ends, the yes branch continues). The final output is an optimized set of multi-focal distances: Multi-Focus Distances [d1, d2, ..., d...]. n This allows for maximum coverage of the scene's depth range with the fewest possible shots.

[0052] Furthermore, the process of depth prior dynamic focusing is explained in detail.

[0053] 1. Use a depth estimation model to obtain scene depth information and generate a discrete depth-pixel density histogram. The goal is to select a limited focal length plane. This maximizes depth domain coverage; the depth of field (depth domain) formula is: in, It's the camera aperture value. It is the permissible diameter of the circle of confusion. It's the lens focal length. This refers to the focusing distance. It determines the allowable pixel loss threshold. Determine the Gaussian kernel density function : in, It is the currently selected focal plane position. , It is the proportionality coefficient.

[0054] 2. A mean-shift clustering algorithm combined with an adaptive truncated Gaussian kernel density function is used to dynamically adjust the focal plane position and obtain the focusing distance. Specifically, for the depth pixel density histogram... Select the depth value with the highest pixel density. As the initial focus, an iterative formula is used to apply a weighted average of the pixels within the current focal plane using a Gaussian kernel density function to update the focal plane position: Iterate continuously until the condition is met. Or it reaches the maximum number of iterations, at which point... This refers to the focusing distance selected in this iteration.

[0055] 3. Use this focusing distance renew Depth pixel density histogram ,Right now and guarantee Add the selected focus distance to the result set. , examine Compared to whether it is initially less than the pixel loss threshold If the result is less than the given value, the algorithm terminates and the result set is obtained. Otherwise, return to step 2 to continue iterating and select the next focusing distance.

[0056] The dynamic scene full-focus image synthesis method provided in this embodiment transforms scene depth information into a pixel density histogram and uses mean-shift clustering and depth-of-field adaptive kernel function for iterative optimization. This achieves intelligent and adaptive focus distance selection, significantly reducing redundant shooting while ensuring image acquisition integrity. This not only improves image acquisition efficiency but also provides a precise and reliable data foundation for subsequent high-quality full-focus image synthesis.

[0057] Based on the above embodiments, in this embodiment, step 300 performs full-focus reconstruction on the dynamic region and the static region based on the multiple source images to generate a first full-focus image and a second full-focus image, including: Step 310: Based on the multiple source images, a first neural network model is used to perform full-focus reconstruction on the dynamic region to generate the first full-focus image, and a second neural network model is used to perform full-focus reconstruction on the static region to generate the second full-focus image.

[0058] Step 320: The first neural network model and the second neural network model have different network architectures.

[0059] Specifically, for dynamic regions, a first neural network model (e.g., an architecture based on conditional diffusion) is used for full-focus reconstruction to generate a first full-focus image. This model, by introducing a motion mask and cross-frame attention mechanisms (such as cross-attention), can effectively align temporal features, eliminate motion blur and artifacts, and ensure the spatiotemporal consistency and edge sharpness of dynamic objects in the synthesized image. For static regions, a second neural network model (e.g., an architecture based on multi-scale convolutional neural networks) is used for full-focus reconstruction to generate a second full-focus image. This model, through parallel multi-scale transposed convolutional layers and feature fusion modules, focuses on detail enhancement and structural fidelity to achieve high-definition reconstruction of the background region. It is worth noting that the first neural network model and the second neural network model have different network architectures, which is one of the core aspects of this invention: the dynamic processing model needs to emphasize temporal modeling and motion compensation capabilities (such as iterative denoising and attention mechanisms in the diffusion model), while the static processing model needs to emphasize local feature extraction and multi-scale fusion capabilities (such as convolutional kernels and upsampling structures in CNNs). This difference in architecture allows each model to be optimally adapted to the unique needs of its processing object, thereby significantly improving the overall quality and visual consistency of the final full-focus image.

[0060] Figure 3 This is a comparison diagram of the training and inference processes of the conditional diffusion model provided by this invention. Figure 3 The left side illustrates the training process. First, all images in the focal stack are encoded into the latent space using a pre-trained variational autoencoder (VAE). Then, the MSOS (Moving-Static Object Separator) module extracts the motion mask. and dynamic Latent Tokens During the diffusion process, noise (As diffusion input) and the latent vector of the target frame and motion mask (As an explicit spatial condition) they connect along the channel dimension, forming a joint input. Here, Provides structural guidance from the target frame, while Define dynamic regions in space, instructing the model to maintain temporal coherence within these regions. Simultaneously, dynamic Latent Tokens... A trainable motion self-attention layer interacts with the noisy latent vector, calculating query-key dot product similarity to align cross-frame motion features. This is enhanced by leveraging multi-frame priors through learnable Q-K-V projection. To preserve the knowledge of the pre-trained diffusion model, the cross-attention and self-attention layers in the base model are frozen during training; only the parameters of the motion self-attention layer are optimized end-to-end. The training objective uses a modified noise prediction loss. Figure 3 The right side illustrates the inference process, utilizing a Denoising Diffusion Implicit Model (DDIM) for efficient sampling. This process processes the input focus stack, extracting motion masks and markers from the latent space to provide conditions for the diffusion model. During the denoising iteration, the noisy latent vector is concatenated with the motion mask (channel-wise) and input into the U-Net backbone network, while cross-frame motion features are injected with temporal constraints through a trainable motion self-attention layer. By freezing the original self-attention and cross-attention mechanisms, motion-aware attention aggregates multi-focus features in dynamic regions while maintaining generation stability. The deterministic DDIM sampling strategy completes the mapping from latent vectors to pixels within 10 steps, synthesizing a clear AIF image (All-in-Focus Image).

[0061] The following is combined with Figure 3 Provide a complete explanation. Figure 3 The core data flow differences between the training and inference phases of the conditional diffusion model were compared. During training, the model receives the ground truth latent (GT Latent, i.e., the latent representation of the true image obtained through the encoder) and the random noise ε. T (Randomly sampled noise), the noise to be removed is predicted using denoising networks such as U-Net. T'The network parameters are optimized by minimizing the difference between the noise and the real noise. In the inference (generation) phase, the process is reversed: denoising is iteratively performed starting with Gaussian noise, conditioned on the Target Image Latent (the latent representation of the target frame, providing structural guidance) and the Motion Mask (identifying dynamic regions spatially). In each denoising step, the Noised Latent receives external conditional information through the CrossAttn (Cross-Attention) mechanism and processes internal feature relationships through the SelfAttn (Self-Attention) mechanism. Simultaneously, the Focal Stack Latent extracts the Motion Token in Latest Space as an additional temporal prior, which, along with the motion mask, is injected into the attention layer to align cross-frame motion features and ensure temporal consistency of the generated content. This gradual denoising process ultimately produces a clear and coherent output image.

[0062] Furthermore, the process of synthesizing the target full-focus image is described in detail.

[0063] 1. After obtaining the aligned focal stack, the image is compressed into the latent space using a Visual Image Array (VAE), and then dynamic-static separation is performed in the latent space using the Motion-Static Separation Module (MSOS). In the MSOS module, the latent features of the focal stack are first computed using Temporal Latent Residuals (TLR), and then passed through a pre-trained multi-layer convolutional thinning network. This network takes the TLR map and the original latent features as input, and achieves dynamic-static separation by jointly processing this information. The dynamic and static features are then processed separately.

[0064] The dynamic component utilizes a pre-designed conditional diffusion network. Since AIF image synthesis is a single-image process, the model uses the user-selected target image as the synthesis anchor, while a set of reference images injects prior constraints through an attention mechanism. Iterative denoising of the diffusion model allows for the progressive integration of multi-frame depth-of-focus information, enhancing synthesis flexibility by eliminating motion blur and out-of-focus artifacts, while ensuring consistency with real optical propagation through latent spatial alignment.

[0065] The static background is reconstructed globally with clarity through a pre-trained hierarchical convolutional feature fusion architecture and cross-focal plane integration. Specifically, it involves progressive upsampling using a three-stage multi-scale transposed convolutional architecture, with each stage containing three parallel transposed convolutions (kernel sizes of 4×4, 6×6, and 8×8) for feature expansion at specific scales, and feature merging at the channel level. The merged features are then processed by a fusion module, including spatial integration using 5×5 convolutions, group normalization to ensure stability, and the introduction of nonlinearity using the ReLU activation function. The network is optimized using composite loss functions, including SSIM loss to ensure structural consistency, perceptual loss to achieve semantic alignment through pre-trained feature space differences, and L1 Charbonnier loss to ensure gradient stability and pixel-level constraints. These components work together to ensure multi-level reconstruction supervision, achieving high-fidelity AIF synthesis.

[0066] After the model separately synthesizes the AIF images of dynamic objects and static backgrounds, these two parts are fused into a spatially coherent AIF image. The dynamic and static parts are then processed through a pre-trained lightweight fusion network consisting of three CNN layers with non-linear mapping and context alignment to adaptively address feature differences and eliminate edge mismatch issues through spatial domain coordination.

[0067] The dynamic scene full-focus image synthesis method provided in this embodiment separates the training and inference processes of the conditional diffusion model in a structured manner. In the training phase, it uses real latent representation and random noise to optimize noise prediction capability. In the inference phase, it guides the iterative denoising process through multimodal conditions such as latent representation of the target image, motion mask, and local stack latent features. It combines cross-attention and self-attention mechanisms to achieve spatial-temporal constraints, effectively solving the motion blur and spatiotemporal inconsistency problems in dynamic region synthesis, and significantly improving the detail fidelity, motion coherence, and visual realism of the generated full-focus image.

[0068] Figure 4 This is a system architecture diagram of the dynamic scene full-focus image synthesis method provided by the present invention. The following is in conjunction with... Figure 4 The present invention provides a complete description of the dynamic scene full-focus image synthesis method.

[0069] like Figure 4As shown, the complete workflow for achieving dynamic scene all-focus image synthesis is systematically demonstrated. The core workflow begins with a depth prior-based dynamic focusing module. This module performs depth estimation through image capture and then uses an adaptive mean shift clustering algorithm to iteratively optimize and select a set of focusing distances in the depth domain to achieve efficient scene coverage. Next, the module proceeds to the focal stacks alignment stage, aligning the captured focal stack images with user-driven scene selection and generating a target image by combining it with reference images. The workflow then transitions to the latent space, where an encoder maps the target image and reference images to target image latent representation and reference image latent representation, maintaining feature consistency across multiple temporal steps. The core processing is handled by the Diff-MotionAIFNet: its MISOS Module generates static latent representations and dynamic region features (Motion Tokens in Latent Space); the dynamic part models motion textures in the latent space through a Conditional Diffusion model, and reconstructs pixel content via a Variational Autoencoder Decoder; the entire network ensures spatial alignment through a Fusion Network, ultimately synthesizing a high-quality all-in-focus image. This architecture deeply integrates depth priors, motion-sensor separation, and conditional generation models, comprehensively solving the focusing, motion, and synthesis problems in dynamic scenes. Figure 5 This is a schematic diagram illustrating the effect of the dynamic scene full-focus image synthesis method provided by the present invention. The final effect of the full-focus image synthesized by the dynamic scene full-focus image synthesis method provided by the present invention can be seen in [reference needed]. Figure 5 .

[0070] The dynamic scene full-focus image synthesis method provided by this invention significantly improves scene depth coverage efficiency and optimizes focal length intervals through a dynamic focusing strategy, avoiding redundant capture in sparsely distributed depth regions and improving image acquisition efficiency. This provides a solid foundation for synthesizing high-quality full-focus images with fewer images, significantly enhancing the overall performance of full-focus image synthesis. Through a dual-path design, dynamic and static areas are separated, and a lightweight CNN is used for spatial feature aggregation to restore details in static areas, while a conditional diffusion model is employed for dynamic areas to ensure spatiotemporal alignment and compensation. Finally, a multi-scale fusion network seamlessly integrates the two parts. This method simplifies the model, improves synthesis quality, and supports parallel processing on heterogeneous mobile chips, effectively reducing overall latency and solving the motion artifacts and distortion problems in dynamic AIF synthesis.

[0071] The dynamic scene full-focus image synthesis apparatus provided by the present invention is described below. The dynamic scene full-focus image synthesis apparatus described below and the dynamic scene full-focus image synthesis method described above can be referred to in correspondence.

[0072] Figure 6 This is a schematic diagram of the dynamic scene full-focus image synthesis device provided by the present invention, as shown below. Figure 6 As shown, the dynamic scene full-focus image synthesis device provided by the present invention includes: The acquisition module 601 is used to acquire multiple source images at multiple focusing distances collected in the same scene area, with each source image corresponding to a different focusing distance; The decoupling module 602 is used to separate the scene depth information based on the multiple source images to obtain the dynamic region and the static region of the scene area; The reconstruction module 603 is used to perform full-focus reconstruction on the dynamic region and the static region based on the multiple source images respectively, and generate a first full-focus image and a second full-focus image. The fusion module 604 is used to fuse the first full-focus image and the second full-focus image to generate a target full-focus image.

[0073] The dynamic scene full-focus image synthesis device provided by this invention acquires multiple source images at multiple focusing distances within the same scene region, with each source image corresponding to a different focusing distance. Based on the scene depth information of the multiple source images, the device separates the scene region into dynamic and static regions. Full-focus reconstruction is then performed on the dynamic and static regions based on the multiple source images to generate a first full-focus image and a second full-focus image. Finally, the first and second full-focus images are fused to generate a target full-focus image. Therefore, this invention effectively overcomes the degradation in synthesis quality caused by motion blur and defocus artifacts in dynamic scenes using traditional methods by separately processing the dynamic and static regions and performing targeted full-focus reconstruction. This significantly improves the detail clarity and visual consistency of the final full-focus image, making it particularly suitable for complex scenes containing moving objects. It achieves superior image synthesis results while ensuring efficient processing.

[0074] Based on the above embodiments, in this embodiment, the decoupling module 602 is specifically used for: The multiple focusing distances are dynamically optimized based on the scene depth information of the multiple source images to determine the target focusing distance; Based on the target focusing distance, the content in the scene area is separated according to its dynamic and static attributes to obtain the dynamic and static areas of the scene area.

[0075] Based on the above embodiments, in this embodiment, the device further includes a dynamic optimization module, specifically used for: Depth estimation is performed on the multiple source images to generate scene depth information; Based on the scene depth information, a depth-pixel density histogram is constructed; Based on the depth-pixel density histogram, the multiple focusing distances are iterated through an optimization algorithm to determine the target focusing distance, so as to minimize the number of selected focusing distances while satisfying the preset pixel coverage.

[0076] Based on the above embodiments, in this embodiment, the device further includes a dynamic focusing module, specifically used for: Based on the depth-pixel density histogram, the focusing distance corresponding to the depth value with the highest pixel density is determined as the initial focusing distance; The initial focusing distance is iteratively optimized using the mean-shift clustering algorithm and the adaptive Gaussian kernel function based on the depth-of-field model to obtain the optimized focusing distance; Based on the optimized focusing distance and the corresponding depth of field range, update the depth-pixel density histogram and remove pixels that have been covered by the depth of field range; Repeat the above determination, optimization and update steps until the number of remaining pixels in the updated depth-pixel density histogram is less than the preset threshold, and use all optimized focusing distances as the target focusing distance.

[0077] Based on the above embodiments, in this embodiment, the reconstruction module 603 is specifically used for: Based on the multiple source images, a first neural network model is used to perform full-focus reconstruction on the dynamic region to generate the first full-focus image, and a second neural network model is used to perform full-focus reconstruction on the static region to generate the second full-focus image. The first neural network model and the second neural network model have different network architectures.

[0078] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device can be a robot or other electronic device, and may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. The processor 710, communication interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a dynamic scene full-focus image synthesis method, including: Acquire multiple source images at different focusing distances captured in the same scene area, with each source image corresponding to a different focusing distance; Based on the scene depth information of the multiple source images, the dynamic region and static region of the scene area are obtained; Based on the multiple source images, full-focus reconstruction is performed on the dynamic region and the static region respectively to generate a first full-focus image and a second full-focus image; The first full-focus image and the second full-focus image are fused to generate the target full-focus image.

[0079] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the dynamic scene full-focus image synthesis method provided by the above methods, including: Acquire multiple source images at different focusing distances captured in the same scene area, with each source image corresponding to a different focusing distance; Based on the scene depth information of the multiple source images, the dynamic region and static region of the scene area are obtained; Based on the multiple source images, full-focus reconstruction is performed on the dynamic region and the static region respectively to generate a first full-focus image and a second full-focus image; The first full-focus image and the second full-focus image are fused to generate the target full-focus image.

[0081] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic scene full-focus image synthesis method provided by the methods described above, including: Acquire multiple source images at different focusing distances captured in the same scene area, with each source image corresponding to a different focusing distance; Based on the scene depth information of the multiple source images, the dynamic region and static region of the scene area are obtained; Based on the multiple source images, full-focus reconstruction is performed on the dynamic region and the static region respectively to generate a first full-focus image and a second full-focus image; The first full-focus image and the second full-focus image are fused to generate the target full-focus image.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synthesizing dynamic scene full-focus images, characterized in that, include: Acquire multiple source images at different focusing distances captured in the same scene area, with each source image corresponding to a different focusing distance; Based on the scene depth information of the multiple source images, the dynamic region and static region of the scene area are obtained; Based on the multiple source images, full-focus reconstruction is performed on the dynamic region and the static region respectively to generate a first full-focus image and a second full-focus image; The first full-focus image and the second full-focus image are fused to generate the target full-focus image.

2. The dynamic scene full-focus image synthesis method according to claim 1, characterized in that, The scene depth information based on the multiple source images is used to separate the scene region into dynamic and static regions, including: The multiple focusing distances are dynamically optimized based on the scene depth information of the multiple source images to determine the target focusing distance; Based on the target focusing distance, the content in the scene area is separated according to its dynamic and static attributes to obtain the dynamic and static areas of the scene area.

3. The dynamic scene full-focus image synthesis method according to claim 2, characterized in that, The step of dynamically optimizing the multiple focusing distances based on the scene depth information of the multiple source images to determine the target focusing distance includes: Depth estimation is performed on the multiple source images to generate scene depth information; Based on the scene depth information, a depth-pixel density histogram is constructed; Based on the depth-pixel density histogram, the multiple focusing distances are iterated through an optimization algorithm to determine the target focusing distance, so as to minimize the number of selected focusing distances while satisfying the preset pixel coverage.

4. The dynamic scene full-focus image synthesis method according to claim 3, characterized in that, The process of determining the target focusing distance by iteratively optimizing the multiple focusing distances based on the depth-pixel density histogram using an optimization algorithm includes: Based on the depth-pixel density histogram, the focusing distance corresponding to the depth value with the highest pixel density is determined as the initial focusing distance; The initial focusing distance is iteratively optimized using the mean-shift clustering algorithm and the adaptive Gaussian kernel function based on the depth-of-field model to obtain the optimized focusing distance; Based on the optimized focusing distance and the corresponding depth of field range, update the depth-pixel density histogram and remove pixels that have been covered by the depth of field range; Repeat the above determination, optimization and update steps until the number of remaining pixels in the updated depth-pixel density histogram is less than the preset threshold, and use all optimized focusing distances as the target focusing distance.

5. The dynamic scene full-focus image synthesis method according to claim 1, characterized in that, The step of performing full-focus reconstruction on the dynamic region and the static region based on the multiple source images to generate a first full-focus image and a second full-focus image includes: Based on the multiple source images, a first neural network model is used to perform full-focus reconstruction on the dynamic region to generate the first full-focus image, and a second neural network model is used to perform full-focus reconstruction on the static region to generate the second full-focus image. The first neural network model and the second neural network model have different network architectures.

6. A dynamic scene full-focus image synthesis device, characterized in that, include: The acquisition module is used to acquire multiple source images at multiple focusing distances in the same scene area, with each source image corresponding to a different focusing distance; The decoupling module is used to separate the scene depth information based on the multiple source images to obtain the dynamic region and the static region of the scene area; The reconstruction module is used to perform full-focus reconstruction on the dynamic region and the static region based on the multiple source images respectively, and generate a first full-focus image and a second full-focus image; The fusion module is used to fuse the first full-focus image and the second full-focus image to generate a target full-focus image.

7. The dynamic scene full-focus image synthesis device according to claim 6, characterized in that, The decoupling module is specifically used for: The multiple focusing distances are dynamically optimized based on the scene depth information of the multiple source images to determine the target focusing distance; Based on the target focusing distance, the content in the scene area is separated according to its dynamic and static attributes to obtain the dynamic and static areas of the scene area.

8. The dynamic scene full-focus image synthesis device according to claim 7, characterized in that, The device also includes a dynamic optimization module, specifically used for: Depth estimation is performed on the multiple source images to generate scene depth information; Based on the scene depth information, a depth-pixel density histogram is constructed; Based on the depth-pixel density histogram, the multiple focusing distances are iterated through an optimization algorithm to determine the target focusing distance, so as to minimize the number of selected focusing distances while satisfying the preset pixel coverage.

9. The dynamic scene full-focus image synthesis device according to claim 8, characterized in that, The device also includes a dynamic focusing module, specifically used for: Based on the depth-pixel density histogram, the focusing distance corresponding to the depth value with the highest pixel density is determined as the initial focusing distance; The initial focusing distance is iteratively optimized using the mean-shift clustering algorithm and the adaptive Gaussian kernel function based on the depth-of-field model to obtain the optimized focusing distance; Based on the optimized focusing distance and the corresponding depth of field range, update the depth-pixel density histogram and remove pixels that have been covered by the depth of field range; Repeat the above determination, optimization and update steps until the number of remaining pixels in the updated depth-pixel density histogram is less than the preset threshold, and use all optimized focusing distances as the target focusing distance.

10. The dynamic scene full-focus image synthesis device according to claim 6, characterized in that, The reconstruction module is specifically used for: Based on the multiple source images, a first neural network model is used to perform full-focus reconstruction on the dynamic region to generate the first full-focus image, and a second neural network model is used to perform full-focus reconstruction on the static region to generate the second full-focus image. The first neural network model and the second neural network model have different network architectures.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the dynamic scene full-focus image synthesis method as described in any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dynamic scene full-focus image synthesis method as described in any one of claims 1 to 5.

Images