Image generation method and apparatus therefor

Abstract

This application discloses an image generation method and apparatus, belonging to the field of artificial intelligence technology. An embodiment of the method includes: acquiring an original image; inputting the original image into a bokeh parameter prediction model to obtain a bokeh parameter map; performing bokeh processing on the original image based on the bokeh parameter map to generate an initial bokeh image; and inputting the initial bokeh image into a bokeh image generation model to obtain a target bokeh image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an image generation method and apparatus. Background Technology

[0002] With the popularization of mobile imaging technology, users have increasingly higher demands for image quality, especially for professional-grade background blur effects that can highlight the subject and create a sense of atmosphere. This effect typically requires a large aperture lens and a large image sensor to achieve naturally through optical means. However, due to the size and cost limitations of mobile devices, the physical characteristics of their camera modules dictate a relatively deep native depth of field, making it difficult to directly capture shallow depth-of-field blurred images. Therefore, simulating optical blur through computational photography algorithms has become one of the key technological challenges in the field of mobile imaging.

[0003] In existing technologies, achieving image bokeh mainly relies on multi-camera hardware systems or pure software semantic segmentation algorithms. The former requires additional cameras, which increases hardware costs, while the latter suffers from problems such as harsh edges, false shapes of bokeh spots, and a lack of realistic physical depth in the imaging effect. Summary of the Invention

[0004] The purpose of this application is to provide an image generation method and apparatus that can improve the realism of image blurring effects without increasing hardware costs.

[0005] In a first aspect, embodiments of this application provide an image generation method, the method comprising: acquiring an original image; inputting the original image into a bokeh parameter prediction model to obtain a bokeh parameter map; performing bokeh processing on the original image based on the bokeh parameter map to generate an initial bokeh image; and inputting the initial bokeh image into a bokeh image generation model to obtain a target bokeh image.

[0006] Secondly, embodiments of this application provide an image generation apparatus, which includes: an acquisition unit for acquiring an original image; a first input unit for inputting the original image into a bokeh parameter prediction model to obtain a bokeh parameter map; a bokeh processing unit for performing bokeh processing on the original image based on the bokeh parameter map to generate an initial bokeh image; and a second input unit for inputting the initial bokeh image into a bokeh image generation model to obtain a target bokeh image.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first or third aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the method as described in the first or third aspect above.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the first or third aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first or third aspect.

[0011] In this embodiment, the original image is first acquired; then, the original image is input into a bokeh parameter prediction model to obtain a bokeh parameter map; subsequently, the original image is blurred based on the bokeh parameter map to generate an initial blurred image; finally, the initial blurred image is input into a bokeh image generation model to obtain the target blurred image. In this process, the scene understanding of the original image through the parameter prediction model to generate bokeh parameters enhances the display control of the blurring process, allowing the bokeh parameter map to serve as an intermediate control signal to precisely guide the blurring process and ensure the physical plausibility of the generated initial blurred image. Image optimization processing through the bokeh image generation model to generate the target blurred image further achieves a more accurate and realistic blurring effect based on the initial blurred image. This process does not rely on hardware assistance or simple semantic segmentation, but is achieved through the collaborative processing of two models, which improves the realism of the image blurring effect without increasing hardware costs. Attached Figure Description

[0012] Figure 1 This is a flowchart of the image generation method provided in the embodiments of this application; Figure 2 This is a flowchart of the initial blurred image generation process in the image generation method provided in the embodiments of this application; Figure 3 This is a flowchart of the model training process in the image generation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the processing procedure of the second model in the image generation method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the processing procedure of the first model in the image generation method provided in the embodiments of this application; Figure 6 This is a flowchart of the training process of the first model and the second model in the image generation method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the image generation apparatus provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0015] The methods and apparatus provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0016] Please refer to Figure 1 The diagram illustrates a flowchart of an image generation method provided in an embodiment of this application. This image generation method can be applied to electronic devices. For example, the electronic device may be a smartphone, tablet computer, laptop computer, wearable device, etc.

[0017] The image generation method provided in this application includes the following steps: Step 101: Obtain the original image.

[0018] In this embodiment, the original image refers to an image directly captured or acquired by the user through an electronic device with a camera function, without any blurring or blurring processing. The original image can be an image from a real-time preview or capture, or an image read from a photo album. It can be a single image or multiple images. There is no limitation here.

[0019] Step 102: Input the original image into the bokeh parameter prediction model to obtain the bokeh parameter map.

[0020] In this embodiment, the electronic device can store a pre-trained bokeh parameter prediction model. This model can be pre-trained using machine learning methods to train a Convolutional Neural Network (CNN). Its function is to receive an original image as input, predict and output the bokeh parameter map corresponding to that image through the mapping relationships learned internally. Its role is to quickly and automatically parse the bokeh parameters required for high-quality bokeh from any image during the inference phase, replacing complex physical modeling or manual analysis.

[0021] A blur parameter map can consist of one or more feature maps. Each feature map can contain blur parameters for each pixel in the original image. Different feature maps can encode different types of blur parameters. Blur parameters are the parameters used during the blurring process of an image.

[0022] Step 103: Blur the original image based on the blur parameter map to generate an initial blurred image.

[0023] In this embodiment, blurring refers to the process of performing a blurring operation on the original image based on a blurring parameter map to simulate an optical bokeh effect. During the blurring process, the blurring method and intensity can be dynamically adjusted according to the blurring parameters corresponding to different pixels in the original image, thereby generating an initial blurring map. This process simulates an optical blurring effect, but its blurring degree and method are dynamically controlled by the blurring parameter map predicted by the blurring parameter prediction model, rather than a fixed Gaussian blur.

[0024] In practice, the point spread function (PSF) kernel applicable to each pixel in the original image can be determined first based on the bokeh parameter map. The PSF describes the brightness distribution image formed on an imaging sensor after an ideal point light source passes through an optical system. The PSF kernel is an instance of the PSF. A PSF kernel of a specific shape can be obtained by defining a kernel with a set of kernel parameters. After determining the PSF kernel applicable to each pixel in the original image, a differentiable convolution operation can be performed on the original image based on the determined PSF kernel to generate a new image whose bokeh effect is controlled by the bokeh parameter map. This new image is the initial bokeh image corresponding to the original image.

[0025] It should be noted that the initial blurred image can be a single image or multiple images; there is no limitation here. For example, multiple initial blurred images can be generated by using different image blurring processing methods or different point spread function kernels to determine the results. This allows the blurred image generation model to combine the advantages of multiple initial blurred images and obtain the target blurred image after intelligent fusion and image quality enhancement.

[0026] Step 104: Input the initial blurred image into the blurred image generation model to obtain the target blurred image.

[0027] In this embodiment, the blurred image generation model can be pre-trained on another convolutional neural network using machine learning methods. It can receive one or more initial blurred images and, through its internally learned image generation and intelligent fusion strategies, output a target blurred image that achieves a high level of edge sharpness, background blur, and realistic bokeh.

[0028] The target blurred image is the final blurred result output by the blurred image generation model, which is obtained by intelligent fusion and image quality enhancement after the blurred image generation model. The image should have the visual characteristics of a clear and sharp subject, a natural background blur, and realistic bokeh, comparable to the shallow depth of field effect of a professional SLR camera with a large aperture.

[0029] In some scenarios, the original image is a single image. In such cases, inference can be performed online. Specifically, the single original image is first input into the bokeh parameter prediction model to predict the bokeh parameter map. Then, multiple initial bokeh maps are generated based on the bokeh parameter map. Subsequently, a lightweight bokeh image generation model is used to fuse the multiple initial bokeh maps, outputting the target bokeh image. For mobile deployment requirements, INT8 (8-bit integer) quantization and a structured pruning compression model can be used. Convolution operations are optimized using the Winograd (Winograd minimum filtering algorithm) or GEMM (General Matrix Multiply) algorithm, and downsampling computation paths are enabled for large-radius bokeh regions to improve inference speed. Among them, the Winograd algorithm is an algorithm that efficiently computes small convolutions by reducing the number of multiplications. For large-radius bokeh regions, a "downsampling-convolution-upsampling" strategy is adopted. In the upsampling stage, edge-preserving reconstruction technology guided by the edge confidence map E is used to protect image details.

[0030] In some scenarios, the original image consists of multiple images. In such cases, offline inference can be performed. Specifically, the multiple original images are first preprocessed, including alignment and filtering. These preprocessed images are then input into a bokeh parameter prediction model to obtain multiple bokeh parameter maps. Next, a bokeh image generation model fuses these initial bokeh maps, outputting the target bokeh image. During the fusion process, the main frame and auxiliary frames can be selected based on inter-frame sharpness and motion score to reduce the impact of low-quality frames on the fusion effect.

[0031] In this embodiment, the original image is first acquired; then, the original image is input into the bokeh parameter prediction model to obtain a bokeh parameter map; subsequently, the original image is blurred based on the bokeh parameter map to generate an initial blurred image; finally, the initial blurred image is input into the bokeh image generation model to obtain the target blurred image. In the above process, the scene understanding of the original image through the parameter prediction model to generate bokeh parameters enhances the display control of the blurring process, allowing the bokeh parameter map to serve as an intermediate control signal to precisely guide the blurring process and ensure the physical plausibility of the generated initial blurred image; image optimization processing through the bokeh image generation model to generate the target blurred image further achieves a more accurate and realistic blurring effect based on the initial blurred image. This process does not rely on hardware assistance or simple semantic segmentation, but is achieved through the collaborative processing of two models, which can improve the realism of the image blurring effect without increasing hardware costs.

[0032] In some optional embodiments, the bokeh parameter maps may include, but are not limited to, a blur radius map, an edge confidence map, and a specular highlight saliency map. The blur radius map can be denoted as R_n, the edge confidence map as E, and the specular highlight saliency map as S. As an example, the original image includes two images taken with different aperture values, denoted as j1 and j2. After inputting j1 into the bokeh parameter prediction model, the model can output a set of bokeh parameter maps, including the blur radius map R_1, the edge confidence map E_1, and the specular highlight saliency map S_1. Similarly, after inputting j2 into the bokeh parameter prediction model, the model can output the blur radius map R_2, the edge confidence map E_2, and the specular highlight saliency map S_2.

[0033] The blur radius map includes the blur radius corresponding to each pixel in the original image. It can be a single-channel non-negative image, with larger values ​​indicating that the pixel should be blurred more.

[0034] An edge confidence map indicates the confidence level of each pixel in the original image to be located at the edge of the image field. It can be a single-channel image with pixel values ​​in the range [0,1], where a value closer to 1 indicates a higher probability that the pixel is located at the edge of the image field. High-value areas represent sharp, well-defined edges, such as the boundary between a person's outline and the background; low-value areas may correspond to edges with complex textures, semi-transparency, or motion blur. The image field edge refers to the area on the imaging plane (sensor) far from the optical center. Due to the physical characteristics of optical lenses, when light is incident at an angle to the image field edge, it causes two main effects: vignetting (attenuation of edge light intensity) and aberrations, such as coma and astigmatism. This causes the originally circular blur spot to become elliptical or even more complex shapes, sometimes accompanied by rotation. The image field edge is usually determined by whether the radial distance r between the pixel and the image center exceeds a preset threshold R_edge.

[0035] The highlight saliency map is used to identify out-of-focus highlight areas in the original image. It can also be a single-channel image within the range [0,1], with values ​​closer to 1 indicating a higher probability that the pixel belongs to an out-of-focus highlight area. An out-of-focus highlight area refers to a bright point light source region in an image that appears as a diffuse circular spot due to being outside the depth of field. Examples include streetlights, string lights during festivals, or the reflected light from water droplets in backlight. These areas appear as points in a sharp image, but diffuse into spots with specific shapes in a blurred image, i.e., bokeh. Its core characteristics are high brightness and shape changes due to defocus. Here, if a pixel's value in the highlight saliency map is higher than the threshold T_spec, it is determined that the point is located in an out-of-focus highlight area, i.e., it is an out-of-focus highlight point; if a pixel's radial distance r_dist from the image center is greater than another threshold, it is determined that the point is located at the edge of the image field. The point spread function kernel type can be decided based on pre-defined rules, with the core being the selection of the most suitable blur shape based on scene semantics and geometric location.

[0036] Based on this, step 102 may include the following steps: Step S11: Determine the blur radius of each pixel in the original image based on the blur radius map.

[0037] Step S12: Based on the specular saliency map and the image field position of each pixel in the original image, determine the point spread function kernel type corresponding to each pixel in the original image.

[0038] Point spread function kernel types may include, but are not limited to, at least one of the following: blade-shaped point spread function kernel, elliptical disk-shaped point spread function kernel, Gaussian point spread function kernel, etc.

[0039] The leaf-shaped point spread function kernel is a convolution kernel modeled based on the geometry of real camera aperture blades. A real lens aperture consists of multiple blades, such as 7 or 9. When the aperture closes, the polygonal openings formed by these blades are directly projected onto the out-of-focus highlights, creating angular highlights such as heptagons or nonagons, and potentially producing onion-ring-like textures. The leaf-shaped point spread function kernel simulates this physical phenomenon; its number of sides, rotation angle, and brightness attenuation curve are all determined by calibrated PSF parameters. Its application is specifically designed for out-of-focus highlight areas, replacing simple blurry spots with beautiful highlights characteristic of the lens, greatly enhancing visual realism and artistry.

[0040] The elliptical disk-shaped dot spread function kernel is a convolution kernel used to simulate optical aberrations at the image field edges. Essentially, it is an elliptical, uniform disk kernel whose eccentricity *e* and rotation angle *θ* dynamically change with the radial distance and azimuth angle of the pixel from the image field center. The eccentricity simulates the stretching and deformation of the light spot caused by astigmatism or coma, while the rotation angle simulates the focusing effect. Its core application is in the non-highlight areas at the image field edges. By introducing these subtle imperfections into the background blur, the overall blur effect avoids the overly digitalized uniform blur of central symmetry, and more closely resembles the imaging style of a real lens.

[0041] The Gaussian point spread function kernel is a convolution kernel based on the Gaussian function. It produces a blur effect with the highest weight at the center, gradually decreasing towards the edges. Its characteristics include isotropy, computational efficiency, and smooth transition. The Gaussian point spread function kernel, as the default or fallback option, is primarily used in two scenarios: first, in flat or low-frequency textured background areas in the image that are neither out-of-focus highlights nor image field edges. In these cases, complex light spot shapes are unnecessary; smooth blurring achieves the best effect and saves computational power. Second, when a region with low signal-to-noise ratio is detected, a Gaussian kernel is forced to avoid complex kernels amplifying noise and texture, thus reducing noise and stabilizing the output.

[0042] Here, if the value of a pixel in the specular saliency map is higher than the threshold T_spec, then the pixel is determined to be located in the out-of-focus specular region, i.e., the pixel is an out-of-focus specular point. If the radial distance r_dist of a pixel from the image center is greater than another threshold, then the pixel is determined to be located at the edge of the image field. The point spread function kernel type can be decided based on pre-defined rules, the core of which is to select the most suitable blur shape based on scene semantics and geometric location.

[0043] Optionally, for each pixel, if it is located within the out-of-focus highlight area, the corresponding leaf-shaped dot spread function kernel can be determined. Since the bokeh shape in the real world is determined by the number of aperture blades, by using the leaf-shaped dot spread function kernel, simple brightness diffusion is transformed into bokeh with specific polygonal characteristics. This allows for physically accurate simulation of highlights in the image, ensuring that the generated bokeh effect is highly consistent with the result of optical lens shooting in key visual aesthetic details, completely eliminating the digital and artificial feel produced by traditional Gaussian blur.

[0044] Optionally, for each pixel, if it is located outside the out-of-focus highlight area and at the edge of the image field, the corresponding elliptical disk-shaped dot spread function kernel can be determined. Traditional algorithms typically ignore the effect of image field edge aberration, resulting in uniform blurring across the entire image with low realism. By adaptively enabling the elliptical disk-shaped dot spread function kernel at the image field edge, the distortion and rotation effects of light spots caused by aperture erosion and astigmatism are introduced into the background blurring, increasing the image's depth and realism.

[0045] Optionally, for each pixel, in cases other than the two mentioned above, the corresponding Gaussian point spread function kernel can be determined. The Gaussian kernel possesses excellent smoothness and computational separability mathematically, resulting in significantly higher execution efficiency than complex leaf-shaped or elliptical disk-shaped point spread function kernels. Applying the Gaussian kernel to the mid-to-low frequency background regions, which occupy a large portion of the image and are insensitive to specific light spot shapes, can significantly reduce the overall computational complexity with almost no loss of final visual effect. Simultaneously, this avoids unnecessary texture interference or noise amplification problems that may arise from using complex kernels in flat or noisy regions, ensuring the algorithm's output stability and robustness across various scenarios.

[0046] Step S13: Generate the target point spread function kernel based on the fuzzy radius and the point spread function kernel type.

[0047] For each pixel, the corresponding target point spread function kernel is the kernel function ultimately determined for that pixel and used in the convolution calculation. It can be constructed based on two key parameters: the blur radius of the pixel and the kernel type. For example, for a highlight in the background, the target kernel can be a heptagonal leaf-shaped point spread function kernel with a radius of 7 pixels; while for a flat grassy area, the target kernel can be a Gaussian point spread function kernel with a radius of 5 pixels.

[0048] Alternatively, the following steps can be followed: First, based on the fuzzy radius, at least two candidate kernel radii are determined from the discrete kernel radius set. The discrete kernel radius set R_set covers multiple fuzziness intensity levels from zero to the maximum allowable radius, and employs a non-uniform spacing to balance accuracy with storage and computational overhead. Non-uniform spacing can mean denser spacing for smaller radii and sparser spacing for larger radii. For example, the discrete kernel radius set R_set could be {0,1,2,3,5,7,9,12,16}. The purpose of the discrete kernel radius set is to provide a discretized, achievable anchor point for the continuously varying predicted fuzzy radius. All complex, parameterized point spread function kernels are pre-calculated or predefined with parameters for each radius value in this set, forming a kernel library. For example, for a predicted radius r=7.3, the candidate kernel radii are 7 and 9. Its function is to provide a reference point for subsequent interpolation operations. Choosing at least two candidate kernel radii instead of a single closest value is to support smooth interpolation, thereby generating kernels that precisely match the continuous predicted radii and avoiding blur intensity steps or jitter caused by directly rounding to a single discrete radius.

[0049] Then, based on the point spread function kernel type and at least two candidate kernel radii, at least two candidate point spread function kernels are generated. Here, based on each candidate kernel radius and point spread function kernel type, a point spread function kernel can be generated according to a preset parameterized formula. For example, for a configuration with a candidate kernel radius of 9 and a kernel type of "7-blade kernel", the generated candidate kernel is a heptagonal convolution kernel with specific rotation angles, brightness attenuation curves, and other parameters.

[0050] Finally, based on the proportional relationship between the blur radius and the radii of at least two candidate point spread function kernels, the parameters of at least two candidate point spread function kernels are interpolated to obtain the target point spread function kernel. Interpolation is a mathematical method that weights and fuses the parameters of the at least two candidate point spread function kernels to generate the target point spread function kernel. In practice, linear interpolation can be used. It is understandable that if interpolation is not performed and the continuous radius is directly rounded to the nearest discrete radius (e.g., treating 7.3 as 7), adjacent pixels may suddenly switch to different point spread function kernels due to small differences in radius prediction, potentially producing obvious and unnatural radius jump bands or ring artifacts in the background. Interpolation improves the continuity of the blur radius and the smoothness of kernel generation, enabling the generation of point spread function kernels corresponding to any continuous radius value, rather than being limited to predefined discrete radii. This results in incredibly smooth changes in blur intensity, avoiding visual jumps or band artifacts.

[0051] As an example, a set of discrete kernel radii, R_set, such as {0,1,2,3,5,7,9,12,16}, and a parameterized generation formula corresponding to each point spread function kernel type can be predefined. For each pixel, firstly, based on its blur radius r, the two nearest candidate radii are found in R_set; for example, if r = 7.3 pixels, then 7 and 9 are found. Then, based on its point spread function kernel type, two point spread function kernels with radii of 7 and 9 are generated respectively. Finally, based on the distance ratio between r and 7 and 9, all parameters of these two point spread function kernels, such as the standard deviation of the Gaussian kernel, the eccentricity of the elliptic kernel, and the rotation angle of the blade kernel, are linearly interpolated and fused to generate a continuously differentiable target point spread function kernel K_target that exactly matches the target radius r.

[0052] By determining at least two candidate kernel radii from a set of discrete kernel radii based on the blur radius, complex kernel functions only need to be pre-computed or stored for a limited number of discrete radius values, significantly reducing storage overhead and computational complexity of kernel generation. Simultaneously, by selecting two adjacent candidate radii, a smooth transition is prepared for subsequent processing, structurally ensuring the efficiency and feasibility of the kernel generation process and avoiding the enormous overhead of generating independent kernels for every possible consecutive radius value. By interpolating the parameters of the candidate kernels based on the proportional relationship between the blur radius and the candidate kernel radii to obtain the target kernel, the size and shape of the generated target point spread function kernel change continuously and smoothly with the prediction radius r. This results in a completely smooth spatial gradient of blur in the image, completely eliminating visual artifacts caused by quantization jumps and achieving an infinitely progressive depth-of-field transition effect indistinguishable from real optical imaging, greatly enhancing the overall naturalness and detail of the blurred image.

[0053] Step S14: Based on the edge confidence map, the original image is convolved using the target point spread function kernel to obtain the initial blurred image.

[0054] Specifically, the generated target point spread function kernel K_target can be used to convolve the original image. To preserve edges, an edge confidence map E can be introduced as a modulation factor. During convolution, for each pixel, the convolution result is dynamically blended with the original image based on its edge confidence value E. A higher E value indicates a more important edge, resulting in more original image pixels being retained during blending and a weaker blurring effect; conversely, a lower E value means the convolution result is used entirely. Furthermore, when generating the target point spread function kernel, the E value can be used to constrain the kernel's diffusion intensity in the edge normal direction. The convolution operation itself is performed in the frequency domain or accelerated through optimization methods such as im2col (image columnarization operation) and GEMM.

[0055] As an example, j2 is a portrait image, which can be blurred. At the edges of the hair in the portrait, the E value is close to 1. During convolution, the weight of the convolution result at this point can be reduced and blended with the sharp original pixels, resulting in a semi-blurred, natural transition effect at the hair, rather than a harsh, completely blurred paper-cut edge. Finally, after performing this edge-aware, pixel-level adaptive convolution on all pixels of the entire image, the output is the initial blurred image v2.

[0056] By determining the blur radius of a pixel based on a blur radius map, continuous and differentiable blur control can be provided for each pixel, rather than foreground / background binarization or layered quantization. This allows the generated blur effect to simulate the smooth and gradual depth-of-field transition (roll-off) from the focal plane to the defocus plane in real optical imaging, effectively avoiding a harsh sense of layering.

[0057] By determining the point spread function kernel type based on the highlight saliency map and image field position, adaptive simulation of bokeh morphology for different image regions is achieved. This makes the bokeh effect no longer a monotonous Gaussian blur, but possesses complex aberration characteristics such as optical aperture erosion and bilinearity of real lenses, greatly improving the realism of out-of-focus imaging.

[0058] By performing convolution processing based on edge confidence maps, an edge protection mechanism is incorporated while performing blurring, ensuring the sharpness and clarity of the main outline. This effectively solves the halo artifacts and edge blurring problems that are prone to occur in traditional algorithms when the depth map is inaccurate.

[0059] In summary, by performing blurring processing based on the blur radius map, edge confidence map, and highlight saliency map, the generated initial blurred image can be qualitatively improved in three aspects: the continuity of depth-of-field transition, the realism of out-of-focus bokeh, and the sharpness of the subject edges, laying a solid foundation for subsequent high-quality fusion.

[0060] In practical applications, see Figure 2This system can construct differentiable virtualizers supporting multiple types of point spread function (PSF) kernels, including Gaussian PSF kernels, disk / blade-shaped PSF kernels (Disk), and elliptical disk-shaped PSF kernels. The discrete kernel radius set R_set={0, 1, 2, 3, 5, 7, 9, 12, 16}, with kernel parameters dynamically adjusted according to the image field position. By calculating the difference between the blurred radius map R_n and the predefined radii in the kernel library, the weights of each kernel are obtained through the Softmax function. The convolution results of different kernels are then weighted and fused. During this process, the smoothness of the weight distribution can be controlled by a temperature parameter. The system can linearly interpolate kernel parameters such as standard deviation, ellipticity, blade angle, and brightness attenuation to generate the required target PSF kernel. Furthermore, various optimization strategies can be employed to improve the efficiency of the virtualization computation. For example, Fast Fourier Transform (FFT) kernel spectra can be cached for each radius in the discrete kernel radius set R_set, allowing linear combination and convolution operations of the kernels to be performed in the frequency domain to reduce time-domain overhead. Another example is the Gaussian point spread function kernel, which uses separable convolution to optimize computational speed. Complex point spread functions are accelerated using block convolution or the im2col+GEMM and Winograd algorithms. Furthermore, the blurred radius map R_n is optimized. For instance, bilateral smoothing can be performed using the image j_small as the edge guide map, applying a maximum radius constraint r_edge_max in high-gradient regions, while cropping R_n values ​​to the [0, R_max] interval, where R_max is the preset maximum radius, to avoid ring artifacts caused by radius jumps.

[0061] In some optional embodiments, various edge optimization and memory management strategies can be employed during the blurring process to generate the initial blurred image, and during subsequent fusion, to ensure efficient and stable operation on resource-constrained mobile devices. Specifically, these strategies may include, but are not limited to, at least one of the following: Block processing and adaptive computation path. For high-resolution input images, they can be divided into multiple overlapping or non-overlapping image blocks for processing. For each image block, the mean and variance of its corresponding blur radius map R_n block are calculated. The computation path is dynamically selected based on the block-level statistical results. If the average radius within the block is small and the variance is low, it indicates a flat or weakly textured area, so a complete and high-precision blur calculation is performed on that block; if the average radius is large and the variance is low, it indicates a large area of ​​uniformly blurred background, so a low-resolution computation path is used, that is, the image block is first downsampled, fast convolution is performed at low resolution, and then upsampled back to the original size, which significantly improves the processing speed of large blurred areas.

[0062] Block edge blending. Block processing can lead to seams between blocks due to computational differences. To avoid this problem, a blending technique (Feathering) can be used at block boundaries. Specifically, each block is assigned a weight map that gradually changes from the center to the edge, such as using a Gaussian window. When stitching the final full image, the pixel values ​​of the overlapping areas are a weighted average of the results from adjacent blocks, thus achieving a smooth transition and eliminating visible seams.

[0063] Caching and memory optimization. To improve computational efficiency, frequently used resources can be cached. For example, for each radius in a predefined set of discrete kernel radii R_set, the spectrum of its point spread function kernel in the frequency domain can be pre-calculated and cached. This spectrum can then be directly called during frequency domain convolution, avoiding redundant computation. Simultaneously, dynamic memory management is implemented for intermediate feature maps during network inference, promptly releasing features no longer needed and reusing memory buffers to reduce peak GPU / RAM usage and meet the memory constraints of mobile devices.

[0064] In some optional embodiments, a fault protection and rollback mechanism can be integrated before or during the final generation of the target blurred image to cope with complex or extreme scenarios, ensure the stability of the output results, and avoid generating images with severe visual artifacts. Specifically, this includes, but is not limited to, at least one of the following: Protection based on prediction confidence. The system can monitor the local statistical characteristics, such as volatility and uniformity, of the edge confidence map E and blur radius map R_n output by the blur parameter prediction model in real time. When it detects that the edge confidence E is generally low in a certain region and the variance of R_n in that region is large, it indicates that the model's prediction confidence in the geometry and blur degree of that region is very low, which can trigger a fault protection mode. In this region, it reverts to using a simple Gaussian kernel for blurring and forces the upper limit of the blur radius to a low value to avoid generating uncontrollable complex spots or excessive blurring.

[0065] Input quality detection-based backoff. At the input stage, the original input image undergoes quality analysis to detect severe noise or significant exposure mismatches, such as extreme motion blur or high ISO noise. When such issues severely impacting algorithm prerequisites, such as image alignment and signal-to-noise ratio, are detected, a global backoff mechanism is triggered. Depending on the severity, either an unblurred, small-aperture original image (j_small) can be directly output, or only a global, low-intensity Gaussian blur can be applied. This avoids forcing the full algorithm onto poor-quality input, which could produce unpleasant artifacts and ensure a minimum acceptable user experience.

[0066] In some alternative embodiments, see Figure 3 The blur parameter prediction model and the blur image generation model are trained through the following steps: Step 301: Obtain a sample set. Each sample in the sample set includes multiple sample images and a reference blurred image. The multiple sample images are taken by the same electronic device in the same scene using different aperture values.

[0067] In this embodiment, the sample set is a dataset used to train a machine learning model, which may include multiple samples, each of which includes multiple sample images and a reference blurred image.

[0068] Multiple sample images are a set of images taken by the same imaging device in the same scene and with the same composition, using multiple different aperture values, and can be denoted as j_n. Aperture values ​​are usually expressed in the form of "f / coefficient," such as f / 1.4, f / 2.8, and f / 16. The aperture value is inversely proportional to the physical aperture size; that is, the smaller the aperture value, the larger the aperture size; and vice versa. Sample images record the most direct optical differences caused by aperture changes: the larger the aperture value, the larger the aperture size, the shallower the depth of field, and the more blurred the background; the smaller the aperture value, the deeper the depth of field, and the sharper the background. Sample images provide raw visual data containing real physical information such as aperture differences, allowing models to learn the relationship between depth of field, image content, and aperture settings.

[0069] A reference bokeh image, serving as the ground truth (GT), is a high-quality bokeh image corresponding to multiple sample images and can be used as the standard answer for supervised model training. The reference bokeh image represents the ideal, high-quality bokeh effect for the corresponding scene. For static scenes, it is typically obtained by capturing images with a professional camera featuring a larger sensor and a larger aperture; for dynamic or difficult-to-reproduce scenes, it can be rendered and synthesized by combining a high-precision depth map and precisely calibrated lens optical parameters. High-precision depth maps can be obtained using LiDAR or structured light scanners. The reference bokeh image provides a clear optimization direction for the model during training, ensuring that the bokeh effect generated by the model approaches professional-grade results in terms of visual realism and plausibility.

[0070] In this embodiment, a smartphone or other imaging device with a variable aperture can be used to take multiple shots of the same static or controllable dynamic scene. Each shot only changes the aperture value, maintaining consistent composition and focus. For example, four images can be taken sequentially using aperture values ​​f / 1.6, f / 2.0, f / 2.8, and f / 4.0, denoted as j1, j2, j3, and j4. During shooting, the aperture switching delay needs to be controlled within the range of 50-150ms. Simultaneously, for this scene, a high-end full-frame SLR camera can be used to capture a photo with a naturally blurred background as a reference image. This set of images is then correlated to form a training sample. Repeating this process several times, covering various scenes such as portraits, still life, and landscapes, allows for the construction of a large-scale sample set.

[0071] It should be noted that complete metadata must be recorded according to a specific protocol during the acquisition process. This may include, but is not limited to, aperture value, shutter speed, ISO, focal length, lens temperature, AF focusing distance estimation, drive timestamp, and sensor timestamp for each shot. Exposure time and ISO parameters can be fixed during image acquisition. If complex scene lighting conditions prevent a single parameter from meeting brightness requirements, exposure fusion technology can be used to unify image brightness. A global shutter type should be preferred for the image sensor. If a rolling shutter is used, the inter-line timing differences must be considered in the subsequent alignment algorithm, or restrictions should be imposed during the acquisition phase through shooting strategies to reduce negative impacts.

[0072] The differences between the sample images are purely caused by aperture variations, encompassing the true patterns of depth of field and bokeh morphology as aperture changes. By acquiring the aforementioned sample set, the model can ground its learning objectives in optical physics, rather than semantic guessing or geometric assumptions. This allows it to understand the optical differences caused by different apertures and optimize using high-quality bokeh effects as a supervisory signal.

[0073] Step 302: Input multiple sample images into the first model to obtain multiple bokeh parameter maps.

[0074] In this embodiment, the first model is a convolutional neural network, such as U-Net or a variant thereof. The first model can be denoted as CNN1. The first model accepts any image as input, analyzes the image content to determine the degree of blurring and characteristics of each pixel, such as whether it is an edge or a highlight, and thus predicts and outputs a blur parameter map corresponding to that image. Here, each sample image from multiple sample images can be input into the first model to obtain the blur parameter map corresponding to that sample image.

[0075] Step 303: Based on multiple blurring parameter maps, multiple sample images are blurred to obtain multiple initial blurred images.

[0076] In this embodiment, blurring refers to the process of performing a blurring operation on sample images according to a blurring parameter map to simulate an optical bokeh effect. This process can be differentiable to support gradient calculation, thereby allowing errors to propagate back from the final result to the parameters of the first model. During blurring, for each sample image, the blurring method and intensity can be dynamically adjusted according to the blurring parameters corresponding to different pixels, thereby generating a corresponding initial blurring result map. Since different sample images correspond to different blurring parameter maps, the blurring effects of multiple initial blurring images are different. This process simulates an optical blurring effect, but its blurring degree and method are dynamically controlled by the blurring parameter map predicted by the first model, rather than a fixed Gaussian blur.

[0077] In practice, for each sample image, the point spread function kernel applicable to each pixel in the sample image can first be determined based on the bokeh parameter map corresponding to that sample image. Then, a differentiable convolution operation can be performed on the sample image based on the determined point spread function kernel to generate a new image whose bokeh effect is controlled by the bokeh parameter map. This new image is the initial bokeh image corresponding to the sample image.

[0078] Step 304: Input multiple initial blurred images into the second model to obtain a fused blurred image.

[0079] In this embodiment, the second model is another convolutional neural network, which can be denoted as CNN2. The second model can receive multiple initial blurred images as input, select and fuse them, and thus output a single fused blurred image. Through model training, the fused blurred image output by the second model can outperform any single initial blurred image in terms of edge sharpness, naturalness of background blur, and realism of highlights.

[0080] In practice, all initial blurred images generated from the same sample can be stitched together along the channel dimension to form a multi-channel input. This stitched tensor is then input into the second model. The second model learns and analyzes the advantages and disadvantages of each initial blurred image in different regions. For example, the initial blurred image v4 corresponding to sample image j4 has sharper edges, while the initial blurred image v1 corresponding to sample image j1 has a softer background blur, etc., and predicts a set of pixel-level fusion weight maps and gating information. Finally, through weighted summation and gating adjustment, a fused blurred image that integrates the advantages of all initial blurred images is output, denoted as V_fused.

[0081] As an example, after receiving the initial virtual images v1, v2, v3, and v4, the second model calculates and assigns higher weights to the initial blurred image v4 corresponding to the sharp image j4 with a small aperture in the edge area of ​​the desk to maintain sharp edges; in the background area of ​​the bookshelf in the distance, it assigns higher weights to the initial blurred image v1 corresponding to the blurred image j1 with a large aperture to enhance the sense of blur, and finally merges them into a blended blurred image V_fused with sharp edges and a soft background.

[0082] Step 305: Train the first model and the second model based on multiple sample images, a reference blurred image, and a fused blurred image to obtain a blurred parameter prediction model and a blurred image generation model.

[0083] In this embodiment, machine learning methods, such as supervised learning, can be used to train the first and second models. The trained first model is designated as the bokeh parameter prediction model, and the trained second model is designated as the bokeh image generation model. In practice, during training, the loss value can be calculated using a loss function based on the output results of the first and second models. Then, backpropagation and gradient descent algorithms can be used to calculate the gradient based on the loss value and update the parameters in the models.

[0084] It should be noted that the training order of the first and second models can be set as needed. For example, the first model can be trained first, followed by the second model, and finally the two can be trained together to obtain the bokeh parameter prediction model and the bokeh image generation model. Alternatively, the two can be trained together directly to obtain the bokeh parameter prediction model and the bokeh image generation model. No specific limitation is made here.

[0085] The method provided in the above embodiments of this application, during training, uses sample images derived from real optical differences, enabling the model to learn natural optical blurring characteristics, rather than relying on hard boundaries from semantic segmentation or a single blur filter. This avoids harsh boundaries or globally uniform blurring in the image blurring results. Furthermore, the image blurring task is decoupled into two sub-tasks: blurring parameter prediction and blurring image fusion. The first model learns the mapping relationship between scene content and blurring requirements, while the second model learns the trade-offs and fusion capabilities for different blurring results. This makes the training objectives of each model clearer and easier to optimize, resulting in more accurate and realistic blurring effects. Thus, relying solely on image data collected by a single camera, a model capable of generating blurred images with natural transition layers, high-fidelity edges, and realistic light spot morphology can be trained. Using this model for image blurring improves the realism of image blurring effects under single-camera conditions.

[0086] In some optional embodiments, the bokeh parameter maps may include, but are not limited to, a blur radius map, an edge confidence map, and a highlight saliency map. The blur radius map can be denoted as R_n, the edge confidence map as E, and the highlight saliency map as S. As an example, after inputting image j1, captured using an aperture of f / 1.6, into CNN1, CNN1 can output a set of bokeh parameter maps, including a blur radius map R_1, an edge confidence map E_1, and a highlight saliency map S_1. Similarly, for j2, it outputs a blur radius map R_2, an edge confidence map E_2, and a highlight saliency map S_2, and so on.

[0087] Based on this, in step 303 above, for each of the multiple sample images, the following steps can be performed to obtain the initial blurred image corresponding to that sample image: Step S21: Based on the blur radius map, determine the blur radius of each pixel in the sample image.

[0088] Step S22: Based on the specular saliency map and the image field position of each pixel in the sample image, determine the point spread function kernel type corresponding to each pixel in the sample image.

[0089] Step S23: Generate the target point spread function kernel based on the fuzzy radius and the point spread function kernel type.

[0090] Step S24: Based on the edge confidence map, the sample image is convolved using the target point spread function kernel to obtain the initial blurred image corresponding to the sample image.

[0091] It should be noted that the method for generating the initial blurred image corresponding to the sample image can be found in the above embodiment. To avoid repetition, it will not be repeated here.

[0092] In some optional embodiments, step 304 above may further include the following steps: Step S31: Extract the local statistical feature map of the fuzzy radius map.

[0093] The local statistical feature map may contain local statistical features of the fuzzy radius map. Local statistical features specifically refer to a set of quantified values ​​derived from the fuzzy radius map by calculating mathematical statistics within its local neighborhood, which characterize the distribution pattern of fuzzy attributes in that region. These statistics may include, but are not limited to, at least one of the following: mean, variance, maximum value, etc. For example, each pixel in the local mean feature map represents the average level of the fuzzy radius of the region surrounding that pixel in the fuzzy radius map. Its function is to provide the second model with quantified information about the spatial distribution context of the fuzziness level. The fuzzy radius value of a single pixel cannot reflect whether its surroundings are uniformly fuzzy or have sharp boundaries, while local statistical features help the second model more intelligently determine the characteristics of the region, thereby making a more reasonable fusion decision.

[0094] Step S32: The sample image with the smallest aperture value among multiple sample images is used as the sharp reference image. Multiple initial blurred images, sharp reference images, edge confidence maps, highlight saliency maps, and local statistical feature maps are input into the second model to obtain the fused blurred image.

[0095] The sample image with the smallest aperture value can be denoted as j_small. According to optical principles, the smaller the aperture, the greater the depth of field, the larger the area in sharpness in the image, and the richer the overall image detail. Therefore, j_small is the sharpest among the sample images. The sharp reference image provides the most reliable edge, texture, and microstructure information. The second model can refer to it to correct for potential detail loss in the initial blurred images generated from other sample images with larger apertures, especially in object edge regions, ensuring that the final fused result does not lose its intended sharpness.

[0096] Since each initial blurred image is based on sample images taken with different aperture values ​​and corresponding blur parameter maps, different initial blurred images have their own advantages and disadvantages in different regions. The goal of the second model is to integrate the advantages of each initial blurred image into a single, globally optimal high-quality blurred result, achieving a professional-grade effect with a soft background, sharp subject, and natural transitions.

[0097] Understandably, the blurred radius map contains only pixel-level information, while local statistical features describe the environment of the local region where the point is located. For example, high variance features can indicate edge transition regions. By extracting the local statistical feature map of the blurred radius map and inputting it into the second model, the second model can use these features to distinguish between flat background areas and complex object boundary areas. This allows for the application of appropriate weight allocation strategies to the latter during fusion, greatly improving the second network's ability to understand scene structure and effectively avoiding blurred edges or light penetration artifacts caused by incorrect fusion in edge regions.

[0098] By inputting multiple initial blurred images, sharp reference images, edge confidence maps, highlight saliency maps, and local statistical feature maps into the second model, the second model can simultaneously grasp the blurring results, detail, areas requiring protection, areas needing special bokeh processing, and the depth transition structure of the scene from sample images with different aperture values. Through cross-analysis and joint reasoning of this heterogeneous information, the second model can make fusion decisions far superior to those from a single information source. This results in a highly synergistic and unified fused blurred image in terms of overall naturalness, local sharpness, and bokeh realism, ultimately outputting a bokeh effect with optimal image quality.

[0099] In some alternative embodiments, see Figure 4 The second model includes a feature pyramid network, a weight prediction module, and a gating information generation module. Step S32 above can further include the following steps: Step S41: Input multiple initial blurred images, clear reference images, edge confidence maps, highlight saliency maps, and local statistical feature maps into the feature pyramid network to obtain fused features.

[0100] The Feature Pyramid Network (FPN) is a deep learning backbone network architecture for computer vision. Its core feature is its ability to generate a set of multi-scale feature maps. It extracts features from different levels through bottom-up forward propagation, and then fuses deep, high-semantic features with shallow, high-resolution features through top-down paths and lateral connections. In the second model, the FPN serves as the core skeleton, efficiently encoding and fusing heterogeneous input information from multiple initial blurred images, sharp reference images, and various parametric maps. Simultaneously, it extracts deep features to understand the global scene structure of the initial blurred image and extracts shallow features to understand the local fine details of the initial blurred image, providing rich and hierarchical contextual information for subsequent weight prediction and gating information generation.

[0101] Specifically, all input data can first be concatenated along the channel dimension to form a multi-channel tensor. This tensor is then fed into the encoder part of the Feature Pyramid Network (FPN). The encoder consists of multiple downsampling blocks, which may include convolution, normalization, and activation functions. The encoder progressively extracts features and reduces the resolution, forming a set of multi-scale feature maps, such as C2, C3, C4, and C5. Subsequently, based on the FPN mechanism, starting from the deepest feature map C5, it is upsampled to align with the spatial dimensions of the previous layer's feature map C4. Then, the two are fused element-wise by addition or concatenation to generate a fused feature map P4. This process is repeated to generate P3, P2, and so on. The final series of feature maps, from P2 to P5, together constitute the fused features. The fused features contain both deep semantic information and shallow detailed information.

[0102] Step S42: Input the fused features into the weight prediction module to obtain multiple weight maps corresponding to multiple initial blurred images.

[0103] The weight prediction module is a dedicated sub-network in the second model, also known as the weight prediction head. It receives the fusion features output from the feature pyramid network as input. Internally, it can consist of several convolutional layers and activation functions, with the final output channel number equal to the number of candidate initial blurred images, such as four. The core function of this module is to predict a set of pixel-level weight maps, each corresponding to one initial blurred image. Each pixel value in the weight map represents the contribution ratio of the corresponding initial blurred image in the final fusion result. Its role is to achieve adaptive, content-aware fusion source selection, allowing the model to decide which input image's blurring effect to trust based on local scene content such as texture, edges, and depth.

[0104] As an example, the weight prediction module takes the fused features of one or more layers of the FPN as input. Internally, this module performs non-linear transformations through several convolutional layers. The output channel number of its last convolutional layer is set to N, where N is the initial number of blurred images, for example, 4. The Sigmoid function is first applied to this output, mapping each value to the (0,1) interval to obtain the initial weight response. Then, to ensure that the sum of all N weights at each pixel position is 1, a Softmax function is applied along the channel dimension for normalization, ultimately outputting N normalized pixel-level weight maps W1, W2, W3, and W4. During training, the temperature parameter τ of Softmax can be adjusted to control the sharpness of the weight distribution.

[0105] Step S43: Obtain the high-frequency features of the clear reference image, and input the high-frequency features, edge confidence map, and highlight saliency map into the gating information generation module to obtain gating information.

[0106] The gating information generation module is another dedicated sub-network in the second model. It receives specific inputs strongly correlated with image quality optimization goals, such as high-frequency features of a sharp reference image, edge confidence maps, and highlight saliency maps. This module learns to generate gating information from these inputs through a small network, primarily including edge gating signals for enhancing edge sharpness and background gating signals for enhancing background blur. Its role is to generate high-level, goal-oriented control signals, providing explicit instructions for the fusion process regarding which edges to strengthen or which backgrounds to soften, thus complementing and regulating the weight prediction module.

[0107] Gating information consists of control signals output by the gating information generation module, including the edge gating signal G_edge and the background gating signal G_bg. The edge gating signal can be a control signal with a value between 0 and 1; a higher value indicates a stronger need to enhance the edge sharpness of the current area. The background gating signal is also a control signal with a value between 0 and 1; a higher value indicates a stronger need to blur the current background area more strongly.

[0108] As an example, gating information generation can be performed in parallel with weight prediction. First, a high-pass filter, such as a Laplacian operator, is used to convolve a sharp reference image j_small to extract its high-frequency feature map H, where bright regions represent edges and textures. Next, H, the edge confidence map E, and the specular saliency map S are concatenated along the channel dimension. This concatenated tensor is then fed into a lightweight gating information generation module, such as a small network with 2-3 convolutional layers. The output layer of this module has 2 channels, which are passed through a sigmoid function to independently generate two single-channel images: an edge-gated signal G_edge and a background-gated signal G_bg.

[0109] Step S44: Generate a fused blurred image based on fusion features, multiple weight maps, and gating information.

[0110] Here, the FPN decoder can start upsampling reconstruction with the fused features as input. During reconstruction, it is adjusted by the weight map and gating information. Specifically, the decoder learns to implicitly or explicitly perform the operation V_weighted=Σ(W_k×v_k). That is, it simulates the weighted fusion of multiple input images in the feature space. When the decoder upsamples to the edge region, G_edge acts as a gating switch or mixing weight, controlling the process of injecting high-frequency detail features obtained from the j_small skip connection into the current feature. It can be approximated by the formula: Feature = G_edge×j_small detail features + (1-G_edge)×current feature. G_bg can act on the weight map. In regions with high G_bg, the large aperture weight W1 is increased and renormalized, thereby indirectly affecting V_weighted. Finally, after multiple levels of upsampling and gating adjustment, the decoder outputs the final fused blurred image V_fused.

[0111] By employing a second model structure comprising a feature pyramid network, a weight prediction module, and a gating information generation module, a hierarchical, multi-mechanism collaborative image fusion scheme is achieved. The feature pyramid network enables deep scene understanding, weight prediction achieves content-adaptive fusion, and the gating mechanism precisely enhances image quality targets. This results in a systematic improvement in the overall naturalness, local detail fidelity, and core image quality of the generated fused blurred image, thus consistently producing high-quality, highly realistic bokeh effects.

[0112] In some alternative embodiments, see Figure 5 The first model consists of an encoder and a decoder. The encoder, which may include convolutional layers and downsampling layers, transforms the input image from a high-resolution, low-semantic pixel space to a low-resolution, high-semantic feature space. It extracts abstract features of the image through hierarchical convolution and compresses the spatial dimensions. The decoder is a network part that is symmetrical or partially symmetrical to the encoder and may include upsampling layers and convolutional layers. Its function is to gradually restore the high-semantic features output by the encoder to high resolution, fusing detailed features from corresponding layers of the encoder in the process, and finally outputting a blurred parameter map of the same size as the input image, such as a blur radius map, edge confidence map, and highlight saliency map. The encoder-decoder structure effectively balances global scene understanding and local detail prediction. See also... Figure 6 Step 305 above may include the following steps: Step S51: Perform the first training on the first model based on multiple sample images, a reference blurred image, and multiple initial blurred images.

[0113] The first and second models can employ a phased training strategy. The first training phase refers to the initial stage of this strategy, where the goal is to optimize the first model independently. Specifically, each sample image is first input into the first model to obtain its corresponding blurring parameter map. Then, based on the blurring parameter map, the corresponding sample images are blurred to obtain multiple initial blurred images. Next, the reconstruction loss, such as Charbonnier loss, between each initial blurred image and a reference blurred image in the samples is calculated. Simultaneously, other constraint losses are calculated, such as edge fidelity loss calculated using the minimum aperture image j_small and v_n; ranking loss and total variation regularization loss calculated using the blur radius map R_n itself. These losses are then weighted and summed to obtain the total loss L_stage1. Finally, based on the total loss L_stage1, the parameters of the first model are updated using the backpropagation algorithm.

[0114] Understandably, the core task of the first model is to understand the image content and predict physically reasonable blur parameters. Directly training the first and second models together could easily lead to training oscillations or getting stuck in local optima. Through independent training in the first stage, the first model can learn the essential mapping relationship between scene content and blur parameters with focused and efficient learning under clear supervision, providing high-quality input for all subsequent steps.

[0115] Step S52: Fix the parameters of the encoder after the first training, and perform a second training on the decoder and the second model based on the reference blurred image and the fused blurred image.

[0116] The second training stage refers to the second phase of the phased training strategy. In this stage, the parameters of the encoder in the first model are fixed, and the encoder can serve as a fixed feature extractor. The decoder of the first model and the parameters of the second model can be trained. Specifically, each sample image in the sample is first input into the first model to obtain the corresponding blurring parameter map. Then, based on the blurring parameter map corresponding to each sample image, the corresponding sample image is blurred to obtain multiple initial blurring images. Then, the multiple initial blurring images are input into the second model to obtain the fused blurring image V_fused. Then, the reconstruction loss between V_fused and the reference blurring image is calculated, such as a combination of Charbonnier loss and SSIM (Structural Similarity Index Measure) loss. At the same time, losses specifically designed for the fusion stage are calculated, such as edge consistency loss constrained between V_fused and the sharp image j_small in the edge region; hierarchical consistency loss constrained between the weight map W_k and the blur radius R_n; and constraints on the smoothness of the weight map. These constitute the total loss L_stage2. Then, based on the total loss L_stage1, the parameters of the decoder of the first model and the parameters of the second model are updated using the backpropagation algorithm.

[0117] By fixing the encoder of the first model and jointly training the decoder and the second model, it is ensured that the robust visual features learned in the first stage are not compromised, allowing the generated results to be adaptively adjusted based on feedback from the second model. The second model, in a relatively stable input feature environment, can then focus on learning complex multi-image fusion and gating logic. This strategy effectively decouples the optimization objectives of the two models, reducing the difficulty of simultaneous optimization.

[0118] It should be noted that in the second stage, the encoder of the first model can be fixed, and only the parameters of the second model and a small number of layers at the end of the decoder in the first model can be updated. Alternatively, the parameters of each model can be left unfixed, and a small learning rate can be applied to the first model to focus on training the second model. No restrictions are imposed here.

[0119] Step S53: Based on the reference blurred image and the fused blurred image, the first model and the second model are jointly trained to obtain the blurred parameter prediction model and the blurred image generation model.

[0120] The third training stage refers to the third phase of the phased training strategy. In this stage, the first and second models are trained jointly. This allows for the identification and correction of potential problems in the phased training, improving the accuracy of both models. In practice, a smaller learning rate can be set for the first model, such as lr_cnn1 = 0.2 × lr_cnn2, while a larger learning rate can be set for the second model. The learning rate can be adjusted using Cosine Annealing or OneCycle strategies to gradually reduce the temperature parameter τ. The SSIM loss weights are increased in the early stages of training, while the edge contour loss and ranking loss weights are enhanced in the later stages. By setting different learning rates for the first and second models, the front-end parameter predictions can be gently calibrated while finely adjusting the back-end fusion strategy, ensuring a high degree of synergy between the two models in pursuit of the final output quality. By employing the aforementioned phased, local-to-global training strategy, the training risks and difficulties of the dual models were effectively managed, ensuring the stability and success rate of the training process.

[0121] In some optional embodiments, a first loss function is used for supervision during the first training process. The first loss function is a composite loss function used to supervise and optimize the first model in the first stage of the phased training strategy. It consists of a weighted sum of multiple loss terms with different physical meanings and optimization objectives. Its core function is to comprehensively and multidimensionally constrain the output of the first model, requiring not only that the generated initial blurred image closely approximate the real blurred effect of the reference blurred image, but also that the predicted blurred parameter map itself possesses good geometric rationality, edge fidelity, and spatial smoothness, providing high-quality input for the subsequent fusion stage. Specifically, the first loss function includes a first reconstruction loss term, an edge fidelity loss term, a ranking loss term, and a regularization loss term. Step S51 may further include the following steps: Step S61: Based on the reference blurred image and the initial blurred image, determine the first loss value corresponding to the first reconstruction loss term.

[0122] The first reconstruction loss term measures the pixel-level or structural-level difference between the initial blurred image generated based on the blurred parameter map predicted by the first model and the reference blurred image. Specifically, a combination of Charbonnier loss and SSIM loss can be used. Charbonnier loss is more robust to outliers such as highlights and noise, and can be trained stably; SSIM loss can evaluate the similarity of images in brightness, contrast, and structure. The role of the first reconstruction loss term is to force the mapping relationship learned by the first model to generate an image that is visually highly consistent with the real optical blur, which is fundamental to ensuring the realism of the final effect.

[0123] Alternatively, the first loss value can be determined through the following steps: The first step is to obtain dynamic region masks and exposure difference masks from multiple sample images. The dynamic region mask is used to identify image regions where the optical flow displacement between multiple sample images exceeds a first threshold or the brightness difference exceeds a second threshold. The exposure difference mask is used to identify image regions where the brightness difference between multiple sample images due to aperture value differences exceeds a third threshold.

[0124] Specifically, a dynamic region mask is a binary image or weight map with the same resolution as the original image, used to identify pixel regions that undergo significant motion between multiple consecutively captured sample images with different apertures. Its construction is based on two criteria: first, geometric motion detection, which marks regions whose displacement modulus exceeds a first threshold by calculating the optical flow field between adjacent images; and second, photometric difference detection, which marks regions whose difference exceeds a second threshold by comparing the brightness differences between aligned images. The role of the dynamic region mask is to reduce the loss weight of these moving regions during training, because the position and shape of moving objects may differ in different images, leading to inconsistent or inaccurate blurring supervision signals for these regions. For example, when capturing multiple sample images, there may be fluttering curtains or walking pedestrians; these regions should be marked and weighted less heavily.

[0125] Optical flow displacement refers to the pixel-level motion vector of the same scene point between two images, estimated using optical flow algorithms. Typically, the sample image acquired at the smallest aperture is used as the reference frame, and the dense optical flow field from other aperture images to the reference frame is calculated. The optical flow displacement of each pixel is a two-dimensional vector, with its magnitude representing the motion amplitude. Optical flow displacement is used to detect the motion of macroscopic objects in a scene. For example, a falling leaf might move 5 pixels between two frames, and its optical flow displacement magnitude is 5.

[0126] An exposure difference mask is a binary image or weight map with the same resolution as the original image, used to identify regions with significant brightness differences caused by variations in the amount of light received due to physical changes in aperture value. Even though exposure parameters are carefully controlled during image capture, aperture changes inevitably alter the amount of light received, potentially causing brightness variations in localized areas such as highlights to exceed acceptable limits. This mask compares the brightness differences between aligned images taken at different apertures and marks regions where the difference exceeds a third threshold. Its purpose is to reduce the loss weight of these regions during training, preventing the model from incorrectly learning exposure changes as part of the bokeh effect. For example, an overexposed window area in a sample image captured at a large aperture value might appear properly exposed in a sample image captured at a small aperture value; this area should be marked.

[0127] The second step involves assigning pixel-level weights to different regions in multiple sample images based on dynamic region masks and exposure difference masks.

[0128] Pixel-level weights are a weight map with the same resolution as the image, where each pixel has a value between 0 and 1. They are used to assign different importance to different pixels when calculating the loss function. Pixel-level weights are jointly determined by dynamic region masks and exposure difference masks. For example, pixels in regions marked by dynamic or exposure-difference masks can be assigned lower weights, such as 0.1 or even 0; pixels in static regions with consistent exposure are assigned higher weights, such as 1.0. The purpose of pixel-level weights is to make the model focus more on learning reliable regions during training and suppress interference from unreliable regions, thereby improving training robustness and model performance.

[0129] The third step is to determine the first loss value based on pixel-level weights, the reference blurred image, and the initial blurred image.

[0130] Specifically, the first reconstruction loss can be the Charbonnier loss. For each sample image, the Charbonnier error between it and each pixel of the reference blurred image can be calculated to obtain an error map. The weighted errors of all pixels are summed and normalized to obtain the weighted loss of the image, i.e., the first loss value.

[0131] By constructing dynamic and exposure masks to identify unreliable regions in the data and suppressing their impact on training through pixel-level weight allocation, the training of the first model can focus on static, exposure-stable, high-quality regions, reducing the attention paid to interference signals caused by physical limitations of shooting. This effectively prevents the model from overfitting to motion blur, brightness changes, etc. in the data, thereby learning more accurate and robust bokeh parameter mappings, enhancing the robustness of the training process, and ensuring that a high-performance bokeh parameter prediction model can be trained even under imperfect shooting conditions in the real world.

[0132] Step S62: Take the sample image with the smallest aperture value among multiple sample images as the sharp reference image, and determine the second loss value corresponding to the edge fidelity loss term based on the initial blurred image and the sharp reference image.

[0133] The edge fidelity loss term is a geometric constraint term in the first loss function, used to maintain and optimize the quality of the initial blurred image in the object's edge region. Its core idea is that at the edges, the blur transition of the blurred image should be natural and conform to physical laws, rather than abruptly cut off. In practice, gradient profiles of the initial blurred image and the sharp reference image j_small can be extracted along the edge normal direction, and then the difference between these two profiles can be calculated. For example, by combining Charbonnier loss and Dynamic Time Warping (DTW) loss, a second loss value corresponding to the edge fidelity loss term can be obtained. The role of the edge fidelity loss term is to ensure that even in a blurred state, the roll-off of the object's outline is smooth and conforms to human visual perception, thus effectively mitigating blurred edges or overly sharp edges.

[0134] Step S63: Based on the fuzzy radius map, determine the third loss value corresponding to the ranking loss term.

[0135] The ranking loss term is the geometric-logical consistency constraint term in the first loss function. It doesn't directly constrain the image appearance, but rather constrains the logical plausibility of the spatial distribution of the blurred radius map R_n output by the first model. The principle is that within a local neighborhood, points farther from the camera should generally have larger blurred radii. This loss term randomly samples pixel pairs (p1, p2) within a local window. If, based on the gradient of the sharp image, p1 is farther than p2, then the predicted value of R_n(p1) is forced to be greater than R_n(p2). The ranking loss term introduces a weak geometric prior, preventing the model from completely deviating from the physical depth-of-field rules in its predictions, ensuring that the distribution of the blurred radius map has basic scene depth consistency, and improving the physical reliability of the prediction results.

[0136] Step S64: Based on the fuzzy radius map and the edge confidence map, determine the fourth loss value corresponding to the regularization loss term.

[0137] The regularization loss term is a prior constraint and smoothness constraint term in the first loss function, designed to impose beneficial inductive biases on the predicted blurred radius map R_n and the edge confidence map E itself. The regularization loss term may include, but is not limited to, at least one of the following: total variation (TV) loss term, edge region radius constraint term, low-texture region smoothing loss, etc. Specifically, the TV loss term acts on the blurred radius map R_n, penalizing excessive radius jumps between adjacent pixels, promoting spatial smoothness of the blurred radius, and suppressing unnecessary noise or block artifacts; the edge region radius constraint term is used in regions with high edge confidence E to force the value of R_n to not exceed a preset maximum threshold r_edge_max, preventing important object edges from being excessively blurred; the low-texture region smoothing loss term is used in regions with weak image texture, i.e., regions with small gradients, to impose stronger smoothing constraints. The role of the regularization loss term is to improve the visual quality and numerical stability of the predicted parameter map, and reduce generation artifacts caused by noise or outliers in the parameter map itself.

[0138] Step S65: Determine the first total loss value based on the first loss value, the second loss value, the third loss value, and the fourth loss value. Here, the first total loss value can be obtained by weighted summation of the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0139] Step S66: Update the parameters of the first model based on the first total loss value.

[0140] By employing a first loss function composed of the aforementioned multiple loss terms, and using this function to calculate the total loss and update the first model, a refined, multi-objective collaborative model training supervision mechanism is achieved. This mechanism not only drives the model to pursue visual realism in the final image, but also constrains and guides it from multiple deep dimensions such as edge geometry, physical logic, and output stability. This enables the finally trained bokeh parameter prediction model to output high-quality, highly reasonable, and highly stable bokeh parameters, laying the most crucial and reliable foundation for generating natural, realistic, and artifact-free single-camera high-quality bokeh effects.

[0141] In some optional embodiments, the second model includes a weight prediction module, which generates multiple weight maps corresponding to multiple initial blurred images. A second loss function is used for supervision during the second training process. The second loss function is a composite loss function used to supervise and optimize the decoder of the first model and the second model in the second stage of the phased training strategy, and consists of multiple loss terms. Its core function is to comprehensively optimize the performance of the second model, requiring not only that the final output fused blurred image closely approximate the real blurred effect overall, but also that the fusion process itself has edge consistency, logical rationality in depth hierarchy, and smoothness in weight distribution. The second loss function includes a second reconstruction loss term, an edge consistency loss term, a hierarchy consistency loss term, and a weight smoothing loss term. Step S62 may further include the following steps: Step S71: Based on the reference blurred image and the fused blurred image, determine the fifth loss value corresponding to the second reconstruction loss term.

[0142] The second reconstruction loss term measures the overall difference between the fused blurred image V_fused output by the second model and the reference blurred image. Similar to the reconstruction loss in the first training phase, it typically employs a combination of Charbonnier loss and SSIM loss. Further details are omitted here.

[0143] Step S72: Take the sample image with the smallest aperture value among multiple sample images as the sharp reference image, and determine the sixth loss value corresponding to the edge consistency loss term based on the fused blurred image and the sharp reference image.

[0144] The edge consistency loss term ensures that the fused blurred image maintains the same sharpness and structure as the sharp reference image in important edge regions. It is typically implemented by combining an edge confidence map E and high-frequency features of the sharp reference image to construct an edge region mask. Within the area covered by this mask, the difference between the gradients of the fused image V_fused and the sharp reference image j_small is calculated, such as using an L1 loss. Its function is to force the fusion network to retain or recover more details from the sharp reference image in edge regions, thus ensuring that key structures such as subject outlines, hair strands, and texture edges remain sharp against the overall blurred background, avoiding edge blurring or smudged edges caused by fusion.

[0145] Step S73: Based on multiple weight maps and fuzzy radius maps, determine the seventh loss value corresponding to the hierarchical consistency loss term.

[0146] The hierarchical consistency loss term does not directly constrain the image appearance, but rather constrains the logical consistency between the weight map W_k generated by the second model and the blur radius map R_n predicted by the first model. Its core principle is that for any two pixels, if the blur radius of point p1 is greater than that of point p2 (i.e., p1 is more blurred), then during fusion, the relative weight of the blurred image corresponding to the larger aperture at p1 should be higher than that corresponding to the smaller aperture at p2. This loss term samples pixel pairs and constrains the relationship between their weight ratio and the relative size of the blur radius to satisfy the aforementioned monotonicity. Its function is to ensure that the weight allocation decisions of the fusion network are consistent with the depth hierarchy of the scene, making the spatial logic of the fusion result coherent and avoiding contradictory weight allocations that violate physical laws, such as blurry foregrounds and clear backgrounds.

[0147] In practice, the weight map W_large corresponding to the bokeh image with the largest aperture and the weight map W_small corresponding to the bokeh image with the smallest aperture can be identified from the multiple weight maps output by the second model. A large number of pixel pairs (p_i, p_j) are randomly sampled on the blur radius map R_n predicted by the first model. The blur radius values ​​of each pixel pair are compared. If the blur radius R(p_i) > R(p_j), meaning p_i should be more blurred, then the ratio of the large aperture weight to the small aperture weight at p_i is expected to be higher than that at p_j. The specific implementation of the loss function can be based on the interval logarithmic ratio constraint: L_rank=ReLU(margin-(log((W_large(p_i)+ε) / (W_small(p_i)+ε))+log((W_large(p_j)+ε) / (W_small(p_j)+ε)))) Here, margin is a small positive interval, and ε is a constant used to prevent division by zero. The losses of all pixel pairs that violate this monotonic relationship can be summed and averaged.

[0148] Step S74: Based on multiple weight maps, determine the eighth loss value corresponding to the weight smoothing loss term.

[0149] The weight smoothing loss term applies to multiple weight maps W_k output by the second model, imposing smoothness constraints on their spatial variations. For example, total variation loss can be used to penalize abrupt changes in weight values ​​between adjacent pixels. Its purpose is to suppress blocky, speckled, or discontinuous segmentation artifacts that may appear in the weight maps, promoting a smooth spatial transition of the weight maps and avoiding visible boundaries or traces in the fused image.

[0150] Step S75: Determine the second total loss value based on at least two of the fifth, sixth, seventh, and eighth loss values. Here, the second total loss value can be obtained by weighted summation of the fifth, sixth, seventh, and eighth loss values.

[0151] Step S76: Update the parameters of the decoder and the second model based on the second total loss value.

[0152] By employing a second loss function that incorporates multiple loss terms and calculating the total loss accordingly to update the decoder and second model, a comprehensive and deep supervision mechanism for high-quality image fusion is achieved. This mechanism not only drives the network to pursue the realism of the final image but also provides precise constraints and guidance from multiple deep dimensions such as edge quality, fusion logic, and operational smoothness. This enables the finally trained decoder and second model to intelligently, reasonably, and stably fuse multiple bokeh candidate images into a high-quality bokeh image with clear edges, logical coherence, natural transitions, and high realism, achieving professional-grade bokeh effects with a single camera.

[0153] In some optional embodiments, after performing step 301, the sample images can be preprocessed and enhanced to optimize the input data quality and improve the robustness and effectiveness of subsequent model training. Specifically, this may include, but is not limited to, at least one of the following: Image alignment. When shooting with different apertures in the same scene, pixel-level misalignment may occur between images due to slight equipment movement, focal length changes during lens focusing, or rolling shutter speeds. Therefore, it is necessary to align multiple sample images from the same sample. First, feature matching algorithms such as Scale-Invariant Feature Transform (SIFT) or Oriented Fast and Rotated BRIEF (ORB) are used in conjunction with the Random Sample Consensus (RANSAC) algorithm to calculate the global transformation matrix, such as homography or affine matrix, for preliminary coarse alignment. Then, based on the coarse alignment, a pyramid-based optical flow method is used to estimate a finer, non-rigid local pixel displacement field, and thin plate spline (TPS) interpolation is used for image resampling to achieve high-precision sub-pixel-level alignment. This ensures that the bokeh parameter maps predicted from multiple sample images acquired in the same scene correspond strictly in space. For example, after alignment, the hair edges of the portrait should be in exactly the same position in multiple sample images, so as to ensure that the subsequent model can learn the accurate relationship between aperture changes and the effect of edge blurring.

[0154] Dynamic region mask construction. To reduce interference from moving objects in the scene, such as walking people and fluttering leaves, during the shooting interval, a dynamic region mask M_motion needs to be constructed. Using the clearest image j_small with the smallest aperture value among multiple sample images as the baseline frame, the dense optical flow between j_small and other sample images j_n is calculated, and j_n is transformed to the baseline viewpoint via optical flow transformation. _n. By analyzing the modulus of the optical flow residual and j_small and The absolute difference in brightness between pixels _n is used to mark pixel regions exceeding a preset threshold as dynamic regions, generating a binary mask M_motion. During training, the prediction loss corresponding to the dynamic regions identified by M_motion can be weighted less or a more robust loss function can be used. For example, if a bird flies by in different positions in several consecutive images, the constructed dynamic mask will cover the bird's region, preventing the model from overfitting the inconsistent signal caused by this moving target when learning the blurring rules.

[0155] Color and exposure standardization. Shooting at different apertures can result in subtle differences in image brightness and color. This step aims to eliminate these content-irrelevant differences, allowing the model to focus on learning the bokeh differences caused by aperture variations. Multiple sample images are adjusted to a consistent standard for color and brightness using white balance correction algorithms such as Gray-world, and tone mapping based on device response curves or Reinhard algorithms. Simultaneously, the images are uniformly converted to linear RGB (Red, Green, Blue) or the standard working color space. For example, suppose a brighter image with slightly higher color saturation is captured at a large aperture of f / 1.6, while the image is darker at a smaller aperture of f / 4.0. Standardization corrects for this overall brightness and color shift caused by exposure, ensuring that the differences in the images input to the first model primarily stem from depth-of-field variations, rather than exposure or white balance settings.

[0156] A weakly supervised depth signal is introduced. To provide a basic geometric scene structure prior for the training of the first model and prevent its predictions from completely deviating from the physical depth logic, a weakly supervised depth map D_weak is introduced. A lightweight monocular depth estimation network is used to process a clear, small-aperture image j_small to generate a corresponding depth estimation map and its confidence map C_depth. In regions with confidence scores above a threshold τ, depth information is added as an auxiliary constraint to the loss function of CNN1 training. The depth map provides a rough representation of the scene's distance, such as people being closer and the background being farther away. When training CNN1 to predict the blur radius map R_n, in addition to fitting the visual effect, a weak constraint is applied in regions with high confidence scores, ensuring that the predicted blur radius maintains a roughly monotonically consistent spatial distribution with the depth information, i.e., a larger blur radius trend corresponds to farther objects, thereby improving the physical plausibility of the prediction results.

[0157] Data augmentation. To improve the model's generalization ability and enable it to cope with various real-world shooting conditions, online or offline data augmentation of the samples is necessary. Augmentation operations include two categories: first, geometric transformations, such as random rotation, scaling, cropping, and horizontal flipping; and second, optical characteristic simulation, such as color jittering, gamma correction, contrast adjustment, and injecting sensor noise of varying intensities. Furthermore, complex lighting scenes such as backlighting, low light, and high dynamic range (HDR) can be simulated. By applying these augmentation transformations to the original sample images and their ground truth (GT), the diversity of the training data can be greatly expanded. For example, through color jittering, the model can learn to correctly predict the bokeh effect in environments with warm or cool skin tones; by injecting noise, the model can more robustly handle noisy images taken by mobile phones in low-light environments.

[0158] It should be noted that the image generation method provided in this application can be executed by an image generation device. This application uses an image generation device executing the image generation method as an example to illustrate the image generation device provided in this application.

[0159] like Figure 7 As shown, the image generation apparatus 700 of this embodiment includes: an acquisition unit 701 for acquiring an original image; a first input unit 702 for inputting the original image into a bokeh parameter prediction model to obtain a bokeh parameter map; a bokeh processing unit 703 for performing bokeh processing on the original image based on the bokeh parameter map to generate an initial bokeh image; and a second input unit 704 for inputting the initial bokeh image into a bokeh image generation model to obtain a target bokeh image.

[0160] In some optional implementations of this embodiment, the bokeh parameter map includes a blur radius map, a highlight saliency map, and an edge confidence map. The blur radius map includes the blur radius corresponding to each pixel in the original image. The highlight saliency map is used to identify the out-of-focus highlight areas in the original image. The edge confidence map is used to indicate the confidence level of each pixel in the original image being located at the edge of the image field. The bokeh processing unit 703 is further configured to: determine the blur radius of each pixel in the original image based on the blur radius map; determine the point spread function kernel type corresponding to each pixel in the original image based on the highlight saliency map and the image field position of each pixel in the original image; generate a target point spread function kernel based on the blur radius and the point spread function kernel type; and perform convolution processing on the original image using the target point spread function kernel based on the edge confidence map to obtain the initial bokeh image.

[0161] In some optional implementations of this embodiment, the blurring processing unit 703 is further configured to: determine at least two candidate kernel radii from the discrete kernel radius set based on the fuzzy radius; generate at least two candidate point spread function kernels based on the point spread function kernel type and the at least two candidate kernel radii; and interpolate the parameters of the at least two candidate point spread function kernels based on the proportional relationship between the fuzzy radius and the at least two candidate kernel radii to obtain the target point spread function kernel.

[0162] In some optional implementations of this embodiment, the bokeh parameter prediction model and the bokeh image generation model are trained through the following steps: obtaining a sample set, wherein each sample in the sample set includes multiple sample images and a reference bokeh image, wherein the multiple sample images are taken by the same electronic device in the same scene using different aperture values; inputting the multiple sample images into a first model to obtain multiple bokeh parameter maps; performing bokeh processing on the multiple sample images based on the multiple bokeh parameter maps to obtain multiple initial bokeh images; inputting the multiple initial bokeh images into a second model to obtain a fused bokeh image; training the first model and the second model based on the multiple sample images, the reference bokeh image, and the fused bokeh image to obtain the bokeh parameter prediction model and the bokeh image generation model.

[0163] In some optional implementations of this embodiment, the first model includes an encoder and a decoder; training the first model and the second model based on the multiple sample images, the reference bokeh image, and the fused bokeh image to obtain a bokeh parameter prediction model and a bokeh image generation model includes: performing a first training on the first model based on the multiple sample images, the reference bokeh image, and the multiple initial bokeh images; fixing the parameters of the encoder after the first training, and performing a second training on the decoder and the second model based on the reference bokeh image and the fused bokeh image; and jointly training the first model and the second model based on the reference bokeh image and the fused bokeh image to obtain the bokeh parameter prediction model and the bokeh image generation model.

[0164] The apparatus provided in the above embodiments of this application uses a parameter prediction model to perform scene understanding on the original image to generate blurring parameters, which enhances the display control of the blurring process. The blurring parameter map serves as an intermediate control signal to precisely guide the blurring process, ensuring the physical plausibility of the generated initial blurred image. Furthermore, by using a blurred image generation model to perform image optimization processing to generate a target blurred image, a more accurate and realistic blurring effect can be obtained based on the initial blurred image. This process does not rely on hardware assistance or simple semantic segmentation, but is achieved through the collaborative processing of two models, which can improve the realism of the image blurring effect without increasing hardware costs.

[0165] The image generation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0166] The image generation device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0167] The image generation apparatus provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments of this application, and the image generation apparatus provided in the embodiments of this application, can realize... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0168] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image generation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0169] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0170] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0171] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. The processor 910 is used to acquire the original image; input the original image into the bokeh parameter prediction model to obtain a bokeh parameter map; perform bokeh processing on the original image based on the bokeh parameter map to generate an initial bokeh image; and input the initial bokeh image into the bokeh image generation model to obtain a target bokeh image.

[0172] By using a parameter prediction model to perform scene understanding on the original image to generate blurring parameters, the display control of the blurring process can be enhanced. The blurring parameter map serves as an intermediate control signal to precisely guide the blurring process, ensuring the physical plausibility of the generated initial blurred image. Furthermore, by using a blurred image generation model to perform image optimization processing to generate the target blurred image, a more accurate and realistic blurring effect can be achieved based on the initial blurred image. This process does not rely on hardware assistance or simple semantic segmentation, but is achieved through the collaborative processing of two models, improving the realism of the image blurring effect without increasing hardware costs.

[0173] Optionally, in some optional implementations of this embodiment, the bokeh parameter map includes a blur radius map, a highlight saliency map, and an edge confidence map. The blur radius map includes the blur radius corresponding to each pixel in the original image. The highlight saliency map is used to identify the out-of-focus highlight region in the original image. The edge confidence map is used to indicate the confidence level of each pixel in the original image being located at the edge of the image field. The processor 910 is further configured to: determine the blur radius of each pixel in the original image based on the blur radius map; determine the point spread function kernel type corresponding to each pixel in the original image based on the highlight saliency map and the image field position of each pixel in the original image; generate a target point spread function kernel based on the blur radius and the point spread function kernel type; and perform convolution processing on the original image using the target point spread function kernel based on the edge confidence map to obtain the initial bokeh image.

[0174] In some optional implementations of this embodiment, the processor 910 is further configured to determine at least two candidate kernel radii from the discrete kernel radius set based on the fuzzy radius; generate at least two candidate point spread function kernels based on the point spread function kernel type and the at least two candidate kernel radii; and interpolate the parameters of the at least two candidate point spread function kernels based on the proportional relationship between the fuzzy radius and the at least two candidate kernel radii to obtain the target point spread function kernel.

[0175] In some optional implementations of this embodiment, the bokeh parameter prediction model and the bokeh image generation model are trained through the following steps: obtaining a sample set, wherein each sample in the sample set includes multiple sample images and a reference bokeh image, wherein the multiple sample images are taken by the same electronic device in the same scene using different aperture values; inputting the multiple sample images into a first model to obtain multiple bokeh parameter maps; performing bokeh processing on the multiple sample images based on the multiple bokeh parameter maps to obtain multiple initial bokeh images; inputting the multiple initial bokeh images into a second model to obtain a fused bokeh image; training the first model and the second model based on the multiple sample images, the reference bokeh image, and the fused bokeh image to obtain the bokeh parameter prediction model and the bokeh image generation model.

[0176] In some optional implementations of this embodiment, the first model includes an encoder and a decoder; training the first model and the second model based on the multiple sample images, the reference bokeh image, and the fused bokeh image to obtain a bokeh parameter prediction model and a bokeh image generation model includes: performing a first training on the first model based on the multiple sample images, the reference bokeh image, and the multiple initial bokeh images; fixing the parameters of the encoder after the first training, and performing a second training on the decoder and the second model based on the reference bokeh image and the fused bokeh image; and jointly training the first model and the second model based on the reference bokeh image and the fused bokeh image to obtain the bokeh parameter prediction model and the bokeh image generation model.

[0177] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0178] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0179] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0180] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image generation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0181] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0182] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0183] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0184] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described method embodiments and achieve the same technical effects. To avoid repetition, further details are omitted here.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0187] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image generation method characterized by, The method includes: Obtain the original image; The original image is input into the blur parameter prediction model to obtain a blur parameter map; The original image is blurred based on the blurring parameter map to generate an initial blurred image; The initial blurred image is input into the blurred image generation model to obtain the target blurred image.

2. The method of claim 1, wherein, The blurring parameter map includes a blur radius map, a highlight saliency map, and an edge confidence map. The blur radius map includes the blur radius corresponding to each pixel in the original image. The highlight saliency map is used to identify the out-of-focus highlight areas in the original image. The edge confidence map is used to indicate the confidence level of each pixel in the original image located at the edge of the image field. The step of blurring the original image based on the blurring parameter map to generate an initial blurred image includes: Based on the blur radius map, the blur radius of each pixel in the original image is determined; Based on the specular saliency map and the image field position of each pixel in the original image, the point spread function kernel type corresponding to each pixel in the original image is determined; Based on the fuzzy radius and the point spread function kernel type, a target point spread function kernel is generated; Based on the edge confidence map, the original image is convolved using the target point spread function kernel to obtain the initial blurred image.

3. The method of claim 2, wherein, The step of generating the target point spread function kernel based on the fuzzy radius and the point spread function kernel type includes: Based on the fuzzy radius, at least two candidate kernel radii are determined from the set of discrete kernel radii; Based on the point spread function kernel type and the radii of the at least two candidate kernels, at least two candidate point spread function kernels are generated; Based on the proportional relationship between the fuzzy radius and the radii of the at least two candidate kernels, the parameters of the at least two candidate point diffusion function kernels are interpolated to obtain the target point diffusion function kernel.

4. The method of claim 1, wherein, The bokeh parameter prediction model and the bokeh image generation model are trained through the following steps: A sample set is obtained, wherein each sample in the sample set includes multiple sample images and a reference blurred image, wherein the multiple sample images are taken by the same electronic device in the same scene using different aperture values; The multiple sample images are input into the first model respectively to obtain multiple blurring parameter maps; Based on the multiple blurring parameter maps, the multiple sample images are blurred to obtain multiple initial blurred images; The multiple initial blurred images are input into the second model to obtain the fused blurred image; The first model and the second model are trained based on the multiple sample images, the reference blurred image, and the fused blurred image to obtain the blurred parameter prediction model and the blurred image generation model.

5. The method according to claim 4, characterized in that, The first model includes an encoder and a decoder; the step of training the first model and the second model based on the multiple sample images, the reference blurred image, and the fused blurred image to obtain a blurred parameter prediction model and a blurred image generation model includes: The first model is trained based on the multiple sample images, the reference blurred image, and the multiple initial blurred images. The parameters of the encoder after the first training are fixed, and the decoder and the second model are trained a second time based on the reference blurred image and the fused blurred image; Based on the reference blurred image and the fused blurred image, the first model and the second model are jointly trained to obtain the blurred parameter prediction model and the blurred image generation model.

6. An image generation apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the original image; The first input unit is used to input the original image into the bokeh parameter prediction model to obtain a bokeh parameter map; A blurring processing unit is used to blur the original image based on the blurring parameter map to generate an initial blurred image; The second input unit is used to input the initial blurred image into the blurred image generation model to obtain the target blurred image.

7. The apparatus according to claim 6, characterized in that, The bokeh parameter map includes a blur radius map, a highlight saliency map, and an edge confidence map. The blur radius map includes the blur radius corresponding to each pixel in the original image. The highlight saliency map is used to identify out-of-focus highlight areas in the original image. The edge confidence map is used to indicate the confidence level of each pixel in the original image located at the edge of the image field. The bokeh processing unit is further used for: Based on the blur radius map, the blur radius of each pixel in the original image is determined; Based on the specular saliency map and the image field position of each pixel in the original image, the point spread function kernel type corresponding to each pixel in the original image is determined; Based on the fuzzy radius and the point spread function kernel type, a target point spread function kernel is generated; Based on the edge confidence map, the original image is convolved using the target point spread function kernel to obtain the initial blurred image.

8. The apparatus according to claim 7, characterized in that, The virtualization processing unit is also used for: Based on the fuzzy radius, at least two candidate kernel radii are determined from the set of discrete kernel radii; Based on the point spread function kernel type and the radii of the at least two candidate kernels, at least two candidate point spread function kernels are generated; Based on the proportional relationship between the fuzzy radius and the radii of the at least two candidate kernels, the parameters of the at least two candidate point diffusion function kernels are interpolated to obtain the target point diffusion function kernel.

9. The apparatus according to claim 6, characterized in that, The bokeh parameter prediction model and the bokeh image generation model are trained through the following steps: A sample set is obtained, wherein each sample in the sample set includes multiple sample images and a reference blurred image, wherein the multiple sample images are taken by the same electronic device in the same scene using different aperture values; The multiple sample images are input into the first model respectively to obtain multiple blurring parameter maps; Based on the multiple blurring parameter maps, the multiple sample images are blurred to obtain multiple initial blurred images; The multiple initial blurred images are input into the second model to obtain the fused blurred image; The first model and the second model are trained based on the multiple sample images, the reference blurred image, and the fused blurred image to obtain a blurred parameter prediction model and a blurred image generation model.

10. The apparatus according to claim 9, characterized in that, The first model includes an encoder and a decoder; the step of training the first model and the second model based on the multiple sample images, the reference blurred image, and the fused blurred image to obtain a blurred parameter prediction model and a blurred image generation model includes: The first model is trained based on the multiple sample images, the reference blurred image, and the multiple initial blurred images. The parameters of the encoder after the first training are fixed, and the decoder and the second model are trained a second time based on the reference blurred image and the fused blurred image; Based on the reference blurred image and the fused blurred image, the first model and the second model are jointly trained to obtain the blurred parameter prediction model and the blurred image generation model.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the image generation method as described in any one of claims 1-5.

Images