A variable aperture bokeh rendering method and system

Abstract

The present application relates to the technical field of computer vision, and discloses a variable aperture bokeh rendering method and system. The method first acquires a full-focus input image, a depth map thereof and a target aperture parameter; extracts features of the full-focus input image by inputting the full-focus input image into a neural network of an encoder-bottleneck layer-decoder structure; inputs the depth map and the target aperture parameter into an adaptive routing prompt module to generate a prompt vector; in the encoder stage, the prompt vector is used to differentially process the features through adaptive spatial routing, and a layered sparse attention mechanism based on the depth map is combined to suppress inter-region interference; in the bottleneck layer and the decoder stage, the prompt vector is used for channel modulation and fusion of the jump connection features to gradually restore image details; and finally, the output bokeh residual image is fused with the full-focus input image to generate a target aperture bokeh image. The present application effectively improves the controllability, boundary naturalness and detail fidelity of bokeh rendering.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a variable aperture bokeh rendering method and system. Background Technology

[0002] Bokeh blur is a computational photography technique that simulates the depth-of-field effect of an optical lens. It aims to maintain sharp focus while creating a natural blur in the out-of-focus areas, enhancing the aesthetics of the image and the prominence of the subject. With the development of mobile photography, user demands have evolved from generating fixed bokeh to "variable aperture rendering." This requires the ability to generate high-quality bokeh effects of corresponding intensity in real-time based on different aperture parameters for a single fully focused image, while maintaining robustness in scenes containing complex edges and structures.

[0003] To achieve this goal, existing technologies are mainly divided into two categories: those based on explicit physical models (such as layered depth-of-field compositing) and those based on deep learning (such as end-to-end networks). However, both have systemic defects when dealing with high-order variable aperture rendering tasks: First, the lack of explicit and depth modeling of aperture parameters leads to inaccurate control and unstable output; second, they are sensitive to depth estimation errors, easily producing problems such as halos and artifacts at boundaries and thin structures; third, the lack of regional isolation constraints within the network leads to mutual contamination between in-focus and out-of-focus features, destroying depth-of-field consistency; fourth, the use of globally shared parameters makes it difficult to achieve differentiated processing of spatial positions; and fifth, general feature fusion strategies cannot reconcile the inherent conflict between high-frequency details and low-frequency blur in bokeh tasks.

[0004] In summary, existing bokeh rendering methods still have shortcomings in terms of controllability of variable aperture, robustness to depth error, regional consistency constraints, and location-based differential modeling. There is an urgent need for a variable aperture bokeh rendering method that can explicitly inject target aperture conditions to drive feature routing, while suppressing cross-regional erroneous interactions through hierarchical regionalized sparse attention, in order to obtain more natural, stable, and controllable bokeh rendering results. Summary of the Invention

[0005] To address the problems of insufficient variable aperture adjustment capability, the tendency for subjects to be unnecessarily blurred in complex depth-of-field conditions, and the unnatural blurring caused by confusion between in-focus and out-of-focus areas in existing methods, the present invention aims to provide a variable aperture bokeh rendering method and system.

[0006] To achieve the above objectives, the present invention provides a variable aperture bokeh rendering method, the method comprising:

[0007] Acquire the full-focus input image, the corresponding depth map of the full-focus input image, and the target aperture parameters set by the user;

[0008] The fully focused input image is processed through a convolutional layer to extract shallow features, which are then input into a neural network with an encoder-bottleneck-decoder structure.

[0009] The depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector;

[0010] In the encoder stage of the neural network, the shallow features are subjected to adaptive spatial routing processing based on the cue vector to obtain spatial routing enhanced features;

[0011] The depth map is used to divide the field region, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features;

[0012] The hierarchical attention enhancement features are input into the encoder for downsampling. During the downsampling process, skip connection features are generated and saved, and deep coding features are obtained.

[0013] In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features;

[0014] In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features;

[0015] The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features;

[0016] The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output the target aperture bokeh image.

[0017] Further, the depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector; specifically including:

[0018] The depth map is encoded using a convolutional neural network, and a depth weight vector is obtained after global average pooling and normalization.

[0019] The target aperture parameters are encoded using a multilayer perceptron to obtain an aperture weight vector;

[0020] The depth weight vector and the aperture weight vector are added together and normalized to obtain the routing weight vector;

[0021] The routing weight vector is multiplied by a cue dictionary consisting of multiple learnable vectors to generate the cue vector.

[0022] Furthermore, in the encoder stage of the neural network, adaptive spatial routing processing is performed on the shallow features based on the cue vector to obtain spatial routing enhancement features; specifically including:

[0023] The cue vector is mapped through a linear layer and then broadcast to a spatial size with the same spatial dimensions as the shallow features to obtain a spatial cue feature map.

[0024] The spatial cue feature map is concatenated with the shallow features along the channel dimension to obtain the intermediate features;

[0025] For each spatial location on the intermediate feature, calculate its corresponding set of expert weight distributions;

[0026] Based on the expert weight distribution, select the top K expert networks with the highest weights for each spatial location;

[0027] The intermediate features of each spatial location are input into the corresponding selected expert network for calculation, and the outputs of each expert network are weighted and fused according to the expert weights to obtain the spatial routing enhancement features.

[0028] Furthermore, based on the depth map, the depth region is divided, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features; specifically including:

[0029] The depth map is normalized and preprocessed with bilateral filtering to obtain a preprocessed depth map;

[0030] Based on the preprocessed depth map, the main focal region in the preprocessed depth map is determined by saliency detection and connected component analysis, and the average depth of the main focal region is calculated as the focal depth.

[0031] Based on the depth values ​​of the preprocessed depth map, the cumulative distribution function is calculated, and the initial threshold for dividing the depth area is determined according to the cumulative distribution function.

[0032] The initial threshold is translated and corrected based on the focal depth to obtain the corrected threshold.

[0033] Based on the corrected threshold, a focal region mask, a transition region mask, and a background region mask are generated, and the pixel ratio of each region is calculated.

[0034] The spatial routing enhancement features are transformed by a feedforward network and then split into a query matrix, a key matrix, and a value matrix.

[0035] Using the focal region mask, transition region mask, and background region mask, the query matrix, key matrix, and value matrix are respectively isolated to their corresponding depth regions;

[0036] For each depth region, calculate the attention weight matrix within that region, and for each row of the attention weight matrix, retain only the weights corresponding to the top K key vectors with the highest weight values, and reset the remaining weights to zero to obtain sparse attention weights.

[0037] The sparse attention weights are normalized, and the normalized result is multiplied by the corresponding value vector to obtain the sparse attention output for each region.

[0038] Based on the pixel proportion of each region and the learnable weight coefficients, the sparse attention outputs of each region are weighted and fused, and the fused output is transformed by a convolutional layer and added to the spatial routing enhancement feature to obtain the hierarchical attention enhancement feature.

[0039] Furthermore, the hierarchical attention enhancement features are input into the encoder for downsampling processing. During the downsampling process, skip connection features are generated and saved to obtain deep coding features; specifically including:

[0040] The hierarchical attention enhancement features are input into the encoder for downsampling to obtain the first-scale deep coding features.

[0041] The first-scale deep coding features are input into the encoder for downsampling to obtain the second-scale deep coding features.

[0042] The hierarchical attention enhancement features, the first-scale deep coding features, and the second-scale deep coding features are respectively used as the first skip connection features, the second skip connection features, and the third skip connection features; the first skip connection features, the second skip connection features, and the third skip connection features are saved;

[0043] The second-scale deep coding features are used as input bottleneck layers for the deep coding features.

[0044] Furthermore, in the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; specifically including:

[0045] The cue vector is input into a linear layer for dimensionality transformation, and the transformed output is mapped to a Sigmoid activation function. The channel gating coefficient is obtained from the interval;

[0046] The channel gating coefficients are broadcast to the same spatial size as the deep coding features to obtain a channel gating coefficient map;

[0047] The channel gating coefficient map is multiplied element-wise with the deep coding features to complete the global modulation of the feature channels and obtain the channel routing enhancement features.

[0048] Furthermore, in the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the stored skip connection features to obtain concatenated features; specifically including:

[0049] The channel routing enhancement features are upsampled to obtain the first upsampled features;

[0050] The first upsampled feature and the third skip connection feature are concatenated in the channel dimension to obtain the first concatenated feature;

[0051] The first spliced ​​feature is upsampled to obtain the second upsampled feature;

[0052] The second upsampled feature and the second skip connection feature are concatenated along the channel dimension to obtain the second concatenated feature;

[0053] The second concatenated feature is upsampled to obtain the third upsampled feature;

[0054] The third upsampling feature is concatenated with the first skip connection feature in the channel dimension to obtain the third concatenated feature.

[0055] Furthermore, the concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; specifically including:

[0056] Based on the depth map and target aperture parameters, the adaptive routing prompting module regenerates the prompting vector;

[0057] The regenerated cue vector is mapped to channel gating coefficients and multiplied element-wise with the first concatenation feature to obtain the first modulation feature;

[0058] The first modulation feature, the corresponding depth-of-field mask, and the pixel ratio of the region are used to perform layered sparse attention calculation to obtain the first attention enhancement feature; wherein, the depth-of-field mask includes a focal region mask, a transition region mask, and a background region mask.

[0059] The first attention-enhanced feature is input into the decoder for processing to obtain the first decoded output feature;

[0060] The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the second concatenation feature to obtain the second modulation feature. The second modulation feature is then input into the decoder for processing to obtain the second decoding output feature.

[0061] The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the third concatenation feature to obtain the third modulation feature. The third modulation feature is then input into the decoder for processing to obtain the third decoding output feature.

[0062] The third decoding output feature is the final decoding feature.

[0063] Further, the final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output a target aperture bokeh image; specifically including:

[0064] The final decoded features are mapped through a 3x3 convolutional layer to convert their channel count to 3, resulting in a bokeh residual image.

[0065] The pixel-level residuals of the bokeh residual image and the fully focused input image are added together to obtain the final target aperture bokeh image.

[0066] Furthermore, the present invention also includes a variable aperture bokeh rendering system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the steps of the variable aperture bokeh rendering method.

[0067] The beneficial effects of this invention are:

[0068] This invention fuses depth information with user aperture commands through an adaptive routing cue module to generate a unified cue vector. This vector is then used to drive adaptive routing in the spatial dimension and global modulation in the channel dimension. This ensures that the neural network with its encoder-bottleneck-decoder structure can accurately understand and respond to different aperture conditions. This results in rendering of the same input image with linearly varying intensity, physically consistent bokeh, and a stable style, completely resolving the style drift and effect abruptness problems caused by ambiguous conditional responses in existing methods.

[0069] The layered sparse attention proposed in this invention precisely regionalizes the scene based on the depth map and forces Top-K sparse attention calculations within each region. This mechanism of region isolation and internal selection fundamentally cuts off erroneous feature interactions between in-focus, transition, and background regions, effectively suppressing halos, artifacts, and texture fragmentation caused by inaccurate depth maps or complex object boundaries (such as hair strands or fences). Thus, while achieving natural background blur, it maintains the sharpness of high-frequency details in the focus area.

[0070] This invention can improve the realism, structural consistency and controllability of variable aperture bokeh rendering, and reduce problems such as artifacts, edge halos and regional discontinuities in complex scenes. It has strong robustness and engineering application value. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0072] This embodiment provides a variable aperture bokeh rendering method, such as... Figure 1 As shown, it includes the following steps:

[0073] Acquire the full-focus input image, the corresponding depth map of the full-focus input image, and the target aperture parameters set by the user;

[0074] The fully focused input image is processed through a convolutional layer to extract shallow features, which are then input into a neural network with an encoder-bottleneck-decoder structure.

[0075] The depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector;

[0076] In the encoder stage of the neural network, the shallow features are subjected to adaptive spatial routing processing based on the cue vector to obtain spatial routing enhanced features;

[0077] The depth map is used to divide the field region, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features;

[0078] The hierarchical attention enhancement features are input into the encoder for downsampling. During the downsampling process, skip connection features are generated and saved, and deep coding features are obtained.

[0079] In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features;

[0080] In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features;

[0081] The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features;

[0082] The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output the target aperture bokeh image.

[0083] The methods for obtaining the depth map corresponding to the fully focused input image include, but are not limited to, the following two methods: one is to predict it based on the fully focused input image using a depth estimation network (DepthAnything); the other is to directly acquire it using devices such as binocular vision cameras, time-of-flight (ToF) sensors, or structured light.

[0084] This involves acquiring the user-defined target aperture parameter, which can be a discrete aperture level or a continuous scalar. To facilitate processing by the subsequent layer perceptron, the target aperture parameter needs to be processed as follows: If the target aperture parameter is a discrete aperture level (e.g., level 1 to level 5), it is one-hot encoded; if the user-defined target aperture parameter is a continuous scalar (e.g., aperture value), it is processed as follows: ), and then normalize it.

[0085] Specifically, the fully focused input image is processed through a convolutional layer to extract shallow features, which are then input into a neural network with an encoder-bottleneck-decoder structure; the convolutional layer is a 3x3 convolutional layer.

[0086] Among them, the neural network with encoder-bottleneck-decoder structure refers to a U-Net.

[0087] In this embodiment, the present invention uses a depth estimation network (DepthAnything) based on the fully focused input image. The depth map is obtained by predicting the depth map corresponding to the fully focused input image. Users set the strong bokeh level to a continuous scalar value (such as aperture value). The target aperture parameter is obtained by normalizing the sample. .

[0088] The fully focused input image Shallow features are extracted using a 3x3 convolutional layer. , ,in, Represent real numbers, Indicates the height of shallow features. Indicates the width of shallow features. This represents the number of channels for shallow features, and the shallow features are... The input is fed into a U-shaped network (U-Net) with an encoder-bottleneck-decoder structure.

[0089] Further, the depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector; specifically including:

[0090] The depth map is encoded using a convolutional neural network, and the depth weight vector is obtained after global average pooling and normalization; the formula is as follows:

[0091] ;

[0092] in, Represents the depth weight vector. This represents a convolutional neural network (CNN). Indicates global pooling. This indicates normalization processing;

[0093] The target aperture parameters are encoded using a multilayer perceptron to obtain the aperture weight vector; the formula is as follows:

[0094]

[0095] in, Represents the aperture weight vector. Represents a multilayer perceptron;

[0096] The depth weight vector is added to the aperture weight vector and then normalized to obtain the routing weight vector; the formula is as follows:

[0097]

[0098] in, Represents the route weight vector;

[0099] The routing weight vector is multiplied by a cue dictionary consisting of multiple learnable vectors to generate the cue vector; the formula is as follows:

[0100] ;

[0101] ;

[0102] ;

[0103] in, This represents the first component of the routing weight vector. This represents the nth component of the routing weight vector; This indicates a dictionary suggestion. This indicates the first learnable vector in the dictionary. This indicates the nth learnable vector in the dictionary; Represents the cue vector, This represents the total number of learnable vectors.

[0104] Furthermore, in the encoder stage of the neural network, adaptive spatial routing processing is performed on the shallow features based on the cue vector to obtain spatial routing enhancement features; specifically including:

[0105] The cue vector is mapped through a linear layer and then broadcast to a spatial size with the same dimension as the shallow features to obtain a spatial cue feature map; the formula is as follows:

[0106] ;

[0107] in, Representation spatial cue feature map, Indicates a broadcast operation. Indicates a linear layer;

[0108] The spatial cue feature map is concatenated with the shallow features along the channel dimension to obtain the intermediate features; the formula is as follows:

[0109] ;

[0110] in, Indicating intermediate features, in the formula Indicates a splicing operation;

[0111] For each spatial location on the intermediate feature, calculate its corresponding set of expert weight distributions; the formula is as follows:

[0112] ;

[0113] in, Indicates the spatial location of intermediate features Expert scores, This represents the transpose of a learnable linear transformation matrix. Indicates the spatial location of intermediate features eigenvectors, Indicates the spatial location of intermediate features Corresponding expert weight distribution;

[0114] Based on the expert weight distribution, select the top K expert networks with the highest weights for each spatial location; the formula is as follows:

[0115]

[0116] in, This represents the set of indices for the top K expert networks with the highest weights. This represents the Top-K index selection function, used to select the indices corresponding to the K experts with the highest weights from the expert weight distribution. This indicates the number of expert networks selected for each spatial location;

[0117] The intermediate features of each spatial location are input into the corresponding selected expert network for computation, and the outputs of each expert network are weighted and fused according to expert weights to obtain the spatial routing enhancement features; the formula is as follows:

[0118] ;

[0119] in, Indicates the spatial location of intermediate features The corresponding spatial routing enhancement features, Indicates the spatial location of intermediate features No. The expert weights corresponding to each expert network Indicates the first A network of experts.

[0120] Furthermore, based on the depth map, the depth region is divided, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features; specifically including:

[0121] The depth map is normalized and preprocessed using bilateral filtering to obtain a preprocessed depth map; the formula is as follows:

[0122] ;

[0123] in, This represents the preprocessed depth map. This represents a bilateral filtering function used to smooth and denoise an image while preserving its edge information.

[0124] Based on the preprocessed depth map, the main focal region in the preprocessed depth map is determined through saliency detection and connected component analysis, and the average depth of the main focal region is calculated as the focal depth; the formula is as follows:

[0125] ;

[0126] in, This indicates the main focal area in the preprocessed depth map. This represents the maximum connected component extraction function, used to extract the connected component with the largest area from the saliency detection results. This represents a saliency detection function used to identify salient regions in a depth map;

[0127] ;

[0128] in, Indicates the depth of focus. This indicates the number of pixels within the main focal area. This represents the pixel coordinates within the focal area of ​​the subject. This indicates the preprocessed depth map in pixel coordinates. The depth value;

[0129] Based on the depth values ​​of the preprocessed depth map, a cumulative distribution function is calculated, and an initial threshold for dividing the depth-of-field region is determined according to the cumulative distribution function; wherein, the initial threshold is divided into an initial low threshold and an initial high threshold; the formula is as follows:

[0130] ;

[0131] in, Indicates the initial low threshold. This indicates taking the minimum value. This represents the depth value of the preprocessed depth map. Represents the cumulative distribution function. Indicates the initial high threshold. This indicates taking the maximum value;

[0132] The initial threshold is translated and corrected based on the focal depth to obtain the corrected threshold; wherein, the corrected threshold is divided into a corrected low threshold and a corrected high threshold; the corrected threshold divides the depth-of-field region into a focal region, a transition region, and a background region; the formula is as follows:

[0133] ;

[0134] ;

[0135] in, This indicates that the focus depth is normalized and mapped to... grayscale values ​​within the range This indicates rounding down. Indicates the threshold shift amount. This represents a numerical truncation function. This indicates the translation limit threshold. Indicates the lower threshold after correction. Indicates the high threshold after correction;

[0136] Based on the corrected threshold, a focal region mask, a transition region mask, and a background region mask are generated, and the pixel ratio of each region is calculated.

[0137] ;

[0138] ;

[0139] ; ;in, Indicates the focal area mask. This indicates an indicator function that takes the value 1 if the condition is true, and 0 otherwise. Indicates the transition region mask. Indicates the background area mask. This indicates the percentage of pixels occupied by the region. Indicates the depth-of-field mask. This indicates the width of the preprocessed depth map. Indicates the height of the preprocessed depth map;

[0140] The spatial routing enhancement features are transformed by a feedforward network and then split into a query matrix, a key matrix, and a value matrix.

[0141] Using the focal region mask, transition region mask, and background region mask, the query matrix, key matrix, and value matrix are respectively isolated to their corresponding depth regions;

[0142] For each depth region, calculate the attention weight matrix within that region, and for each row of the attention weight matrix, retain only the weights corresponding to the top K key vectors with the highest weight values, and reset the remaining weights to zero to obtain sparse attention weights.

[0143] The sparse attention weights are normalized, and the normalized result is multiplied by the corresponding value vector to obtain the sparse attention output for each region.

[0144] Based on the pixel proportion of each region and the learnable weight coefficients, the sparse attention outputs of each region are weighted and fused, and the fused output is transformed by a convolutional layer and added to the spatial routing enhancement feature to obtain the hierarchical attention enhancement feature.

[0145] Furthermore, the hierarchical attention enhancement features are input into the encoder for downsampling processing. During the downsampling process, skip connection features are generated and saved to obtain deep coding features; specifically including:

[0146] The hierarchical attention enhancement features are input into the encoder for downsampling to obtain the first-scale deep coding features.

[0147] The first-scale deep coding features are input into the encoder for downsampling to obtain the second-scale deep coding features.

[0148] The hierarchical attention enhancement features, the first-scale deep coding features, and the second-scale deep coding features are respectively used as the first skip connection features, the second skip connection features, and the third skip connection features; the first skip connection features, the second skip connection features, and the third skip connection features are saved;

[0149] The second-scale deep coding features are used as input bottleneck layers for the deep coding features.

[0150] Furthermore, in the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; specifically including:

[0151] The cue vector is input into a linear layer for dimensionality transformation, and the transformed output is mapped to a Sigmoid activation function. The channel gating coefficient is obtained from the interval;

[0152] The channel gating coefficients are broadcast to the same spatial size as the deep coding features to obtain a channel gating coefficient map;

[0153] The channel gating coefficient map is multiplied element-wise with the deep coding features to complete the global modulation of the feature channels and obtain the channel routing enhancement features.

[0154] Furthermore, in the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the stored skip connection features to obtain concatenated features; specifically including:

[0155] The channel routing enhancement features are upsampled to obtain the first upsampled features;

[0156] The first upsampled feature and the third skip connection feature are concatenated in the channel dimension to obtain the first concatenated feature;

[0157] The first spliced ​​feature is upsampled to obtain the second upsampled feature;

[0158] The second upsampled feature and the second skip connection feature are concatenated along the channel dimension to obtain the second concatenated feature;

[0159] The second concatenated feature is upsampled to obtain the third upsampled feature;

[0160] The third upsampling feature is concatenated with the first skip connection feature in the channel dimension to obtain the third concatenated feature.

[0161] Furthermore, the concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; specifically including:

[0162] Based on the depth map and target aperture parameters, the adaptive routing prompting module regenerates the prompting vector;

[0163] The regenerated cue vector is mapped to channel gating coefficients and multiplied element-wise with the first concatenation feature to obtain the first modulation feature;

[0164] The first modulation feature, the corresponding depth-of-field mask, and the pixel ratio of the region are used to perform layered sparse attention calculation to obtain the first attention enhancement feature; wherein, the depth-of-field mask includes a focal region mask, a transition region mask, and a background region mask.

[0165] The first attention-enhanced feature is input into the decoder for processing to obtain the first decoded output feature;

[0166] The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the second concatenation feature to obtain the second modulation feature. The second modulation feature is then input into the decoder for processing to obtain the second decoding output feature.

[0167] The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the third concatenation feature to obtain the third modulation feature. The third modulation feature is then input into the decoder for processing to obtain the third decoding output feature.

[0168] The third decoding output feature is the final decoding feature.

[0169] Further, the final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output a target aperture bokeh image; specifically including:

[0170] The final decoded features are mapped through a 3x3 convolutional layer to convert their channel count to 3, resulting in a bokeh residual image.

[0171] The pixel-level residuals of the bokeh residual image and the fully focused input image are added together to obtain the final target aperture bokeh image.

[0172] Furthermore, the present invention also includes a variable aperture bokeh rendering system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the steps of the variable aperture bokeh rendering method.

[0173] This embodiment provides a variable aperture bokeh rendering system. The system includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the variable aperture bokeh rendering method.

[0174] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A variable aperture bokeh rendering method, characterized in that, The methods include: Acquire the full-focus input image, the corresponding depth map of the full-focus input image, and the target aperture parameters set by the user; The fully focused input image is processed through a convolutional layer to extract shallow features, which are then input into a neural network with an encoder-bottleneck-decoder structure. The depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector; The generation of the prompt vector specifically includes: The depth map is encoded using a convolutional neural network, and a depth weight vector is obtained after global average pooling and normalization. The target aperture parameters are encoded using a multilayer perceptron to obtain an aperture weight vector; The depth weight vector and the aperture weight vector are added together and normalized to obtain the routing weight vector; The routing weight vector is multiplied by a cue dictionary consisting of multiple learnable vectors to generate the cue vector; In the encoder stage of the neural network, adaptive spatial routing processing is performed on the shallow features based on the cue vector to obtain spatial routing enhancement features; specifically, obtaining the spatial routing enhancement features includes: The cue vector is mapped through a linear layer and then broadcast to a spatial size with the same spatial dimensions as the shallow features to obtain a spatial cue feature map. The spatial cue feature map is concatenated with the shallow features along the channel dimension to obtain the intermediate features; For each spatial location on the intermediate feature, calculate its corresponding set of expert weight distributions; Based on the expert weight distribution, select the top K expert networks with the highest weights for each spatial location; The intermediate features of each spatial location are input into the corresponding selected expert network for calculation, and the outputs of each expert network are weighted and fused according to the expert weights to obtain the spatial routing enhancement features. The depth map is used to divide the field region, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features; the obtaining of hierarchical attention enhancement features specifically includes: The depth map is normalized and preprocessed with bilateral filtering to obtain a preprocessed depth map; Based on the preprocessed depth map, the main focal region in the preprocessed depth map is determined by saliency detection and connected component analysis, and the average depth of the main focal region is calculated as the focal depth. Based on the depth values ​​of the preprocessed depth map, the cumulative distribution function is calculated, and the initial threshold for dividing the depth area is determined according to the cumulative distribution function. The initial threshold is translated and corrected based on the focal depth to obtain the corrected threshold. Based on the corrected threshold, a focal region mask, a transition region mask, and a background region mask are generated, and the pixel ratio of each region is calculated. The spatial routing enhancement features are transformed by a feedforward network and then split into a query matrix, a key matrix, and a value matrix. Using the focal region mask, transition region mask, and background region mask, the query matrix, key matrix, and value matrix are respectively isolated to their corresponding depth regions; For each depth region, calculate the attention weight matrix within that region, and for each row of the attention weight matrix, retain only the weights corresponding to the top K key vectors with the highest weight values, and reset the remaining weights to zero to obtain sparse attention weights. The sparse attention weights are normalized, and the normalized result is multiplied by the corresponding value vector to obtain the sparse attention output for each region. Based on the pixel proportion of each region and the learnable weight coefficients, the sparse attention outputs of each region are weighted and fused, and the fused output is transformed by a convolutional layer and added to the spatial routing enhancement feature to obtain the hierarchical attention enhancement feature. The hierarchical attention enhancement features are input into the encoder for downsampling. During the downsampling process, skip connection features are generated and saved, and deep coding features are obtained. In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features; The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output the target aperture bokeh image.

2. The variable aperture bokeh rendering method according to claim 1, characterized in that, The hierarchical attention enhancement features are input into the encoder for downsampling. During the downsampling process, skip connection features are generated and saved, resulting in deep coding features; specifically including: The hierarchical attention enhancement features are input into the encoder for downsampling to obtain the first-scale deep coding features. The first-scale deep coding features are input into the encoder for downsampling to obtain the second-scale deep coding features. The hierarchical attention enhancement features, the first-scale deep coding features, and the second-scale deep coding features are respectively used as the first skip connection features, the second skip connection features, and the third skip connection features; the first skip connection features, the second skip connection features, and the third skip connection features are saved; The second-scale deep coding features are used as input bottleneck layers for the deep coding features.

3. The variable aperture bokeh rendering method according to claim 2, characterized in that, In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; Specifically, it includes: The cue vector is input into a linear layer for dimensionality transformation, and the transformed output is mapped to a Sigmoid activation function. The channel gating coefficient is obtained from the interval; The channel gating coefficients are broadcast to the same spatial size as the deep coding features to obtain a channel gating coefficient map; The channel gating coefficient map is multiplied element-wise with the deep coding features to complete the global modulation of the feature channels and obtain the channel routing enhancement features.

4. The variable aperture bokeh rendering method according to claim 3, characterized in that, In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features; specifically including: The channel routing enhancement features are upsampled to obtain the first upsampled features; The first upsampled feature and the third skip connection feature are concatenated in the channel dimension to obtain the first concatenated feature; The first spliced ​​feature is upsampled to obtain the second upsampled feature; The second upsampled feature and the second skip connection feature are concatenated along the channel dimension to obtain the second concatenated feature; The second concatenated feature is upsampled to obtain the third upsampled feature; The third upsampling feature is concatenated with the first skip connection feature in the channel dimension to obtain the third concatenated feature.

5. The variable aperture bokeh rendering method according to claim 4, characterized in that, The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; Specifically, it includes: Based on the depth map and target aperture parameters, the adaptive routing prompting module regenerates the prompting vector; The regenerated cue vector is mapped to channel gating coefficients and multiplied element-wise with the first concatenation feature to obtain the first modulation feature; The first modulation feature, the corresponding depth-of-field mask, and the pixel ratio of the region are used to perform layered sparse attention calculation to obtain the first attention enhancement feature; wherein, the depth-of-field mask includes a focal region mask, a transition region mask, and a background region mask. The first attention-enhanced feature is input into the decoder for processing to obtain the first decoded output feature; The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the second concatenation feature to obtain the second modulation feature. The second modulation feature is then input into the decoder for processing to obtain the second decoding output feature. The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the third concatenation feature to obtain the third modulation feature. The third modulation feature is then input into the decoder for processing to obtain the third decoding output feature. The third decoding output feature is the final decoding feature.

6. The variable aperture bokeh rendering method according to claim 5, characterized in that, The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output a target aperture bokeh image; specifically including: The final decoded features are mapped through a 3x3 convolutional layer to convert their channel count to 3, resulting in a bokeh residual image. The pixel-level residuals of the bokeh residual image and the fully focused input image are added together to obtain the final target aperture bokeh image.

7. A variable aperture bokeh rendering system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the variable aperture bokeh rendering method as described in any one of claims 1 to 6.

Images