Variable aperture scatter rendering method and system

Abstract

The invention relates to the technical field of computer vision, and discloses a variable aperture scatter rendering method and system. The method comprises the following steps: firstly, acquiring a full-focus input image, a depth map and a target aperture parameter; inputting the full-focus input image into a neural network of an encoder-bottleneck layer-decoder structure to extract features; inputting the depth map and the target aperture parameter to an adaptive routing prompt module to generate a prompt vector; in an encoder stage, performing differentiation processing on the features through adaptive spatial routing by using a prompt vector, and inhibiting inter-region interference in combination with a depth map-based layered sparse attention mechanism; in a bottleneck layer and decoder stage, channel modulation is carried out by using a prompt vector and jump connection features are fused, and image details are gradually recovered; and finally, fusing the output scatter-scene residual image with the full-focus input image to generate a target aperture scatter-scene image. According to the method, the controllability, the boundary naturalness and the detail fidelity of boarder rendering are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a variable aperture bokeh rendering method and system. BACKGROUND

[0002] Bokeh is a kind of computational photography technology that simulates the depth of field effect of an optical lens, aiming to keep the focus clear while making the out-of-focus area naturally blurred to improve the picture beauty and subject prominence. With the development of mobile photography, user demand has upgraded from generating fixed bokeh to "variable aperture rendering", that is, it is required to generate corresponding intensity and high-quality bokeh effects in real time according to different aperture parameters for a single all-focus image, and it is required to maintain robustness in scenes containing complex edges and structures.

[0003] To achieve this goal, existing technologies are mainly divided into two categories based on explicit physical models (such as layered depth of field synthesis) and deep learning (such as end-to-end networks). However, they all have systematic defects when dealing with high-order variable aperture rendering tasks: first, there is a lack of explicit and deep modeling of aperture parameters, resulting in inaccurate control and unstable output; second, they are sensitive to depth estimation errors and prone to halos, artifacts and other problems at boundaries and thin structures; third, there is a lack of regional isolation constraints within the network, resulting in mutual contamination of in-focus and out-of-focus features, which destroys the consistency of the depth of field; fourth, the use of globally shared parameters makes it difficult to achieve differentiated processing of spatial positions; and fifth, the universal feature fusion strategy cannot reconcile the inherent conflict between high-frequency details and low-frequency blur in the bokeh task.

[0004] In summary, existing bokeh rendering methods still have deficiencies in variable aperture controllability, depth error robustness, regional consistency constraints, and position-differentiated modeling, and there is an urgent need for a variable aperture bokeh rendering method that can explicitly inject and drive feature routing under target aperture conditions, while suppressing cross-zone misinteraction through layered regionalized sparse attention, to obtain more natural, stable and controllable bokeh rendering results. SUMMARY

[0005] In order to solve the problems of insufficient variable aperture adjustment capability of existing methods, easy mis-virtualization of the subject under complex depth of field relationships, and unnatural virtualization caused by confusion of in-focus and out-of-focus regions, the purpose of the present application is to provide a variable aperture bokeh rendering method and system.

[0006] In order to achieve the above purpose, the present application provides a variable aperture bokeh rendering method, which comprises: obtaining an all-focus input image, a depth map corresponding to the all-focus input image, and a target aperture parameter set by a user; extracting shallow layer features from the all-focus input image through a convolution layer and inputting the shallow layer features into a neural network of an encoder-bottleneck layer-decoder structure; inputting the depth map and the target aperture parameter into an adaptive routing hint module to generate a hint vector; in an encoder stage of the neural network, performing adaptive spatial routing processing on the shallow layer features according to the hint vector to obtain spatial routing enhanced features; dividing depth regions based on the depth map and performing layered sparse attention calculation and fusion on the spatial routing enhanced features to obtain layered attention enhanced features; inputting the layered attention enhanced features into an encoder for down-sampling processing, generating and saving a skip connection feature in the down-sampling processing, and obtaining deep layer encoding features; in a bottleneck layer of the neural network, mapping the hint vector into a channel gate coefficient, performing global modulation on the deep layer encoding features to obtain channel routing enhanced features; in a decoder stage of the neural network, performing up-sampling on the channel routing enhanced features and splicing the up-sampled channel routing enhanced features with the saved skip connection feature to obtain spliced features; performing channel modulation on the spliced features using the hint vector and applying layered sparse attention calculation to obtain final decoding features; mapping the final decoding features into a bokeh residual map, fusing the bokeh residual map with the all-focus input image, and outputting a target aperture bokeh image.

[0007] Further, the depth map and the target aperture parameter are inputted into an adaptive routing hint module to generate a hint vector; specifically including: encoding the depth map using a convolutional neural network, obtaining a depth weight vector through global average pooling and normalization processing; encoding the target aperture parameter using a multi-layer perception to obtain an aperture weight vector; adding and normalizing the depth weight vector and the aperture weight vector to obtain a routing weight vector; multiplying the routing weight vector with a hint dictionary composed of multiple learnable vectors to generate the hint vector.

[0008] Further, in the encoder stage of the neural network, the hint vector is used to perform adaptive spatial routing processing on the shallow layer features to obtain spatial routing enhanced features; specifically including: mapping the hint vector through a linear layer and then broadcasting the hint vector to the same spatial size as the shallow layer features to obtain a spatial hint feature map; splicing the spatial hint feature map and the shallow layer features in the channel dimension to obtain intermediate features; For each spatial position on the intermediate feature, a corresponding set of expert weight distribution is calculated; According to the expert weight distribution, the top K expert networks with the highest weights are selected for each spatial position; The intermediate feature of each spatial position is input into the corresponding selected expert network for calculation, and the outputs of each expert network are weighted and fused according to the expert weights to obtain a spatial routing enhanced feature.

[0009] Further, based on the depth map, the depth of field regions are divided, and the spatial routing enhanced feature is calculated and fused with layered sparse attention to obtain a layered attention enhanced feature; specifically including: The depth map is normalized and preprocessed by bilateral filtering to obtain a preprocessed depth map; Based on the preprocessed depth map, the main focus region in the preprocessed depth map is determined by saliency detection and connected component analysis, and the average depth of the main focus region is calculated as the focus depth; According to the depth value of the preprocessed depth map, the cumulative distribution function is calculated, and the initial threshold for dividing the depth of field region is determined according to the cumulative distribution function; According to the focus depth, the initial threshold is translated and corrected to obtain a corrected threshold; According to the corrected threshold, a focus region mask, a transition region mask and a background region mask are generated, and the pixel proportion of each region is calculated; After the spatial routing enhanced feature is transformed by a feedforward network, it is split into a query matrix, a key matrix and a value matrix; Using the focus region mask, the transition region mask and the background region mask, the query matrix, the key matrix and the value matrix are isolated into the corresponding depth of field region respectively; For each depth of field region, the attention weight matrix in the region is calculated, and for each row in the attention weight matrix, only the top K key vectors with the highest weight values are retained, and the remaining weights are set to zero to obtain a sparse attention weight; The sparse attention weight is normalized, and the normalized result is multiplied by the corresponding value vector to obtain the sparse attention output of each region; According to the pixel proportion of each region and the learnable weight coefficient, the sparse attention outputs of each region are weighted and fused, and the fused output is transformed by a convolution layer and added to the spatial routing enhanced feature to obtain the layered attention enhanced feature.

[0010] Further, the layered attention enhanced feature is input into an encoder for down-sampling processing, and in the down-sampling processing, a skip connection feature is generated and saved, and a deep layer encoding feature is obtained; specifically including: input the layered attention enhanced feature into an encoder for down-sampling processing to obtain a first scale deep layer encoding feature; input the first scale deep layer encoding feature into an encoder for down-sampling processing to obtain a second scale deep layer encoding feature; the layered attention enhanced feature, the first scale deep layer encoding feature and the second scale deep layer encoding feature are respectively taken as a first jump connection feature, a second jump connection feature and a third jump connection feature; the first jump connection feature, the second jump connection feature and the third jump connection feature are saved; The second scale deep layer encoding feature is taken as the deep layer encoding feature input into the bottleneck layer.

[0011] Further, in the bottleneck layer of the neural network, the prompt vector is mapped into a channel gating coefficient, and the deep layer encoding feature is globally modulated to obtain a channel routing enhanced feature; specifically including: input the prompt vector into a linear layer for dimension transformation, and map the transformed output to interval through a Sigmoid activation function to obtain a channel gating coefficient; broadcast the channel gating coefficient to the same spatial size as the deep layer encoding feature to obtain a channel gating coefficient map; multiply the channel gating coefficient map and the deep layer encoding feature element by element to complete global modulation of the feature channel, and obtain a channel routing enhanced feature.

[0012] Further, in the decoder stage of the neural network, the channel routing enhanced feature is up-sampled and spliced with the saved jump connection feature to obtain a spliced feature; specifically including: up-sample the channel routing enhanced feature to obtain a first up-sampled feature; splice the first up-sampled feature and the third jump connection feature in the channel dimension to obtain a first spliced feature; up-sample the first spliced feature to obtain a second up-sampled feature; splice the second up-sampled feature and the second jump connection feature in the channel dimension to obtain a second spliced feature; up-sample the second spliced feature to obtain a third up-sampled feature; splice the third up-sampled feature and the first jump connection feature in the channel dimension to obtain a third spliced feature.

[0013] Further, the prompt vector is used to modulate the spliced feature in the channel, and a layered sparse attention calculation is applied to obtain a final decoding feature; specifically including: Regenerate a prompt vector through the adaptive routing prompt module based on the depth map and the target aperture parameter; Map the regenerated prompt vector to a channel gating coefficient, and perform element-wise multiplication with the first spliced feature to obtain a first modulation feature; Perform layered sparse attention calculation on the first modulation feature, a corresponding depth-of-field area mask, and a pixel proportion occupied by the area to obtain a first attention-enhanced feature; wherein the depth-of-field area mask includes a focal point area mask, a transition area mask, and a background area mask; Input the first attention-enhanced feature into a decoder for processing to obtain a first decoding output feature; Map the channel gating coefficient of the regenerated prompt vector to the second spliced feature to obtain a second modulation feature, and input the second modulation feature into a decoder for processing to obtain a second decoding output feature; Map the channel gating coefficient of the regenerated prompt vector to the third spliced feature to obtain a third modulation feature, and input the third modulation feature into a decoder for processing to obtain a third decoding output feature; The third decoding output feature is the final decoding feature.

[0014] Further, map the final decoding feature to a bokeh residual image, fuse it with the all-focus input image, and output a target aperture bokeh image; specifically including: Map the final decoding feature through a 3x3 convolution layer to convert its channel number to 3 channels to obtain a bokeh residual image; Add the bokeh residual image and the all-focus input image at the pixel level to obtain the final target aperture bokeh image.

[0015] Further, the present application also includes a variable aperture bokeh rendering system, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the variable aperture bokeh rendering method when executing the computer program.

[0016] The present application has the following advantages: The application fuses the depth information and the user aperture instruction to generate a unified prompt vector through an adaptive routing prompt module, and uses the vector to drive the adaptive routing of the space dimension and the global modulation of the channel dimension, which ensures that the neural network with the encoder-bottleneck layer-decoder structure can accurately understand and respond to different aperture conditions, so that the same input image can generate a rendering result with linear intensity change, physically consistent bokeh shape and stable style, and completely solves the style drift and effect jump problems caused by the ambiguous condition response of the existing method.

[0017] The layered sparse attention proposed by the application performs accurate regionalization division on the scene according to the depth map, and forces Top-K sparse attention calculation within the region. The mechanism of region isolation and internal selection cuts off the false feature interaction between the in-focus, transition and background regions in principle, effectively suppresses the halo, artifacts and texture breaking phenomenon caused by the inaccuracy of the depth map or the complexity of the object boundary (such as hair and fence), so that the background is naturally blurred while the sharpness of the high-frequency details in the focus area is maintained.

[0018] The application can improve the realism, structural consistency and controllability of variable aperture bokeh rendering, and reduce the problems of artifacts, edge halo and region discontinuity in complex scenes, and has strong robustness and engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the application. DETAILED DESCRIPTION

[0020] The embodiment provides a variable aperture bokeh rendering method, as shown in Figure 1 The method comprises the following steps: obtaining a full-focus input image, a depth map corresponding to the full-focus input image and a target aperture parameter set by a user; extracting shallow layer features of the full-focus input image through a convolution layer, and inputting the shallow layer features into a neural network with an encoder-bottleneck layer-decoder structure; inputting the depth map and the target aperture parameter into an adaptive routing prompt module to generate a prompt vector; in the encoder stage of the neural network, performing adaptive spatial routing processing on the shallow layer features according to the prompt vector to obtain spatial routing enhanced features; dividing the depth regions based on the depth map, and performing layered sparse attention calculation and fusion on the spatial routing enhanced features to obtain layered attention enhanced features; inputting the layered attention enhanced features into the encoder for down-sampling processing, generating and saving the skip connection features in the down-sampling processing, and obtaining deep layer encoding features; In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features; The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output the target aperture bokeh image.

[0021] The methods for obtaining the depth map corresponding to the fully focused input image include, but are not limited to, the following two methods: one is to predict it based on the fully focused input image using a depth estimation network (DepthAnything); the other is to directly acquire it using devices such as binocular vision cameras, time-of-flight (ToF) sensors, or structured light.

[0022] This involves acquiring the user-defined target aperture parameter, which can be a discrete aperture level or a continuous scalar. To facilitate processing by the subsequent layer perceptron, the target aperture parameter needs to be processed as follows: If the target aperture parameter is a discrete aperture level (e.g., level 1 to level 5), it is one-hot encoded; if the user-defined target aperture parameter is a continuous scalar (e.g., aperture value), it is processed as follows: ), and then normalize it.

[0023] Specifically, the fully focused input image is processed through a convolutional layer to extract shallow features, which are then input into a neural network with an encoder-bottleneck-decoder structure; the convolutional layer is a 3x3 convolutional layer.

[0024] Among them, the neural network with encoder-bottleneck-decoder structure refers to a U-Net.

[0025] In this embodiment, the present invention uses a depth estimation network (DepthAnything) based on the fully focused input image. The depth map is obtained by predicting the depth map corresponding to the fully focused input image. Users set the strong bokeh level to a continuous scalar value (such as aperture value). The target aperture parameter is obtained by normalizing the sample. .

[0026] The fully focused input image Shallow features are extracted using a 3x3 convolutional layer. , ,in, Represent real numbers, Indicates the height of shallow features. Indicates the width of shallow features. This represents the number of channels for shallow features, and the shallow features are... The input is fed into a U-shaped network (U-Net) with an encoder-bottleneck-decoder structure.

[0027] Further, the depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector; specifically including: The depth map is encoded using a convolutional neural network, and the depth weight vector is obtained after global average pooling and normalization; the formula is as follows: ; in, Represents the depth weight vector. This represents a convolutional neural network (CNN). Indicates global pooling. This indicates normalization processing; The target aperture parameters are encoded using a multilayer perceptron to obtain an aperture weight vector; the formula is as follows: in, Represents the aperture weight vector. Represents a multilayer perceptron; The depth weight vector is added to the aperture weight vector and then normalized to obtain the routing weight vector; the formula is as follows: in, Represents the route weight vector; The routing weight vector is multiplied by a cue dictionary consisting of multiple learnable vectors to generate the cue vector; the formula is as follows: ; ; ; in, This represents the first component of the routing weight vector. This represents the nth component of the routing weight vector; This indicates a dictionary suggestion. This indicates the first learnable vector in the dictionary. This indicates the nth learnable vector in the dictionary; Represents the cue vector, This represents the total number of learnable vectors.

[0028] Further, in the encoder stage of the neural network, the shallow layer features are adaptively spatially routed according to the prompt vector to obtain spatially routed enhanced features; specifically including: After the prompt vector is mapped through a linear layer, it is broadcast to the same spatial size as the shallow layer features to obtain a spatial prompt feature map; the formula is as follows: ; Wherein, represents the spatial prompt feature map, represents the broadcast operation, represents the linear layer; The spatial prompt feature map and the shallow layer features are spliced in the channel dimension to obtain intermediate features; the formula is as follows: ; Wherein, represents the intermediate features, and in the formula represents the splicing operation; For each spatial position on the intermediate features, a corresponding set of expert weight distributions is calculated; the formula is as follows: ; Wherein, represents the expert score of the intermediate features at the spatial position , represents the transpose of the learnable linear transformation matrix, represents the feature vector of the intermediate features at the spatial position , represents the expert weight distribution corresponding to the spatial position of the intermediate features; According to the expert weight distribution, the top K expert networks with the highest weights are selected for each spatial position; the formula is as follows: Wherein, represents the index set of the top K expert networks with the highest weights, represents a Top-K index selection function for selecting the indexes corresponding to the top K experts with the highest weights from the expert weight distribution, represents the number of expert networks selected for each spatial position; The intermediate features of each spatial position are input into the corresponding selected expert networks for calculation, and the outputs of the expert networks are weighted and fused according to the expert weights to obtain spatially routed enhanced features; the formula is as follows: ; Wherein, represents the expert score of the intermediate features at the spatial position Corresponding spatial routing enhancement features, representing the intermediate features in spatial positions The expert network corresponding to the expert weight, representing the first expert network.

[0029] Further, based on the depth map, the depth area is divided, and the hierarchical sparse attention calculation and fusion of the spatial routing enhancement features are performed to obtain the hierarchical attention enhancement features; specifically including: The depth map is normalized and bilateral filter preprocessed to obtain a preprocessed depth map; the formula is as follows: ; Wherein, representing the preprocessed depth map, representing the bilateral filter function, which is used to smooth the image while preserving the image edge information; Based on the preprocessed depth map, the main focus area in the preprocessed depth map is determined by saliency detection and connected component analysis, and the average depth of the main focus area is calculated as the focus depth; the formula is as follows: ; Wherein, representing the main focus area in the preprocessed depth map, representing the maximum connected component extraction function, which is used to extract the largest connected component from the saliency detection result, representing the saliency detection function, which is used to identify the saliency area in the depth map; ; Wherein, representing the focus depth, representing the number of pixel points in the main focus area, representing the pixel coordinates in the main focus area, representing the depth value of the preprocessed depth map at the pixel coordinates ; According to the depth value of the preprocessed depth map, the cumulative distribution function is calculated, and the initial threshold for dividing the depth area is determined according to the cumulative distribution function; wherein the initial threshold is divided into initial low threshold and initial high threshold; the formula is as follows: ; Wherein, representing the initial low threshold, representing the minimum value, representing the depth value of the preprocessed depth map, representing the cumulative distribution function, representing the initial high threshold, This indicates taking the maximum value; The initial threshold is translated and corrected based on the focal depth to obtain the corrected threshold; wherein, the corrected threshold is divided into a corrected low threshold and a corrected high threshold; the corrected threshold divides the depth-of-field region into a focal region, a transition region, and a background region; the formula is as follows: ; ; in, This indicates that the focus depth is normalized and mapped to... grayscale values ​​within the range Indicates rounding down. Indicates the threshold shift amount. This represents a numerical truncation function. This indicates the translation amount limit threshold. Indicates the lower threshold after correction. Indicates the high threshold after correction; Based on the corrected threshold, a focal region mask, a transition region mask, and a background region mask are generated, and the pixel ratio of each region is calculated. ; ; ; ;in, Indicates the focal area mask. This indicates an indicator function that takes the value 1 if the condition is true, and 0 otherwise. Indicates the transition region mask. Indicates the background area mask. This indicates the percentage of pixels occupied by the region. Indicates the depth-of-field mask. This indicates the width of the preprocessed depth map. Indicates the height of the preprocessed depth map; The spatial routing enhancement features are transformed by a feedforward network and then split into a query matrix, a key matrix, and a value matrix. Using the focal region mask, transition region mask, and background region mask, the query matrix, key matrix, and value matrix are respectively isolated to their corresponding depth regions; For each depth region, calculate the attention weight matrix within that region, and for each row of the attention weight matrix, retain only the weights corresponding to the top K key vectors with the highest weight values, and reset the remaining weights to zero to obtain sparse attention weights. The sparse attention weights are normalized, and the normalized result is multiplied by the corresponding value vector to obtain the sparse attention output for each region. According to the pixel proportion of each region and the learnable weight coefficient, the sparse attention outputs of each region are weighted and fused, and the fused output is transformed through a convolution layer and added to the spatial routing enhanced feature to obtain the hierarchical attention enhanced feature.

[0030] Further, the hierarchical attention enhanced feature is input into an encoder for down-sampling processing, and in the down-sampling processing, a skip connection feature is generated and saved, and a deep layer encoding feature is obtained; specifically including: The hierarchical attention enhanced feature is input into an encoder for down-sampling processing to obtain a first scale deep layer encoding feature; The first scale deep layer encoding feature is input into an encoder for down-sampling processing to obtain a second scale deep layer encoding feature; The hierarchical attention enhanced feature, the first scale deep layer encoding feature, and the second scale deep layer encoding feature are respectively taken as a first skip connection feature, a second skip connection feature, and a third skip connection feature; the first skip connection feature, the second skip connection feature, and the third skip connection feature are saved; The second scale deep layer encoding feature is taken as the deep layer encoding feature input into a bottleneck layer.

[0031] Further, in the bottleneck layer of the neural network, the prompt vector is mapped into a channel gating coefficient, and the deep layer encoding feature is globally modulated to obtain a channel routing enhanced feature; specifically including: The prompt vector is input into a linear layer for dimension transformation, and the transformed output is mapped into interval by a Sigmoid activation function to obtain a channel gating coefficient; The channel gating coefficient is broadcasted to the same spatial size as the deep layer encoding feature to obtain a channel gating coefficient map; The channel gating coefficient map is multiplied element by element with the deep layer encoding feature to complete global modulation of the feature channel, and a channel routing enhanced feature is obtained.

[0032] Further, in the decoder stage of the neural network, the channel routing enhanced feature is up-sampled and spliced with the saved skip connection feature to obtain a spliced feature; specifically including: The channel routing enhanced feature is up-sampled to obtain a first up-sampling feature; The first up-sampling feature is spliced with the third skip connection feature in the channel dimension to obtain a first spliced feature; The first spliced feature is up-sampled to obtain a second up-sampling feature; The second up-sampling feature is spliced with the second jump connection feature in a channel dimension to obtain a second spliced feature; The second spliced feature is up-sampled to obtain a third up-sampled feature; The third up-sampled feature is spliced with the first jump connection feature in a channel dimension to obtain a third spliced feature.

[0033] Further, the spliced feature is channel-modulated by using the prompt vector, and a hierarchical sparse attention calculation is applied to obtain a final decoding feature; specifically including: Based on the depth map and the target aperture parameter, a prompt vector is regenerated by the adaptive routing prompt module; The regenerated prompt vector is mapped into a channel gating coefficient, and is multiplied element by element with the first spliced feature to obtain a first modulated feature; The first modulated feature, a corresponding depth-of-field area mask and a pixel proportion occupied by the area are subjected to hierarchical sparse attention calculation to obtain a first attention-enhanced feature; wherein the depth-of-field area mask includes a focal point area mask, a transition area mask and a background area mask; The first attention-enhanced feature is input into a decoder for processing to obtain a first decoding output feature; The channel gating coefficient mapped by the regenerated prompt vector is multiplied element by element with the second spliced feature to obtain a second modulated feature, and the second modulated feature is input into a decoder for processing to obtain a second decoding output feature; The channel gating coefficient mapped by the regenerated prompt vector is multiplied element by element with the third spliced feature to obtain a third modulated feature, and the third modulated feature is input into a decoder for processing to obtain a third decoding output feature; The third decoding output feature is the final decoding feature.

[0034] Further, the final decoding feature is mapped into a bokeh residual map, and is fused with the all-focus input image to output a target aperture bokeh image; specifically including: The final decoding feature is mapped through a 3x3 convolution layer to convert its channel number into 3 channels to obtain a bokeh residual map; The bokeh residual map is added to the all-focus input image in a pixel level to obtain a final target aperture bokeh image.

[0035] Further, the present application also includes a variable aperture bokeh rendering system, the system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, the processor realizing the steps of the variable aperture bokeh rendering method when executing the computer program.

[0036] The present embodiment provides a variable aperture bokeh rendering system, the system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, the processor realizing the steps of the variable aperture bokeh rendering method when executing the computer program.

[0037] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, the scope of the present application being defined by the appended claims and their equivalents.

Claims

1. A variable aperture bokeh rendering method, characterized in that, The methods include: Acquire the full-focus input image, the corresponding depth map of the full-focus input image, and the target aperture parameters set by the user; The fully focused input image is processed through a convolutional layer to extract shallow features, which are then input into a neural network with an encoder-bottleneck-decoder structure. The depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector; In the encoder stage of the neural network, the shallow features are subjected to adaptive spatial routing processing based on the cue vector to obtain spatial routing enhanced features; The depth map is used to divide the field region, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features; The hierarchical attention enhancement features are input into the encoder for downsampling. During the downsampling process, skip connection features are generated and saved, and deep coding features are obtained. In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features; The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output the target aperture bokeh image.

2. The variable aperture bokeh rendering method according to claim 1, characterized in that, The depth map and target aperture parameters are input into the adaptive routing prompting module to generate a prompting vector; specifically including: The depth map is encoded using a convolutional neural network, and a depth weight vector is obtained after global average pooling and normalization. The target aperture parameters are encoded using a multilayer perceptron to obtain an aperture weight vector; The depth weight vector and the aperture weight vector are added together and normalized to obtain the routing weight vector; The routing weight vector is multiplied by a cue dictionary consisting of multiple learnable vectors to generate the cue vector.

3. The variable aperture bokeh rendering method according to claim 2, characterized in that, In the encoder stage of the neural network, the shallow features are subjected to adaptive spatial routing processing based on the cue vector to obtain spatial routing enhanced features; Specifically, it includes: The cue vector is mapped through a linear layer and then broadcast to a spatial size with the same spatial dimensions as the shallow features to obtain a spatial cue feature map. The spatial cue feature map is concatenated with the shallow features along the channel dimension to obtain the intermediate features; For each spatial location on the intermediate feature, calculate its corresponding set of expert weight distributions; Based on the expert weight distribution, select the top K expert networks with the highest weights for each spatial location; The intermediate features of each spatial location are input into the corresponding selected expert network for calculation, and the outputs of each expert network are weighted and fused according to the expert weights to obtain the spatial routing enhancement features.

4. The variable aperture bokeh rendering method according to claim 3, characterized in that, The depth map is used to divide the field region, and the spatial routing enhancement features are subjected to hierarchical sparse attention calculation and fusion to obtain hierarchical attention enhancement features; Specifically, it includes: The depth map is normalized and preprocessed with bilateral filtering to obtain a preprocessed depth map; Based on the preprocessed depth map, the focal regions in the preprocessed depth map are determined by saliency detection and connected component analysis, and the average depth of the focal regions is calculated as the focal depth. Calculate the cumulative depth distribution function of the preprocessed depth map, and determine the initial threshold for dividing the depth area based on the cumulative depth distribution function; The initial threshold is translated and corrected based on the focal depth to obtain the corrected threshold. Based on the corrected threshold, a focal region mask, a transition region mask, and a background region mask are generated, and the pixel ratio of each region is calculated. The spatial routing enhancement features are transformed by a feedforward network and then split into a query matrix, a key matrix, and a value matrix. Using the focal region mask, transition region mask, and background region mask, the query matrix, key matrix, and value matrix are respectively isolated to their corresponding depth regions; For each depth region, calculate the attention weight matrix within that region, and for each row of the attention weight matrix, retain only the weights corresponding to the top K key vectors with the highest weight values, and reset the remaining weights to zero to obtain sparse attention weights. The sparse attention weights are normalized, and the normalized result is multiplied by the corresponding value vector to obtain the sparse attention output for each region. Based on the pixel ratio and learnable weight coefficients of each region, the sparse attention outputs of each region are weighted and fused, and the fused output is transformed by a convolutional layer and added to the spatial routing enhancement feature to obtain the hierarchical attention enhancement feature.

5. The variable aperture bokeh rendering method according to claim 4, characterized in that, The hierarchical attention enhancement features are input into the encoder for downsampling. During the downsampling process, skip connection features are generated and saved, resulting in deep coding features; specifically including: The hierarchical attention enhancement features are input into the encoder for downsampling to obtain the first-scale deep coding features. The first-scale deep coding features are input into the encoder for downsampling to obtain the second-scale deep coding features. The hierarchical attention enhancement features, the first-scale deep coding features, and the second-scale deep coding features are respectively used as the first skip connection features, the second skip connection features, and the third skip connection features; the first skip connection features, the second skip connection features, and the third skip connection features are saved; The second-scale deep coding features are used as input bottleneck layers for the deep coding features.

6. The variable aperture bokeh rendering method according to claim 5, characterized in that, In the bottleneck layer of the neural network, the cue vector is mapped to channel gating coefficients, and the deep coding features are globally modulated to obtain channel routing enhancement features; Specifically, it includes: The cue vector is input into a linear layer for dimensionality transformation, and the transformed output is mapped to a Sigmoid activation function. The channel gating coefficient is obtained from the interval; The channel gating coefficients are broadcast to the same spatial size as the deep coding features to obtain a channel gating coefficient map; The channel gating coefficient map is multiplied element-wise with the deep coding features to complete the global modulation of the feature channels and obtain the channel routing enhancement features.

7. The variable aperture bokeh rendering method according to claim 6, characterized in that, In the decoder stage of the neural network, the channel routing enhancement features are upsampled and concatenated with the saved skip connection features to obtain concatenated features; specifically including: The channel routing enhancement features are subjected to a first upsampling process to obtain the first upsampled features; The first upsampled feature and the third skip connection feature are concatenated in the channel dimension to obtain the first concatenated feature; The first spliced ​​feature is subjected to a second upsampling process to obtain the second upsampled feature; The second upsampled feature and the second skip connection feature are concatenated along the channel dimension to obtain the second concatenated feature; The second spliced ​​feature is then subjected to a third upsampling process to obtain the third upsampled feature; The third upsampling feature is concatenated with the first skip connection feature in the channel dimension to obtain the third concatenated feature.

8. The variable aperture bokeh rendering method according to claim 7, characterized in that, The concatenated features are channel-modulated using the cue vector, and hierarchical sparse attention computation is applied to obtain the final decoded features; Specifically, it includes: Based on the depth map and target aperture parameters, the adaptive routing prompting module regenerates the prompting vector; The regenerated cue vector is mapped to channel gating coefficients and multiplied element-wise with the first concatenation feature to obtain the first modulation feature; The first modulation feature, the corresponding depth-of-field mask, and the pixel ratio of the region are used to perform hierarchical sparse attention calculation to obtain the first attention enhancement feature; wherein, the depth-of-field mask includes a focal region mask, a transition region mask, and a background region mask. The first attention-enhanced feature is input into the decoder for processing to obtain the first decoded output feature; The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the second concatenation feature to obtain the second modulation feature. The second modulation feature is then input into the decoder for processing to obtain the second decoding output feature. The channel gating coefficients mapped by the regenerated cue vector are multiplied element-wise with the third concatenation feature to obtain the third modulation feature. The third modulation feature is then input into the decoder for processing to obtain the third decoding output feature. The third decoding output feature is the final decoding feature.

9. A variable aperture bokeh rendering method according to claim 8, characterized in that, The final decoded features are mapped to a bokeh residual image, which is then fused with the fully focused input image to output a target aperture bokeh image; specifically including: The final decoded features are mapped through a 3x3 convolutional layer to convert their channel count to 3, resulting in a bokeh residual image. The pixel-level residuals of the bokeh residual image and the fully focused input image are added together to obtain the final target aperture bokeh image.

10. A variable aperture bokeh rendering system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the variable aperture bokeh rendering method as described in any one of claims 1 to 9.

Images