Image editing and fusing method based on structured state space sequence model

Abstract

The invention provides an image editing and fusing method based on a structured state space sequence model, and the method comprises the steps: S1, obtaining at least one input image, and carrying out the time sequence feature extraction of the input image through employing the structured state space sequence model, and obtaining a background feature; s2, background features are separated through mask operation, foreground features output by a conditional generation model are introduced, and latent space feature coding is performed on the background features and the foreground features through a variational auto-encoder to obtain coded background features and coded foreground features; and S3, inputting the encoded background features and the encoded foreground features into a Mamba fusion network for feature fusion to obtain a fusion image, and dynamically adjusting fusion weights of the encoded background features and the encoded foreground features by using a selective scanning mechanism in a fusion process to perform feature space alignment. The method has the beneficial effects that the dynamic feature modeling capability can be realized, the feature fusion accuracy is improved, and the calculation efficiency is improved.

Description

Technical Field

[0001] This invention relates to the technical field of image editing, and more specifically, to an image editing fusion method based on a structured state-space sequence model. Background Technology

[0002] With the rapid development of digital image processing technology, image editing and fusion are increasingly widely used in fields such as film and television special effects, virtual reality, and medical imaging. Traditional methods mainly rely on the following two types of technologies: The first type of traditional methods uses editing techniques based on traditional algorithms (such as Poisson blending and Alpha blending), which achieve foreground embedding by solving partial differential equations in the gradient domain. However, these methods have significant limitations. In complex texture regions (such as hair and flowing water), background features lose details due to the smoothing effect during the global optimization process, resulting in blurring or structural distortion after fusion. Blending functions with fixed weights (such as linear interpolation) are difficult to adapt to changes in local features, producing visible artifacts at the foreground-background boundary. Iteratively solving large-scale linear equation systems is extremely time-consuming, requiring several seconds of delay to process 4K images.

[0003] The second type of technology used in traditional methods is generative models based on deep learning (such as GAN and Diffusion), which use encoder-decoder architecture to achieve end-to-end fusion. However, it still faces challenges. Static network parameters cannot adaptively adjust the fusion strategy according to the characteristics of the input image (such as texture complexity and lighting conditions), resulting in poor cross-scene generalization. Existing convolutional operations are limited by local receptive fields and have difficulty capturing long-range spatiotemporal dependencies in images. Latent vectors generated by encoders such as VAE lack explicit decoupling mechanisms, causing confusion between foreground and background features.

[0004] Recent improvements (such as Attention-GAN) attempt to introduce attention mechanisms to optimize weight allocation, but still have two fundamental flaws: First, while global attention can improve fusion quality, its computational complexity leads to a sharp drop in the processing speed of high-resolution images; second, pure data-driven models ignore the inherent dynamic characteristics of image formation, such as the state evolution process of optical systems, making the generated results lack physical rationality.

[0005] Therefore, there is an urgent need for an image editing and fusion method that combines dynamic feature modeling capabilities, high accuracy in feature fusion, and efficient computation. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to achieve dynamic feature modeling capability, improve feature fusion accuracy, and enhance computational efficiency. To overcome the defects of the above-mentioned existing technologies (or related technologies), this invention provides an image editing and fusion method based on a structured state space sequence model.

[0007] This invention provides an image editing and fusion method based on a structured state-space sequence model, comprising: Step S1: Obtain at least one input image, and use a structured state-space sequence model to extract temporal features from the input image to obtain background features; Step S2: Separate the background features through masking operation, introduce the foreground features output by the conditional generation model, and use a variational autoencoder to encode the latent space features of the background features and the foreground features to obtain the encoded background features and the encoded foreground features. Step S3: Input the encoded background features and the encoded foreground features into the Mamba fusion network to perform feature fusion to obtain a fused image. During the fusion process, a selective scanning mechanism is used to dynamically adjust the fusion weights of the encoded background features and the encoded foreground features to perform feature space alignment.

[0008] Compared with existing technologies, the image editing and fusion method based on a structured state-space sequence model of this invention has the following advantages: This invention introduces a structured state-space sequence model, which enables the dynamic evolution of the input image over time through the state matrix, control matrix, and output matrix of the structured state-space sequence model. This achieves dynamic feature modeling capabilities while preserving spatial dependencies. By dynamically adjusting the fusion weights through a selective scanning mechanism, the transition at the boundary between encoded background features and encoded foreground features becomes natural, reducing outliers in edge gradients and improving the accuracy of feature fusion. Furthermore, the linear complexity of the Mamba fusion network supports efficient feature computation, ensuring computational efficiency.

[0009] In one possible implementation, after acquiring the input image in step S1, the method further includes: The input image is preprocessed by using a Hilbert space-filling curve to convert it into a pseudo-time series preprocessed image, and then the input image is replaced with the preprocessed image.

[0010] Compared with existing technologies, the above-mentioned technical solution can map the input image into a pseudo-temporal signal by Hilbert space filling curve to maintain the spatial continuity of adjacent pixels, so that the structured state space sequence model can effectively capture background features.

[0011] In one possible implementation, in step S1, the preprocessed image is obtained using the following calculation formula: ; in, This refers to the preprocessed image; Indicates the first in the input image Feature vector of 1 pixel ; This represents the total number of pixels in the input image; This represents the feature dimension of each pixel.

[0012] In one possible implementation, in step S1, the input image is provided by an image editing tool or a deep learning generation model.

[0013] In one possible implementation, the process of dynamically adjusting the fusion weights in step S3 includes: Step A1: Spatially align the encoded foreground features with respect to the encoded background features using deformable convolution to obtain aligned foreground features, and generate corresponding query vectors, key vectors, and value vectors for the encoded background features and the aligned foreground features; Step A2: Calculate the pixel-level cosine similarity between the encoded background features and the aligned foreground features; Step A3: Obtain the attention score of each pixel position in the aligned foreground features relative to other positions based on the query vector, the key vector, and the pixel-level cosine similarity. Step A4: For the attention score at each pixel location, calculate the corresponding attention weight by performing a Softmax function on the attention score in the spatial dimension.

[0014] Compared with existing technologies, the above-mentioned technical solution can achieve spatial alignment by compensating for the foreground displacement of encoded foreground features through deformable convolution operations, and improve the accuracy of fused images by calculating pixel-level cosine similarity to quantify the foreground-background matching degree and guide attention weight allocation.

[0015] In one possible implementation, step S3, the process of feature fusion to obtain the fused image, includes: Step B1: Obtain the attention-weighted foreground features based on the value vector and the attention weights at each pixel position; Step B2: The fused image is obtained by fusing the encoded background features and the attention-weighted foreground features through a gating mechanism.

[0016] Compared with existing technologies, the above technical solution can suppress feature conflict regions through gating mechanisms and reduce artifact generation in complex backgrounds.

[0017] In one possible implementation, in step A2, the pixel-level cosine similarity is obtained using the following formula: ; in, Indicates pixel position The corresponding pixel-level cosine similarity; This represents the encoded background features; This represents the aligned foreground features; This represents a deformable convolution operation; This represents the encoded foreground features; This represents the 2D offset field generated by the optical flow prediction network for each pixel; This represents an optical flow prediction network.

[0018] In one possible implementation, in step A3, the attention score of each pixel position relative to other positions is obtained using the following formula: ; in, Indicates pixel position Compared to other positions The attention score; Indicates pixel position The query vector; Indicates other positions The key vector; Indicates matrix transpose; This indicates that the target number of channels needs to be adjusted. Indicates the learnable scaling factor; Indicates other positions The corresponding pixel-level cosine similarity; This represents a 1x1 convolution operation; This represents the encoded background features; This represents the aligned foreground features.

[0019] In one possible implementation, in step B1, the attention-weighted foreground features are obtained using the following calculation formula: ; in, This represents the attention-weighted foreground features; Indicates pixel position Compared to other positions The attention weights; Indicates other positions The value vector; This represents a 1x1 convolution operation; This represents the aligned foreground features; This indicates the target number of channels that needs to be adjusted.

[0020] In one possible implementation, in step B2, the fused image is obtained using the following calculation formula: ; in, Represents the Sigmoid function; This represents a 3x3 convolution operation; This indicates a channel splicing operation; This represents the encoded background features; This represents the attention-weighted foreground features; This represents the fused image; This indicates element-wise multiplication. Attached Figure Description

[0021] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 A detailed flowchart of the dynamic adjustment process for the fusion weights; Figure 3 This is a flowchart illustrating the feature fusion process. Detailed Implementation

[0022] First, those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] See Figure 1 This invention discloses an image editing and fusion method based on a structured state-space sequence model, comprising: Step S1: Obtain at least one input image and use a structured state-space sequence model to extract temporal features from the input image to obtain background features; Step S2: Separate the background features through masking operation, introduce the foreground features output by the conditional generation model, and use a variational autoencoder to encode the latent space features of the background features and the foreground features respectively to obtain the encoded background features and the encoded foreground features. Step S3: Input the encoded background features and encoded foreground features into the Mamba fusion network to perform feature fusion and obtain a fused image. During the fusion process, a selective scanning mechanism is used to dynamically adjust the fusion weights of the encoded background features and encoded foreground features to align the feature space.

[0025] In this embodiment of the invention, the initial input image is provided by traditional image editing tools or deep learning generation models, and may be a grayscale image or a low-resolution image. At this stage, a structured state space sequence model is used to extract dynamic features from the input image. The temporal dynamic evolution process of the input image is constructed through the state matrix, control matrix and output matrix, and key background features are captured.

[0026] In this embodiment of the invention, the time scale parameter can be dynamically adjusted according to the features of the input image in the structured state space sequence model to accurately extract the background features in the input image. By adjusting the time scale parameter, the performance of the structured state space sequence model under different input images can be further improved, ensuring the authenticity and consistency of the extracted background features.

[0027] In this embodiment of the invention, after obtaining the input image in step S1, the method further includes: The input image is preprocessed by using a Hilbert space-filling curve to convert it into a pseudo-time series preprocessed image, and then the input image is replaced with the preprocessed image. Specifically, the preprocessed image is obtained through the following calculation formula: ; in, This represents the preprocessed image; Represents the first in the input image Feature vector of 1 pixel ; This represents the total number of pixels in the input image; This represents the feature dimension of each pixel.

[0028] In this embodiment of the invention, the structured state-space sequence model used in step S1 is described by the following continuous state equations: ; in, Indicates the updated The hidden state vector at time step; Represents a continuous-time variable; express The hidden state vector at time step 1. This represents the hidden state dimension; express Input at any moment; express Output at any moment; Represents the state matrix; Represents the control matrix; Indicates the output matrix; This represents the feedforward matrix, which is usually 0.

[0029] In this embodiment of the invention, the structured state-space sequence model employs a zero-order preserved discretization method to handle discrete time series. The discretized equations are as follows: ; in, Represents the discrete time step. ; Indicates the first Input at discrete time steps; Indicates the first Hidden state vectors at discrete time steps; and Represents the discretized state matrix and the discretized control matrix. and This is a time scale parameter used to control the step size of the discrete time step; Represents the identity matrix.

[0030] See Figure 2The process of dynamically adjusting the fusion weights in step S3 includes: Step A1: Use deformable convolution to spatially align the encoded foreground features with respect to the encoded background features to obtain aligned foreground features, and generate corresponding query vectors, key vectors and value vectors for the encoded background features and aligned foreground features. Step A2: Calculate the pixel-level cosine similarity between the encoded background features and the aligned foreground features; Step A3: Obtain the attention score of each pixel position relative to other positions in the aligned foreground features based on the query vector, key vector, and pixel-level cosine similarity. Step A4: For the attention score at each pixel location, calculate the corresponding attention weight by performing a Softmax function on the attention score in the spatial dimension.

[0031] In this embodiment of the invention, in step A2, the pixel-level cosine similarity is obtained using the following formula: ; in, Indicates pixel position The corresponding pixel-level cosine similarity; Indicates the background features after encoding; Indicates the foreground features after alignment; This represents a deformable convolution operation; Represents the foreground features after encoding; This represents the 2D offset field generated by the optical flow prediction network for each pixel; This represents an optical flow prediction network.

[0032] In this embodiment of the invention, in step A3, the attention score of each pixel position relative to other positions is obtained using the following formula: ; in, Indicates pixel position Compared to other positions Attention score; Indicates pixel position The query vector; Indicates other positions The key vector; Indicates matrix transpose; This indicates that the target number of channels needs to be adjusted. Indicates the learnable scaling factor; Indicates other positions The corresponding pixel-level cosine similarity; This represents a 1x1 convolution operation; Indicates the background features after encoding; This represents the foreground features after alignment.

[0033] In this embodiment of the invention, step A4 involves obtaining the attention weight using the following formula: ; in, Indicates pixel position Compared to other positions Attention weights; Indicates pixel position Compared to other positions Attention score.

[0034] See Figure 3 In step S3, the process of feature fusion to obtain the fused image includes: Step B1: Obtain the attention-weighted foreground features based on the value vector and the attention weights at each pixel position; Step B2: The encoded background features and attention-weighted foreground features are fused using a gating mechanism to obtain a fused image.

[0035] In this embodiment of the invention, in step B1, the attention-weighted foreground features are obtained using the following calculation formula: ; in, This represents the foreground features after attention weighting; Indicates pixel position Compared to other positions Attention weights; Indicates other positions The value vector; This represents a 1x1 convolution operation; Indicates the foreground features after alignment; This indicates the target number of channels that needs to be adjusted.

[0036] In this embodiment of the invention, in step B2, the fused image is obtained using the following calculation formula: ; in, Represents the Sigmoid function; This represents a 3x3 convolution operation; This indicates a channel splicing operation; Indicates the background features after encoding; This represents the foreground features after attention weighting; Represents a fused image; This indicates element-wise multiplication.

[0037] In this embodiment of the invention, to further improve the image fusion effect, the above-mentioned feature fusion process can be applied at multiple scales to enhance the detail preservation capability. The specific process is as follows: Step C1, Constructing the image pyramid: Encoding background features and encoded foreground features Subsampling is performed separately to obtain multi-scale features. ,in Indicates scale; Step C2, at each scale The fusion process is performed independently on each surface, including spatial alignment, similarity calculation, dynamic weight generation, and gated fusion to obtain fused features. ; Step C3: Upsample the fused features at each scale to the original size and sum them: ; Step C4 involves adjusting the number of channels using a 1x1 convolution to obtain the final fused features.

[0038] In the description of this invention, the references to "one embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0039] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image editing fusion method based on a structured state space sequential model, characterized in that, The method comprises the following steps: Step S1, obtaining at least one input image, and extracting background features from the input image by using a structured state space sequence model to perform time sequence feature extraction; Step S2, separating the background features by a mask operation, introducing foreground features output by a conditional generative model, and respectively performing latent space feature coding on the background features and the foreground features by a variational autoencoder to obtain coded background features and coded foreground features; Step S3, inputting the coded background features and the coded foreground features into a Mamba fusion network to perform feature fusion to obtain a fusion image, and dynamically adjusting fusion weights of the coded background features and the coded foreground features in the fusion process by using a selective scanning mechanism to perform feature space alignment.

2. The image editing fusion method based on structured state space sequence model according to claim 1, characterized in that, After obtaining the input image in the step S1, the method further comprises: Preprocessing the input image, converting the input image into a preprocessed image of a pseudo time sequence by using a Hilbert space-filling curve, and replacing the input image with the preprocessed image.

3. The image editing fusion method based on structured state space sequence model according to claim 2, characterized in that, In the step S1, the preprocessed image is obtained by the following calculation formula: ； wherein, denotes the pre-processed image; a feature vector representing a i-th pixel in the input image, ;​ denotes the total number of pixels of the input image; represents the feature dimension of each pixel.

4. The image editing fusion method based on structured state space sequence model according to claim 1, characterized in that, In the step S1, the input image is provided by an image editing tool or a deep learning generative model.

5. The image editing fusion method based on structured state space sequence model according to claim 1, characterized in that, The process of dynamically adjusting the fusion weights in the step S3 comprises: Step A1, performing spatial alignment of the coded foreground features relative to the coded background features by using a deformable convolution to obtain aligned foreground features, and generating corresponding query vectors, key vectors and value vectors for the coded background features and the aligned foreground features; Step A2, calculating a pixel-level cosine similarity between the coded background features and the aligned foreground features; Step A3, obtaining an attention score of each pixel position in the aligned foreground features relative to other positions according to the query vectors, the key vectors and the pixel-level cosine similarity; Step A4, for the attention score of each pixel position, performing Softmax function calculation on the attention score in the spatial dimension to obtain a corresponding attention weight.

6. The image editing fusion method based on structured state space sequence model according to claim 5, characterized in that, In the step S3, the process of feature fusion to obtain the fusion image comprises: Step B1, obtaining an attention-weighted foreground feature according to the value vectors and the attention weights of the pixel positions; Step B2, fusing the coded background features and the attention-weighted foreground features by a gating mechanism to obtain the fusion image.

7. The image editing fusion method based on structured state space sequence model according to claim 5, characterized in that, In the step A2, the pixel-level cosine similarity is obtained by the following calculation formula: ； wherein, representing a pixel position corresponding to the pixel-level cosine similarity; representing said encoded background feature; representing the aligned foreground features; denotes a deformable convolution operation; representing said encoded foreground feature; represents the 2D offset field for each pixel generated by the optical flow prediction network; represents an optical flow prediction network.

8. The image editing fusion method based on structured state space sequence model according to claim 5, characterized in that, In the step A3, the attention score of each pixel position relative to other positions is obtained by the following calculation formula: ； wherein, representing a pixel position relative to other positions the attention score for the position; the query vector representing the pixel position of the pixel position; representing the key vector of the other position ; denotes the matrix transpose; Indicates that the target number of channels needs to be adjusted; denotes a learnable scaling coefficient; representing other locations corresponding to the pixel-level cosine similarity; denotes a 1x1 convolution operation; representing said encoded background feature; representing the aligned foreground features.

9. The image editing fusion method based on structured state space sequence model according to claim 6, characterized in that, In the step B1, the attention-weighted foreground feature is obtained by the following calculation formula: ； wherein, representing the attention weighted foreground features; representing a pixel position relative to other positions the attention weight; representing other positions of the value vector; denotes a 1x1 convolution operation; representing the aligned foreground features; Indicates that the target number of lanes needs to be adjusted.

10. The image editing fusion method based on structured state space sequence model according to claim 6, characterized in that, In the step B2, the fusion image is obtained by the following calculation formula: ； wherein, denotes a sigmoid function; denotes a 3x3 convolution operation; represents a channel concatenation operation; representing said encoded background feature; representing the attention weighted foreground features; representing the fused image; represents element-wise multiplication.

Images