Image light spot editing method and device based on shape condition splicing

Abstract

The invention discloses an image light spot editing method and device based on shape condition splicing. According to the method, improvement and fine adjustment are carried out on the basis of a pre-training image editing model, a light spot shape condition is introduced, and embedded information of the light spot shape condition is spliced behind a noise image token to serve as additional input to guide the model to learn shape features of light spots. In the fine adjustment process, a low-rank adaptation method is adopted to reduce training overhead and improve adaptation efficiency, and meanwhile, the problem of offset of rotation position coding is considered in training, so that artifacts possibly generated by image generation are eliminated. The method can effectively improve the precision and flexibility of light spot shape editing, and has good expandability and practical value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision information technology, and particularly relates to an image spot editing method and apparatus based on shape condition stitching. Background Technology

[0002] With the rapid development of computer vision and image generation technologies, image editing has become an important research and application area. Among these technologies, bokeh, an optical blurring effect determined by lens aperture shape and shooting parameters, is frequently used for enhancing and artisticizing photographic images. However, after shooting, users often find it difficult to directly change the shape and distribution of bokeh in the image, which significantly limits its practical application.

[0003] Existing spot editing methods mainly fall into two categories: The first is traditional image processing-based methods, such as convolutional filtering which achieves local blurring through weighted neighboring pixels, but the generated spot shapes are too uniform; morphological transformations can adjust spot contours on binary masks, but are difficult to integrate with natural image details; kernel-based blurring methods can simulate optical defocusing to some extent, but the spot shape depends on a predefined kernel, lacking flexibility. The second category is deep learning-based generative methods, especially the diffusion models that have emerged in recent years, which have demonstrated strong capabilities in image generation and conditional editing. However, research on spot shape editing is still relatively scarce. Existing general image editing models often suffer from the following problems when applied to spot editing: first, the number of spots in the generated result is inconsistent with the source image, making it difficult to guarantee the accuracy of spot distribution; second, the spot shape is difficult to strictly conform to given conditions, often resulting in the inability to generate the specified shape or only partially edited spots; third, when editing spot while maintaining image structure and semantic consistency, artifacts or detail distortion are easily introduced.

[0004] Therefore, existing image processing and deep learning-based editing methods are insufficient to meet the requirements for precise editing of light spot shapes. There is an urgent need for a technical solution that can achieve controllable and flexible editing of light spot shapes while maintaining the consistency of image structure and semantics. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of current image spot editing methods by proposing an image spot editing method and apparatus based on shape condition stitching.

[0006] The objective of this invention is achieved through the following technical solution: an image spot editing method based on shape condition stitching, the method comprising: Acquire the image dataset for spot editing and preprocess it; Building an image editing model: The source image, target image, spot shape image, and editing instructions in the dataset are encoded. During the encoding process, the target image is noise-added with random noise to form a joint token sequence. A 3D rotation position code is generated for each token in the sequence. The query vector and key vector in the attention mechanism of the Transformer backbone network are rotated. The editing instructions are processed to obtain the average text embedding. The joint token sequence and the average text embedding are input into the Transformer backbone network. The target image token noise is truncated from the obtained prediction noise, and the target image token is updated based on the truncated noise. The image editing model is trained by calculating the difference between the actual noise addition speed and the prediction speed of the target image token. Backpropagation training is performed using stochastic gradient descent and low-rank adaptation methods to obtain the trained image editing model. The initial input is a random noise target image, which is then used as the input to construct an image editing model. The trained image editing model removes noise step by step through iterative denoising sampling to obtain an updated target image token, which is then decoded to obtain an image that meets the constraints of the light spot shape.

[0007] Furthermore, the input to the image editing model is: For a set of images in the same scene, two images with the same lens parameters but different bokeh shapes are randomly selected. One of them is designated as the source image and the other as the target image. The bokeh shape image corresponding to the target image and the editing instructions are combined to form a quadruple data pair. The same lens parameters are specifically: the lens aperture size is the same and the focus depth is the same.

[0008] Furthermore, the image editing model includes a VAE image encoder and decoder, a T5 text encoder, a CLIP text encoder, and a Transformer backbone network.

[0009] Further, the input joint token sequence includes: passing the target image, source image, and spot shape image through a VAE image encoder to obtain corresponding two-dimensional image codes, and further flattening the two-dimensional image codes into a one-dimensional image token sequence; passing the editing instructions through a T5 text encoder to obtain text tokens; sampling random noise from a standard Gaussian distribution to add noise to the target image token; and concatenating the text token, the noisy target image token, the source image token, and the spot shape image token along the token length dimension to obtain the input joint token sequence.

[0010] Furthermore, the three-dimensional rotational position encoding includes: The three-dimensional position of the text token is fixed at [0,0,0]; Set the 3D position of the noise token in the target image to , where i is the index of the token in the height direction of the original two-dimensional image encoding, and j is the index of the token in the width direction of the original two-dimensional image encoding; Set the 3D position of the source image token to Where i is the index of the token in the height direction of the original 2D image encoding, and j is the index of the token in the width direction of the original 2D image encoding. and These represent the number of tokens encoded in the target image along the height and width directions, respectively. Set the 3D position of the light spot shape image token to , where i is the index of the token in the height direction of the original 2D image encoding, and j is the index of the token in the width direction of the original 2D image encoding. and These represent the number of tokens encoded in the source image along the height and width directions, respectively. Generate a 3D rotational position code for each token using its 3D position.

[0011] Furthermore, the rotation transformation of the query vector and key vector in the Transformer backbone network attention mechanism includes: by rotating the query vector and key vector, the attention mechanism can explicitly perceive the relative position information of the token in three-dimensional space when calculating similarity, thereby realizing the modeling of the three-dimensional spatial structural relationship. For a token with three-dimensional coordinates [t, h, w], the specific implementation of the rotation transformation is as follows: Where q is the query vector of the token before the rotation transformation, and k is the key vector of the token before the rotation transformation. Encoding of three-dimensional rotational position.

[0012] Furthermore, the prediction speed is the prediction speed of the image editing model for the target token after inputting the quadruple data pair; the actual noise addition speed is the difference between the sampled random noise and the target image token; when calculating the difference, the L2 loss function is used, the derivative of the loss function with respect to the low-rank matrix is ​​calculated through the PyTorch automatic differentiation mechanism, and the stochastic gradient descent method is used for optimization and update.

[0013] Furthermore, the low-rank adaptation method specifically involves: freezing the model parameters; constructing two learnable low-rank matrices for the projection layers of the query matrix, the key matrix, and the value matrix in the Transformer backbone network; using the product of these two matrices as the update amount for the model weights; and updating the low-rank matrices during gradient descent.

[0014] Furthermore, the iterative noise reduction sampling includes: at time step Input the joint token sequence into the image editing model to obtain the predicted instantaneous velocity of the target image noise token; update the current target image token according to the predicted instantaneous velocity; repeat the above process until sampling is completed, and input the target image token into the VAE decoder to obtain the target image.

[0015] On the other hand, the present invention also provides an image spot editing device based on shape condition stitching, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the image spot editing method based on shape condition stitching.

[0016] On the other hand, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned image spot editing method based on shape condition stitching.

[0017] The beneficial effects of this invention are: 1. This invention improves the accuracy and flexibility of spot editing. By introducing spot shape conditions and stitching them after the noisy image token, and combining them with the attention mechanism of the diffusion model, the generated spot can accurately match the reference shape conditions, achieving high-precision and controllable spot shape editing effects.

[0018] 2. This invention can maintain the structural and semantic consistency of the source image during spot editing. By constructing a joint token input sequence of the source image, target image, and spot shape image, and adjusting the position encoding, combined with a low-rank adaptation method to train the diffusion model, artifacts and detail distortion that may occur during the generation process are effectively reduced. Attached Figure Description

[0019] Figure 1 A flowchart for generating a combined token sequence is provided in an embodiment of the present invention; Figure 2 A flowchart for generating a three-dimensional rotation position encoding of a token provided in an embodiment of the present invention; Figure 3 A flowchart of the model training process provided in this embodiment of the invention; Figure 4 This is a schematic diagram of an image spot editing device based on shape condition stitching provided in an embodiment of the present invention. Detailed Implementation

[0020] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0021] like Figure 1 As shown, this invention provides an image spot editing method based on shape condition stitching. To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] First, construct the combined token sequence, the specific steps of which are as follows: Step one involves encoding the source image, target image, and spot shape image, including the following sub-steps: (1) Prepare and import the source image, target image, and spot shape image. Modify the images to a three-channel format and use cropping transformation to change the resolution of the source and target images to 1024*1024 and the resolution of the spot shape image to 160*160. Normalize the RGB values ​​to the range of [-1,1]. Adjust the dimensional information of the image to [number of channels, height, width] to obtain the preprocessed image. For the image set in the same scene, randomly select two images with the same lens parameters but different spot shapes. Designate one as the source image and the other as the target image. Combine the spot shape image corresponding to the target image and the editing instructions to form a quadruple data pair. (2) Define the image editing model and import the pre-trained model parameters, including VAE image encoder, T5 text encoder, CLIP text encoder, and Transformer backbone network.

[0023] (3) The processed source image, target image and spot shape image are encoded using a VAE image encoder to obtain the distribution representation of the image in the encoding space, and the average value of the distribution is selected as the image encoding. The image encoding is expanded from two dimensions to one dimension to obtain a one-dimensional image token sequence.

[0024] Step two: Encode the editing instructions using a T5 text encoder to obtain a text token; then encode the editing instructions again using a CLIP text encoder, and finally pool the results to obtain the average text embedding. The average text embedding is then fed into the Transformer backbone network as a generation condition.

[0025] Step 3: Sample random noise from a standard Gaussian distribution. Sampling time steps from a uniform distribution Use the optimal matching stream to match the target image tokens To add noise, use the following formula: Step four: Concatenate the text token, the target image noise token, the source image token, and the spot shape image token along the length of the token sequence to form a joint token sequence.

[0026] like Figure 2As shown, after constructing the joint token sequence, a three-dimensional rotation position code is generated for each token in the sequence. The specific steps are as follows: Step 1: Generate 3D coordinates for each token, including the following sub-steps: (1) For text tokens, a fixed method is used. coordinate.

[0027] (2) For the target image noise token, use Coordinates, where The two-dimensional coordinates of the token encoded in the original two-dimensional target image.

[0028] (3) For the source image token, adopt Coordinates, where The coordinates of the token in the original two-dimensional source image. , Encode the number of tokens in the height and width directions of the target image respectively.

[0029] (4) For light spot shape image tokens, adopt coordinates, where The coordinates of the token encoded in the original two-dimensional spot shape image. and These represent the number of tokens encoded in the source image along the height and width directions, respectively.

[0030] Step two: Generate a 3D rotation position code for each token based on the aforementioned 3D coordinates. The method for generating the rotation position code is the same as the existing implementation method for rotation position codes. Specifically, assuming the 3D coordinates of the token are set to [t, h, w], generate the rotation position code as follows: in, The internal structure consists of: The 3D rotation position encoding is applied to the query vector and key vector of the corresponding token during subsequent attention calculation. By rotating the query vector and key vector, the attention mechanism can explicitly perceive the relative position information of the token in 3D space when calculating similarity, thereby achieving modeling of the 3D spatial structural relationships. For a token with 3D coordinates [t, h, w], the specific implementation of the rotation transformation is as follows: Where q is the query vector of the token before the rotation transformation is applied, and k is the key vector of the token before the rotation transformation is applied.

[0031] like Figure 3As shown, the joint token sequence is input into the Transformer backbone network, and the loss function is calculated based on the model output to optimize the model parameters, including the following sub-steps: (1) Freeze all parameters of the original model, including the T5 text encoder, CLIP text encoder, VAE image encoder, and Transformer backbone network. For the neural network layers that need to be trained, including the projection layers of the query matrix, key matrix, and value matrix in the Transformer backbone network, the parameter matrix size is... Construct two learnable low-rank matrices. , Their product is used as the update amount for the neural network layer parameters, and the low-rank matrix is ​​updated during gradient descent.

[0032] (2) The joint token sequence is input into the Transformer backbone network to obtain the corresponding prediction noise; wherein, only the prediction speed of the target image noisy token is used for the calculation of the loss function, and the output of the other tokens does not participate in the loss calculation. The loss function is L2 loss, which calculates the difference between the prediction speed of the target image noisy token and the true speed. The true speed is obtained by the difference between the sampled random noise and the target image token, and the specific formula is as follows: The derivative of the loss function with respect to the low-rank matrix is ​​calculated using PyTorch's automatic differentiation mechanism. Stochastic gradient descent is then used to optimize the update, with the optimizer set to Adam and the learning rate to 5e-5.

[0033] After the image editing model is trained, the shape of the light spot can be edited in the image to be processed, including the following sub-steps: Step 1: Use the image to be processed as the source image, and sample random noise from the standard Gaussian distribution as the initial input of the target image. Combine the preset spot shape image and the corresponding editing instructions to construct a quadruple data pair input. Step 2: In multiple preset time steps, the quadruple data pairs are input into the trained image editing model to obtain the prediction speed of the target image noise token.

[0034] Step 3: The current target image token is iteratively updated based on the predicted noise, thus achieving noise removal. Specifically, a standard diffusion model time-step scheduling strategy is used to discretize the continuous time interval into several time steps: Let the current time be The corresponding target image token is represented as The noise predicted by the Transformer backbone model is Then, the ordinary differential equation is discretized using numerical integration, resulting in the following updated formula: Step 4: Repeat the above process until sampling is complete, decode the target image token to obtain the target image that meets the light spot shape condition constraint.

[0035] Corresponding to the aforementioned embodiment of an image spot editing method based on shape condition stitching, the present invention also provides an embodiment of an image spot editing device based on shape condition stitching.

[0036] See Figure 4 The present invention provides an image spot editing device based on shape condition stitching, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an image spot editing method based on shape condition stitching in the above embodiment.

[0037] An embodiment of an image spot editing device based on shape condition stitching provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the image spot editing device based on shape condition stitching provided by the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0038] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0039] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0040] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements an image spot editing method based on shape-condition stitching as described in the above embodiments.

[0041] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0042] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned image spot editing method based on shape condition stitching.

[0043] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0044] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for editing image light spots based on shape-condition stitching, characterized in that, The method includes: Acquire the image dataset for spot editing and preprocess it; Building an image editing model: The source image, target image, spot shape image, and editing instructions in the dataset are encoded. During the encoding process, the target image is noise-added with random noise to form a joint token sequence. A 3D rotation position code is generated for each token in the sequence. The query vector and key vector in the attention mechanism of the Transformer backbone network are rotated. The editing instructions are processed to obtain the average text embedding. The joint token sequence and the average text embedding are input into the Transformer backbone network. The target image token noise is truncated from the obtained prediction noise, and the target image token is updated based on the truncated noise. The image editing model is trained by calculating the difference between the actual noise addition speed and the prediction speed of the target image token. Backpropagation training is performed using stochastic gradient descent and low-rank adaptation methods to obtain the trained image editing model. The initial input is a random noise target image, which is then used as the input to construct an image editing model. The trained image editing model removes noise step by step through iterative denoising sampling to obtain an updated target image token, which is then decoded to obtain an image that meets the constraints of the light spot shape.

2. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The input to the image editing model is: For a set of images in the same scene, two images with the same lens parameters but different bokeh shapes are randomly selected. One of them is designated as the source image and the other as the target image. The bokeh shape image corresponding to the target image and the editing instructions are combined to form a quadruple data pair. The same lens parameters are specifically: the lens aperture size is the same and the focus depth is the same.

3. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The image editing model includes a VAE image encoder and decoder, a T5 text encoder, a CLIP text encoder, and a Transformer backbone network.

4. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The input joint token sequence includes: passing the target image, source image, and spot shape image through a VAE image encoder to obtain corresponding two-dimensional image codes, and further flattening the two-dimensional image codes into a one-dimensional image token sequence; passing the editing instructions through a T5 text encoder to obtain text tokens; sampling random noise from a standard Gaussian distribution to add noise to the target image token; and concatenating the text token, the noisy target image token, the source image token, and the spot shape image token along the token length dimension to obtain the input joint token sequence.

5. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The three-dimensional rotational position encoding includes: The three-dimensional position of the text token is fixed at [0,0,0]; Set the 3D position of the noise token in the target image to , where i is the index of the token in the height direction of the original two-dimensional image encoding, and j is the index of the token in the width direction of the original two-dimensional image encoding; Set the 3D position of the source image token to Where i is the index of the token in the height direction of the original 2D image encoding, and j is the index of the token in the width direction of the original 2D image encoding. and These represent the number of tokens encoded in the target image along the height and width directions, respectively. Set the 3D position of the light spot shape image token to , where i is the index of the token in the height direction of the original 2D image encoding, and j is the index of the token in the width direction of the original 2D image encoding. and These represent the number of tokens encoded in the source image along the height and width directions, respectively. Generate a 3D rotational position code for each token using its 3D position.

6. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The rotation transformation of the query vector and key vector in the Transformer backbone network attention mechanism includes: by rotating the query vector and key vector, the attention mechanism can explicitly perceive the relative position information of the token in three-dimensional space when calculating similarity, thereby realizing the modeling of the three-dimensional spatial structural relationship. For a token with three-dimensional coordinates [t, h, w], the specific implementation of the rotation transformation is as follows: Where q is the query vector of the token before the rotation transformation, and k is the key vector of the token before the rotation transformation. Encoding of three-dimensional rotational position.

7. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The prediction speed is the speed at which the image editing model predicts the target token after inputting a quadruple data pair; the actual noise addition speed is the difference between the sampled random noise and the target image token. The L2 loss function is used to calculate the difference. The derivative of the loss function with respect to the low-rank matrix is ​​calculated through PyTorch's automatic differentiation mechanism, and the update is optimized using stochastic gradient descent.

8. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The low-rank adaptation method specifically involves: freezing the model parameters; constructing two learnable low-rank matrices for the projection layers of the query matrix, key matrix, and value matrix in the Transformer backbone network; using the product of these two matrices as the update amount for the model weights; and updating the low-rank matrices during gradient descent.

9. The image spot editing method based on shape condition stitching according to claim 1, characterized in that, The iterative noise reduction sampling includes: at time step Input the joint token sequence into the image editing model to obtain the predicted instantaneous velocity of the target image noise token; update the current target image token according to the predicted instantaneous velocity; repeat the above process until sampling is completed, and input the target image token into the VAE decoder to obtain the target image.

10. An image spot editing device based on shape condition stitching, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an image spot editing method based on shape condition stitching as described in any one of claims 1-9.

Images