A text-based scene picture editing method and system

Abstract

The application provides a text-based scene picture editing method and system, object information and position information in a text instruction are converted into a visual space mask in a scene, so that accurate positions needing editing are determined; a received text instruction and multiple targets in a scene needing editing are subjected to multi-modal alignment, and the multi-modal alignment is combined with the space mask to generate the edited scene, so that multiple target objects needing editing in the scene picture can be processed at one time according to the text instruction, and the processing efficiency is high.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a text-based method and system for editing scene images. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the process of constructing real-world scenes or creating art, it may be necessary to make appropriate construction and adjustments to the layout or design. Currently, most text-based scene image editing methods use simple text to edit a single target in the scene image. Some methods also use an iterative approach to edit multiple targets sequentially, but this iterative approach is less efficient, and multiple editing operations may cause problems such as ghosting or blurring at the edges of areas that have been edited multiple times.

[0004] Existing technologies also suffer from inaccuracies in recognizing multiple objects in a scene and incorrect editing operations. Furthermore, there are currently few methods for simultaneously editing multiple targets in a scene image based on complex text instructions. Therefore, how to solve the problem of simultaneously editing multiple targets based on complex text instructions, improve editing efficiency, and lower the barrier to entry for scene editing is an urgent issue to be addressed. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a text-based scene image editing method and system, which can process multiple target objects that need to be edited in a scene image at one time according to text instructions, without the need for iteration, and has high processing efficiency.

[0006] To achieve the above objectives, a first aspect of the present invention provides: a text-based scene image editing method, comprising:

[0007] Obtain text instructions and the scene to be edited;

[0008] The text feature vector is obtained according to the text instruction, and the object information and position information in the text feature vector are converted into a spatial mask.

[0009] A scene feature map is obtained based on the scene to be edited, and the scene feature map is fused with the text feature vector to obtain a visual feature map;

[0010] The visual feature map is combined with the spatial mask to obtain the edited scene.

[0011] A second aspect of the present invention provides a text-based scene image editing system, comprising:

[0012] Acquisition Unit: Acquires text instructions and the scene to be edited;

[0013] Text instruction processing unit: Obtains text feature vectors based on the text instructions, and converts the object information and position information in the text feature vectors into spatial masks;

[0014] Scene processing unit: Obtains a scene feature map based on the scene to be edited, and fuses the scene feature map with the text feature vector to obtain a visual feature map;

[0015] Fusion unit: Combines the visual feature map with the spatial mask to obtain the edited scene.

[0016] A third aspect of the present invention provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a text-based scene image editing method is performed.

[0017] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a text-based scene image editing method.

[0018] The above one or more technical solutions have the following beneficial effects:

[0019] In this invention, by converting object information and position information in text instructions into a visual spatial mask in the scene, all the positions that need to be edited can be accurately determined. By performing multimodal alignment of the received text instructions and multiple targets in the scene to be edited, and combining them with the spatial mask, the edited scene is generated. This allows multiple target objects that need to be edited in the scene image to be processed at once according to the text instructions, without the need for iteration, resulting in high processing efficiency. It specifically solves problems such as object deformation and editing errors in the generated image, thereby improving the quality of the generated edited image.

[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1This is a flowchart of the text-based scene image editing method provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a simplified schematic diagram of the text-based scene image editing method provided in Embodiment 1 of the present invention;

[0024] Figure 3 This is a network structure diagram of the text-based scene image editing method provided in Embodiment 1 of the present invention;

[0025] Figure 4 This is a network structure diagram of the multi-objective detail consistency stage provided in Embodiment 1 of the present invention;

[0026] Figure 5 This is a comparison diagram of the qualitative results of the method provided in Embodiment 1 of the present invention and existing methods. Detailed Implementation

[0027] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0029] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0030] Example 1

[0031] like Figures 1-3 As shown, this embodiment discloses a text-based scene image editing method, including:

[0032] Step 1: Obtain the text instructions and the scene to be edited;

[0033] Step 2: Obtain the text feature vector according to the text instructions, and convert the object information and position information in the text feature vector into a spatial mask;

[0034] Step 3: Obtain a scene feature map based on the scene to be edited, and fuse the scene feature map with the text feature vector to obtain a visual feature map;

[0035] Step 4: Combine the visual feature map with the spatial mask to obtain the edited scene.

[0036] This embodiment inputs text instructions and the scene to be edited into a trained scene editing generation model, and outputs the scene edited according to the text instructions. The scene editing generation model includes two stages: spatial semantic enhancement and multi-objective detail consistency.

[0037] In step 2 of this embodiment, text instructions are obtained, and natural language processing techniques are used to encode the text instructions to obtain feature vectors. For example, a BERT pre-trained model is used to encode the text instructions into text feature vectors.

[0038] In the spatial semantic enhancement stage, a spatial semantic enhancement module is constructed to convert object information and location information in the text feature vector into spatial masks.

[0039] The spatial semantic enhancement module includes a first selection module, a first attention module, a second selection module, a second attention module, a third selection module, a first convolution module, a fourth selection module, a second convolution module, a fifth selection module, a third convolution module, a sixth selection module, and a fourth convolution module, which are connected in sequence.

[0040] Specifically, the input to the first selection module is randomly sampled Gaussian noise with dimensions 2×32×64×64 and a text feature vector with dimensions 2×512. The input to other selection modules is the intermediate hidden feature v output by the previous module, which has dimensions of 2×32×64×64. i and text feature vectors with dimensions of 2×512

[0041] The specific steps for selecting a module are as follows:

[0042]

[0043] Among them, W i For text feature vectors After passing through a linear layer and a ReLU activation function layer, the weights are transformed into weights with a dimension of 2×32×64×64; b i For text feature vectors The layer passes through a linear layer and a ReLU activation function layer, transforming it into a bias layer with dimensions of 2×32×64×64. Multiplication operations then transform the intermediate hidden features v... i Based on the reweighted text feature vector For the purpose of region selection, it helps the model accurately identify the location of the desired image portion that matches the given text. The bias term helps the model stably match text-related content. i+1 Select the output of the i-th block, and the dimension of the output feature is 2×32×64×64.

[0044] The input to the attention module is the intermediate hidden feature v with dimensions 2×32×64×64 output from the previous module. i and text feature vectors with dimensions of 2×512 The specific operation of the attention module is as follows:

[0045]

[0046] Where Q is The hidden feature v is obtained after passing through a linear layer. i K was obtained by using two convolutional layers, each with 32 input and 32 output channels. i and V i c is the number of channels, L proj It is a linear layer.

[0047] The input to the convolutional module is the intermediate hidden feature v with dimensions 2×32×64×64 output from the previous module. i The convolutional module has a kernel size of 1×1, a stride of (1,1), and padding of 0, with an output size of 2×32×64×64. The output of the fourth convolutional module is the spatial mask M.

[0048] In step 3 of this embodiment, the scene to be edited is obtained, and the scene is input in the form of an image. Image processing technology is used to crop and convert the format of the scene to be edited, and it is stored as a 256*256 resolution .png format image. An image encoder is constructed to encode the input scene image to be edited.

[0049] Specifically, the image encoder consists of three downsampling blocks. In the first downsampling block, the input image to be edited is 3×256×256 in size. The structure of the first downsampling block consists of a boundary reflection padding ReflectionPad2d layer, a 2D instance normalization InstanceNorm2d layer, a ReLU activation function layer, and a convolutional layer, connected in sequence.

[0050] The specific configuration includes the ReLU activation function (inplace = True), a 7×7 kernel size for the convolutional layer, a stride of (1, 1), and padding of 3. The output of the first downsampled block is an intermediate image feature of size 2×64×256×256. Here, 2 represents the number of images processed by the image encoder in each batch.

[0051] The input to the second downsampling block is the 2×64×256×256 image features output by the first downsampling block. The second downsampling block also includes a sequentially connected ReflectionPad2d layer, a 2D instance normalization layer, a ReLU activation layer, and a convolutional layer. Specifically, the activation function is ReLU (inplace = True), the convolutional layer has a 4×4 kernel size, a stride of (2, 2), and padding of 1, outputting intermediate image features of size 2×128×128×128.

[0052] The input to the third downsampling block is the 2×128×128×128 image features output by the second downsampling block. The third downsampling block has a basically the same structure as the second downsampling block, except that the convolutional layer has 128 input channels, 256 output channels, and an output size of 2×256×64×64 scene features.

[0053] like Figure 4 As shown, in the multi-target detail consistency stage, conditional enhancement techniques are used to enhance the text feature vectors. standard deviation Noise multiplied by a standard Gaussian distribution ∈ t Then add text feature vectors The average value is used to obtain the conditional text features.

[0054] To enhance the semantic understanding of text features, scaled dot product attention is used to enhance keywords in the text features. in, For conditional text features, W q W k W v This is the linear weight matrix that needs to be learned during training. Before training begins, W... q W k W v Random initialization will be performed, and gradient updates will be performed through backpropagation based on the loss function during training, eventually learning appropriate weight values.

[0055] The first text feature h that is of interest is obtained using the following formula:

[0056]

[0057] Where d is the dimension of the output, and its size is 512.

[0058] The first text feature h is expanded to 2×32×64×64 and combined with the scene image features before editing. The input and output channels are concatenated, and then convolutional operations are performed with 288 input channels and 256 output channels to generate a second text feature h with dimensions of 2×256×64×64. c .

[0059] Second text feature h c The modulation parameters are modeled using adaptive learning to establish the model: β = W1h c γ=W2h c W1 and W2 are learnable convolutional filters. Text features h c The text contains positional information and corresponding operational information for different objects. W1 and W2 learn different instruction information from the text features, match the corresponding operation instructions based on the approximate target location, and generate two modulation parameters γ and β. These two modulation parameters contain information on how to operate on the image features. During training, gradient updates are performed through backpropagation based on the loss function, and W1 and W2 are eventually able to learn appropriate weight values.

[0060] The first modulation parameter γ and the second modulation parameter β are obtained by scaling and shifting the visual feature map of the scene image. ( It is based on scene features The text and image features are fused by sequentially passing through sampling, convolution, and fully connected layers.

[0061]

[0062] Where ⊙ represents the dot product of elements. This represents the edited visual feature map.

[0063] In step 4 of this embodiment, the edited visual feature map Combined with the spatial mask M generated in the spatial semantic enhancement stage, the original scene image is edited and modified. The specific operations are as follows:

[0064]

[0065] Where ⊙ represents the dot product of elements. This represents the edited visual feature map, where M represents the spatial mask generated during the spatial semantic enhancement stage. The scene feature map output by the image encoder before editing, and the final result This is a visual feature map of the scene after editing based on the text.

[0066] In this implementation, a generative adversarial network (GAN) was also constructed, which consists of a generator and a discriminator.

[0067] The input to generator G is a scene visual feature map after editing based on text. Its dimensions are 2×256×64×64, and after passing through the generator, the output image features have dimensions of 2×3×256×256.

[0068] The generator G consists of a ResBlocks residual block layer, an Upsample first upsampling layer, a Conv2dBlock first convolutional layer, an Upsample second upsampling layer, a Conv2dBlock second convolutional layer, and a Conv2dBlock third convolutional layer, all connected in sequence. The ResBlocks residual block layer contains two ResBlock blocks with identical structures and parameters. Each ResBlock block contains two convolutional structures. The first convolutional block structure consists of a ReflectionPad2d layer (boundary reflection padding), an InstanceNorm2d layer (2D instance normalization), a ReLU activation function layer, and a convolutional layer, all connected in sequence. Specifically, the activation function is ReLU (inplace = True), the convolutional kernel size is 3×3, the stride is (1, 1), the padding is 1, and both the input and output channels are 256.

[0069] The second convolutional block consists of a boundary reflection padding (ReflectionPad2d) layer, a 2D instance normalization (InstanceNorm2d) layer, and a convolutional layer connected in sequence. Specific parameters are: a 3×3 kernel size, a stride of (1,1), padding of 1, and 256 input and output channels.

[0070] The specific parameters for the first upsampling layer of Upsample are scale_factor = 2.0 and mode = nearest.

[0071] The structure of the first convolutional layer of Conv2dBlock consists of a ReflectionPad2d layer for boundary reflection padding, a LayerNorm layer for normalization, a ReLU activation function layer, and a convolutional layer connected in sequence. Specifically, the activation function is ReLU (inplace = True), the convolutional kernel size is 5×5, the stride is (1, 1), the padding is 2, the input channels are 256, and the output channels are 128.

[0072] The specific parameters for the second upsampling layer of Upsample are scale_factor = 2.0 and mode = nearest.

[0073] The second convolutional layer of Conv2dBlock consists of a ReflectionPad2d layer (boundary reflection padding), a LayerNorm layer (normalization), a ReLU activation function layer, and a convolutional layer, all connected in sequence. Specifically, the activation function is ReLU (inplace = True), the kernel size is 5×5, the stride is (1, 1), the padding is 2, the input channels are 128, and the output channels are 64.

[0074] The third convolutional layer of Conv2dBlock consists of a series of concatenated boundary reflection padding (ReflectionPad2d) layers, a LayerNorm normalization layer, a ReLU activation function layer, and a convolutional layer. Specifically, it uses the Tanh() hyperbolic tangent activation function, a 7×7 kernel size, a stride of (1,1), padding of 3, 64 input channels, and 3 output channels. The generator's final output dimension is 2×3×256×256 edited image features.

[0075] For the discriminator D, a structure similar to PatchGAN is used. The discriminator D structure consists of five convolutional layers, four LeakyReLU(0.2) activation functions, and three instanceNorm2d two-dimensional instance normalization layers. The input channels of each convolutional layer are 3, 64, 128, 256, 512, and the output channels are 64, 128, 256, 512, 1.

[0076] The discriminator takes two inputs: an edited image (2×3×256×256 dimensions) generated by the generator, and an edited real image (also 2×3×256×256 dimensions) from the training process. These are predicted by the discriminator D, which maps the input image features to a 30×30 patch matrix. The mean of all values ​​in the matrix is ​​then calculated, which is the discriminator's final output, a specific numerical value. When the discriminator identifies an image as real, the output value is closer to 1; when it identifies it as fake, the output value is closer to 0. The ultimate training goal of the model is to make the discriminator unable to accurately distinguish between the real and generated images.

[0077] Before training the scene editing generation model, it is necessary to first collect text instructions and corresponding before-and-after scene images to preprocess the sample data and determine the training set, validation set, and test set. Specifically, select text instructions S = [s1, s2, ..., s...]. t ] and the corresponding pre-edit scene X = [x1, x2, ..., x t ] and the edited scene Y = [y1, y2, ..., y tPair the data, where t represents the number of text instructions and scenes in each pair. The text instructions must be explicit, and the scene images must be clear.

[0078] The specific steps for training and building the scene editing and generation model are as follows: using a standard conditional GAN ​​objective, the loss function includes adversarial loss L. GAN A reconstruction loss L L1 Gradient propagation is performed by minimizing the loss, and the parameters that need to be learned in the model are updated in reverse.

[0079] Conditional GANs learn a scene image X before editing to an image after editing. The mapping, G: in, Y represents the edited scene image, and Y represents the edited real scene image.

[0080]

[0081]

[0082] Where D(X,Y) is the value of the discriminator after inputting the image before editing and the real image. Let represent the expectation of log(X,Y) with respect to X and Y; D(X,G(X)) is the discriminator's value after inputting the image before editing and the generated image. First, the logarithm of 1-D(X,G(X)) is calculated, then the expectation of log(1-D(X,G(X))) with respect to X is calculated. ||YG(X)||1 is the L1 norm calculated between the real image Y and the image G(X) generated by the generator, and then its expectation with respect to X and Y is calculated.

[0083] The generator G attempts to minimize this objective against the discriminator D, which tries to maximize it. After generating the edited scene image, an adversarial loss L is predicted. GAN The discriminator evaluates the visual realism and semantic consistency of the input. By distinguishing between generated images and real samples, the discriminator helps the generator produce images with high quality and textual semantic consistency.

[0084] The ultimate goal is:

[0085]

[0086] in, Meaning to make function L cGAN The value of G when the function reaches its minimum (G,D), and the value of D when the function reaches its maximum. λ is the reconstruction loss L. L1 The weights can be set to different values ​​during training.

[0087] After the scene editing generation model is trained, text instructions and a test dataset of scene images to be edited are input into the model to generate corresponding edited scene images, thus testing the model's effectiveness. The qualitative results of the proposed method and other methods on the CoDraw and Multi-Clevr test datasets are as follows: Figure 5 As shown, the generated image resolution is 256×256. The proposed method in this example demonstrates impressive generation capabilities, particularly in spatial semantic information processing and detail consistency, showing superiority over other methods.

[0088] Example 2

[0089] The purpose of this embodiment is to provide a text-based scene image editing system, including:

[0090] Acquisition Unit: Acquires text instructions and the scene to be edited;

[0091] Text instruction processing unit: Obtains text feature vectors based on the text instructions, and converts the object information and position information in the text feature vectors into spatial masks;

[0092] Scene processing unit: Obtains a scene feature map based on the scene to be edited, and fuses the scene feature map with the text feature vector to obtain a visual feature map;

[0093] Fusion unit: Combines the visual feature map with the spatial mask to obtain the edited scene.

[0094] Example 3

[0095] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0096] Example 4

[0097] The purpose of this embodiment is to provide a computer-readable storage medium.

[0098] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0099] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0100] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0101] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A text-based scene image editing method, characterized in that, include: Obtain text instructions and the scene to be edited; The text feature vector is obtained according to the text instruction, and the object information and position information in the text feature vector are converted into a spatial mask. A scene feature map is obtained based on the scene to be edited, and the scene feature map is fused with the text feature vector to obtain a visual feature map; The visual feature map is combined with the spatial mask to obtain the edited scene; The constructed spatial semantic enhancement module converts the object information and location information in the text feature vector into a spatial mask, specifically: The spatial semantic enhancement module includes a first selection module to a sixth selection module connected in sequence. After the first selection module and the second selection module, attention modules are connected respectively. After the third selection module to the sixth selection module, convolution modules are connected respectively. The selection module includes a linear layer and a ReLU activation function layer. The selection module processes the text feature vector and the output feature vector of the previous module respectively and then merges them to obtain the selected feature vector. The attention module processes the text feature vector and the selection feature vector to obtain the attention feature vector; The convolution module processes the text feature vector and the selected feature vector to obtain the final spatial mask; The scene feature map is fused with the text feature vector to obtain a visual feature map, specifically including: The conditional text features are obtained by multiplying the standard deviation of the text feature vector by the noise of the standard Gaussian distribution and then adding it to the average value of the text feature vector. The conditional text features are weighted using a scaled dot product attention calculation method to obtain the first text feature; The first text feature is concatenated with the scene feature map, and then convolved to obtain the second text feature; The second text features are subjected to convolution operations to obtain the first modulation parameter and the second modulation parameter; The first modulation parameter is multiplied by the scene feature map matrix and then added to the corresponding element of the second modulation parameter to obtain the visual feature map.

2. The text-based scene image editing method as described in claim 1, characterized in that, Combining the visual feature map with the spatial mask specifically includes: The visual feature map is multiplied by the spatial mask, and the result is added to the difference between the result and the spatial mask, and then to the result of the dot product operation on the scene feature map to obtain the edited scene image.

3. The text-based scene image editing method as described in claim 1, characterized in that, The text instructions are encoded into text feature vectors using a BERT pre-trained model.

4. The text-based scene image editing method as described in claim 1, characterized in that, The scene to be edited is encoded into a scene feature map using an image encoder. The image encoder includes three downsampling blocks connected in sequence. Each downsampling block includes a boundary reflection filling layer, a two-dimensional instance normalization layer, an activation function layer, and a convolutional layer connected in sequence.

5. The text-based scene image editing method as described in claim 1, characterized in that, It also includes a generative adversarial network, which is trained using a loss function consisting of adversarial loss and reconstruction loss.

6. A text-based scene image editing system, characterized in that, include: Acquisition Unit: Acquires text instructions and the scene to be edited; Text instruction processing unit: Obtains text feature vectors based on the text instructions, and through the constructed spatial semantic enhancement module, specifically: The spatial semantic enhancement module includes a first selection module to a sixth selection module connected in sequence. After the first selection module and the second selection module, attention modules are connected respectively. After the third selection module to the sixth selection module, convolution modules are connected respectively. The selection module includes a linear layer and a ReLU activation function layer. The selection module processes the text feature vector and the output feature vector of the previous module respectively and then merges them to obtain the selected feature vector. The attention module processes the text feature vector and the selection feature vector to obtain the attention feature vector; The convolution module processes the text feature vector and the selected feature vector to obtain the final spatial mask; Scene processing unit: Obtains a scene feature map based on the scene to be edited, and fuses the scene feature map with the text feature vector to obtain a visual feature map, specifically including: The conditional text features are obtained by multiplying the standard deviation of the text feature vector by the noise of the standard Gaussian distribution and then adding it to the average value of the text feature vector. The conditional text features are weighted using a scaled dot product attention calculation method to obtain the first text feature; The first text feature is concatenated with the scene feature map, and then convolved to obtain the second text feature; The second text features are subjected to convolution operations to obtain the first modulation parameter and the second modulation parameter; The first modulation parameter is multiplied by the scene feature map matrix and then added to the corresponding element of the second modulation parameter to obtain the visual feature map; Fusion unit: Combines the visual feature map with the spatial mask to obtain the edited scene.

7. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform a text-based scene image editing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs a text-based scene image editing method as described in any one of claims 1 to 5.

Images