Automatic editing method of spatially transformed images based on camera view domain coding

Abstract

A method for automatically editing spatially transformed images based on camera perspective domain coding, extracting angle codes from background images through a camera perspective domain coding network, extracting foreground image features and background image features through an image feature extraction network, splicing the copied and tiled background image angle perspective domain code with the foreground and background image features to obtain enhanced image features; inputting the enhanced image features into a spatial transformation network to obtain predicted spatial transformation parameters of the foreground image, sampling from the original foreground image through the parameters, obtaining a distorted foreground image using a bilinear difference method, and then synthesizing the distorted foreground image and the input background image to obtain a synthesized image that conforms to a geometric perspective relationship. The present invention uses the distance between the codes of the foreground and background images in the camera perspective domain to measure the similarity of the perspective angles of the two images when they are taken, and after spatial transformation, the foreground image and the background image have similar perspective angles.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to an automatic editing method for space transformation images based on camera viewing angle domain coding. Background Art

[0002] The main shortcoming of existing image synthesis technology is that it cannot eliminate the geometric inconsistency between the foreground and the background, such as the unreasonable size and position of the foreground in the synthesized image, and the inconsistent perspective angles of the foreground and the background. To eliminate the geometric inconsistency of the synthesized image, it is necessary to obtain the necessary information from the background image to guide the adjustment of the foreground image. Traditional methods use prior information about the size and position of the foreground object to adjust the foreground, without fully considering the information of the background image, and the synthesized image is often not realistic enough. The present invention uses the key guiding role of the background image in correcting the foreground perspective angle to reduce the geometric inconsistency between the foreground and the background in the synthesized image. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes an automatic editing method for spatially transformed images based on camera perspective domain coding, which uses the distance between the codings of the foreground and background images in the camera perspective domain to measure the similarity of the perspective angles when the two images are taken, and achieves similar perspective angles for the foreground image and the background image after spatial transformation.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an automatic editing method for spatial transformation images based on camera view domain coding. The method comprises the following steps: extracting angle codes from a background image through a camera view domain coding network, extracting foreground image features and background image features through an image feature extraction network, splicing the copied and tiled background image angle code with the foreground and background image features to obtain enhanced image features; inputting the enhanced image features into a spatial transformation network to obtain predicted spatial transformation parameters of a foreground image, sampling from an original foreground image through the parameters, obtaining a distorted foreground image through a bilinear difference method, and then synthesizing the distorted foreground image and the input background image to obtain a synthesized image that conforms to a geometric perspective relationship.

[0006] The camera perspective domain coding network maps the input image to the camera perspective domain space according to the foreground or background image, and outputs the coding of the image in the perspective domain space, which represents the information of the camera perspective angle when the image is taken. The Euclidean distance between the codings of two images in the perspective domain space can represent the similarity between the perspective angles of the two images.

[0007] The camera viewing angle domain coding network includes: four basic units, an average pooling layer and a convolution layer arranged after the basic units, wherein: the four basic units perform feature extraction processing according to the feature map information of the image to obtain the high-level feature map result of the image, the average pooling layer performs pooling processing according to the feature map information of the image to obtain the result of aggregating important information of the feature map and reducing the amount of calculation, and the convolution layer performs convolution processing according to the image feature map information aggregated by the pooling layer to obtain the encoding of the image in the camera viewing angle domain.

[0008] Each of the basic units includes: a partial convolution layer, an instance normalization layer and a linear rectification unit, wherein: the partial convolution layer performs convolution processing according to the feature map information of the image to obtain the high-level feature map result of the image, the instance normalization layer performs normalization processing according to the feature map information of the image to ensure the independence of the image instance while making the data distribution consistent and accelerating the convergence of the model, and the linear rectification unit performs linear rectification processing according to the normalized image feature map information, introduces nonlinear operations, and enables the neural network to have the ability to fit nonlinear functions.

[0009] The image feature extraction network maps the input image to a feature space according to the foreground or background image, and outputs a feature map of the image containing spatial features.

[0010] The image feature extraction network includes: five basic units connected in series, each basic unit performs feature aggregation processing according to the feature map information of the image to obtain the high-level feature map result of the image, and each basic unit includes a convolution layer and a linear rectification unit, wherein: the convolution layer performs convolution processing according to the feature map information of the image to obtain the high-level feature map result of the image, and the linear rectification unit performs linear rectification processing according to the feature map information of the image, introduces nonlinear operations, and enables the neural network to have the ability to fit nonlinear functions.

[0011] The spatial transformation network includes: a spatial transformation parameter prediction network, a sampling grid generation network and a sampling network, wherein: the spatial transformation parameter prediction network predicts the spatial transformation parameters according to the original foreground image, the foreground background image feature map and the background image camera view domain coding prediction, the sampling grid generation network calculates the coordinate correspondence of the image pixels before and after the spatial transformation according to the spatial transformation parameters, the sampling network performs sampling according to the image pixel coordinate correspondence, and uses a bilinear difference algorithm on the original foreground image to calculate the pixel value of each point in the image after the spatial transformation.

[0012] The spatial transformation parameter prediction network includes: stacked fully connected layers and linear rectifier units.

[0013] The camera view domain encoding network, image feature extraction network and spatial transformation network all use the adversarial loss function output by the discriminator according to the real image and the synthetic image and the triplet loss function based on contrastive learning as constraints, wherein: the real image is an image obtained by directly shooting the foreground in the data set that contains the foreground belonging to the same category as the target foreground or contains the target foreground; the synthetic image is a background image in the data set that does not contain the foreground of the same category, and is an image directly synthesized with the target foreground adjusted by the spatial transformation network according to the foreground mask.

[0014] The triplet loss function based on contrastive learning is L1+L2, where:

[0015] is the view domain encoding of the geometrically corrected foreground image, is the view domain encoding of the original foreground image, z f and z b They represent the view domain encoding of the foreground image and the background image in the real image containing the foreground, respectively. The d(·, ·) function refers to the Euclidean distance, and m represents the allowed margin.

[0016] The adversarial loss function Where: I t is a synthetic image, I is a real image, is the mathematical expectation of the data distribution corresponding to the synthetic image, E I [·] is the mathematical expectation of the data distribution corresponding to the real image, and D[·] is the output of the discriminator network.

[0017] The discriminator includes: sequentially stacked convolutional layers and leaky rectified linear units, wherein: the convolutional layer performs convolution processing according to the feature map information of the image to obtain the high-level feature map result of the image, and the linear rectifier unit performs linear rectification processing according to the feature map information of the image, introduces nonlinear operations, and enables the neural network to have the ability to fit nonlinear functions.

[0018] The discriminator is constrained by the adversarial loss function so that the target, i.e., the adversarial loss function L D Get the minimum value, the adversarial loss function Where: I t To synthesize the image, is the mathematical expectation of the data distribution corresponding to the synthetic image, p t is the spatial transformation parameter predicted by the spatial transformer network, ||·||2 is the L2 norm, and D[·] is the output of the discriminator network.

[0019] Technical Effects

[0020] The present invention uses a camera view domain encoder restricted by a triplet loss function, utilizes camera view domain encoding of a background image and automatically estimates parameters of a foreground image spatial transformation. These characteristics make the present invention have great practical application value in the field of image editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the present invention;

[0022] Figure 2 A network structure diagram of a camera viewing angle domain encoding network according to the present invention;

[0023] Figure 3 A network structure diagram of the image feature extraction network of the present invention;

[0024] Figure 4 A schematic diagram showing the comparison of the effects of the present invention and other methods;

[0025] Figure 5 This is a result diagram of the effectiveness verification experiment of the camera viewing angle domain encoder in the present invention. DETAILED DESCRIPTION

[0026] like Figure 1 As shown, this embodiment relates to a method for automatically editing a spatially transformed image based on camera view domain coding, comprising:

[0027] Step 1. Camera view domain coding extraction: In order to extract the view domain coding of irregular shaped areas, the input image is mapped to the camera view domain space through the view domain coding network. The foreground or background image is used as input, and the coding of the image in the view domain space is output. The output of the view domain coding network only depends on the aggregated features in the masked area, preventing information leakage from the unmasked area.

[0028] like Figure 2 As shown in FIG. 1 , the view domain encoding network includes stacked partial convolutional layers, rectified linear units, instance normalization layers, and global average pooling layers.

[0029] In order to force the view domain encoding to include the camera view information, a triplet loss function based on contrastive learning is used to constrain the view domain encoding network so that the target L1+L2 achieves the minimum value. in: is the view domain encoding of the geometrically corrected foreground image, is the view domain encoding of the original foreground image, z f and z b They represent the view domain encoding of the foreground image and the background image in the real image containing the foreground. The d(·,·) function refers to the Euclidean distance, and m represents the allowed margin.

[0030] Step 2. Image feature extraction: The feature extraction network maps the input image to the feature space, takes the foreground or background image as input, and outputs a feature map of the image containing spatial features. This feature map provides the necessary information for the spatial transformation network to predict the spatial transformation parameters.

[0031] like Figure 3 As shown, the image feature extraction network includes a convolutional layer and rectified linear units.

[0032] Step 3. Image feature enhancement: Encode the background image obtained in step 1 in the camera view domain F a ∈R l , after copying and tiling, the visual domain encoding is F a ′∈R W·H·l , and the foreground and background image features obtained in step 2 are concatenated in the channel dimension to obtain the enhanced image feature F e ∈R W·H·(l+2c) , input the spatial transformation network to obtain the distorted foreground image, where: l is the camera view domain encoding dimension, and c is the image feature map dimension.

[0033] Step 4. Prediction and implementation of spatial transformation parameters of foreground image: The spatial transformation parameter prediction network uses the original foreground image and enhanced image features as input to predict the spatial transformation parameters; the sampling grid generation network calculates the coordinate correspondence of image pixels before and after the spatial transformation based on the spatial transformation parameters; the sampling network performs sampling based on the image pixel coordinate correspondence, and uses the bilinear interpolation algorithm to calculate the pixel value of each point in the image after the spatial transformation.

[0034] After the above steps, the adjusted foreground image has a perspective angle consistent with the background image, has a size matching the background image and is located in an appropriate spatial position. According to the mask of the foreground image, the area with a mask of 1 in the synthesized image is assigned the pixel value of the corresponding area of ​​the foreground image, and the area with a mask of 0 is assigned the pixel value of the corresponding area of ​​the background image, and a geometrically consistent synthesized image is obtained.

[0035] After specific practical experiments, a single GeForce GTX1080Ti GPU was used to train the neural network under the Ubuntu 18.04.1LTS operating system. The network was written in the TensorFlow1.14.0 framework. The initial value of the weight matrix in the neural network obeys a normal distribution with a mean of 0 and a variance of 0.01. The initial learning rate is 0.00001, and the Adam optimizer is used for optimization. The experimental data that can be obtained are shown in Table 1 and Figure 5 shown. Figure 5 The effect of the camera view domain encoder is demonstrated. The number above each picture represents the Euclidean distance of the camera view domain encoding of the background image and the foreground image in the picture. The smaller the number, the closer the distance between the two images in the camera view domain space, and the higher the similarity of the camera view.

[0036] As shown in Table 1, this is a comparison of the effects of this method and other methods. The first two rows are the original foreground and background images to be edited, respectively. The third row is the synthetic image obtained by the automatic image editing method proposed by this method. The fourth and fifth rows are synthetic images obtained by other automatic image editing methods.

[0037] Table 1

[0038] Fréchet Inception Distance User Research Score The present invention 29.5 49％ ST-GAN 35.8 20％ Compositional-GAN 30.3 31％

[0039] Compared with the prior art, the synthetic images obtained by the automatic image editing algorithm for the same set of foreground and background images in the dataset of the present invention are more similar to the real images in the dataset, which is reflected in the smaller Fréchet Inception distance. At the same time, the synthetic images are more realistic, which is reflected in the higher user research scores.

[0040] This method can extract the information of the background image in the camera perspective domain space and use the information to guide the adjustment of the foreground image, which helps to better eliminate the geometric inconsistencies in the synthesized image. This method uses a camera perspective domain coding network composed of continuously stacked partial convolutional layers, and uses a triplet loss function based on contrastive learning to constrain the camera perspective domain coding network, so that it has the ability to extract angle coding. This method uses the camera perspective domain coding information to guide the adjustment process of the foreground image, so that the adjusted foreground image has a consistent perspective angle with the background image, which plays a great role in improving the harmony of the synthesized image. In the process of adjusting the foreground image, this method uses the camera perspective domain coding of the background image, which is not possible with other image harmonization algorithms. This feature makes this method have great application value in the field of synthesizing harmonious images.

[0041] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for automatically editing spatially transformed images based on camera view domain coding, characterized in that: The camera angle domain coding network is used to extract angle codes from the background image, and the image feature extraction network is used to extract the foreground image features and the background image features. The angle domain codes of the copied and tiled background images are concatenated with the foreground and background image features to obtain enhanced image features. The enhanced image features are input into the spatial transformation network to obtain the predicted spatial transformation parameters of the foreground image, and the parameters are used to sample from the original foreground image, and the distorted foreground image is obtained by using the bilinear difference method, and then the distorted foreground image and the input background image are synthesized to obtain a synthetic image that conforms to the geometric perspective relationship; The camera perspective domain coding network maps the input image to the camera perspective domain space according to the foreground or background image, and outputs the coding of the image in the perspective domain space, wherein the coding represents the information of the camera perspective angle when the image is taken, and the Euclidean distance between the codings of the two images in the perspective domain space can represent the similarity between the perspective angles of the two images; The image feature extraction network maps the input image to a feature space according to the foreground or background image, and outputs a feature map of the image containing spatial features.

2. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 1, characterized in that: The camera viewing angle domain coding network includes: four basic units, an average pooling layer and a convolution layer arranged after the basic units, wherein: the four basic units perform feature extraction processing according to the feature map information of the image to obtain the high-level feature map result of the image, the average pooling layer performs pooling processing according to the feature map information of the image to obtain the result of aggregating important information of the feature map and reducing the amount of calculation, and the convolution layer performs convolution processing according to the image feature map information aggregated by the pooling layer to obtain the encoding of the image in the camera viewing angle domain.

3. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 2, characterized in that: Each of the basic units includes: a partial convolution layer, an instance normalization layer and a linear rectification unit, wherein: the partial convolution layer performs convolution processing according to the feature map information of the image to obtain the high-level feature map result of the image, the instance normalization layer performs normalization processing according to the feature map information of the image to ensure the independence of the image instance while making the data distribution consistent and accelerating the convergence of the model, and the linear rectification unit performs linear rectification processing according to the normalized image feature map information, introduces nonlinear operations, and enables the neural network to have the ability to fit nonlinear functions.

4. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 1, characterized in that: The image feature extraction network includes: five basic units connected in series, each basic unit performs feature aggregation processing according to the feature map information of the image to obtain the high-level feature map result of the image, and each basic unit includes a convolution layer and a linear rectification unit, wherein: the convolution layer performs convolution processing according to the feature map information of the image to obtain the high-level feature map result of the image, and the linear rectification unit performs linear rectification processing according to the feature map information of the image, introduces nonlinear operations, and enables the neural network to have the ability to fit nonlinear functions.

5. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 1, characterized in that: The spatial transformation network comprises: a spatial transformation parameter prediction network, a sampling grid generation network and a sampling network, wherein: the spatial transformation parameter prediction network predicts the spatial transformation parameters according to the original foreground image, the foreground background image feature map and the background image camera view domain coding prediction, the sampling grid generation network calculates the coordinate correspondence of the image pixels before and after the spatial transformation according to the spatial transformation parameters, the sampling network performs sampling according to the image pixel coordinate correspondence, and calculates the pixel value of each point of the image after the spatial transformation using a bilinear difference algorithm on the original foreground image; The spatial transformation parameter prediction network includes: stacked fully connected layers and linear rectifier units.

6. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 1, characterized in that: The camera view domain encoding network, image feature extraction network and spatial transformation network all use the adversarial loss function output by the discriminator according to the real image and the synthetic image and the triplet loss function based on contrastive learning as constraints, wherein: the real image is an image obtained by directly shooting the foreground in the data set that contains the foreground belonging to the same category as the target foreground or contains the target foreground; the synthetic image is a background image in the data set that does not contain the foreground of the same category, and is an image directly synthesized with the target foreground adjusted by the spatial transformation network according to the foreground mask.

7. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 6, characterized in that: The adversarial loss function Where: I t is a synthetic image, I is a real image, is the mathematical expectation of the data distribution corresponding to the synthetic image, E I [·] is the mathematical expectation of the data distribution corresponding to the real image, and D[·] is the output of the discriminator network.

8. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 6, characterized in that: The triplet loss function based on contrastive learning is L1+L2, where: is the view domain encoding of the geometrically corrected foreground image, is the view domain encoding of the original foreground image, z f and z b They represent the view domain encoding of the foreground image and the background image in the real image containing the foreground, respectively. The d(·, ·) function refers to the Euclidean distance, and m represents the allowed margin.

9. The method for automatically editing spatially transformed images based on camera view domain coding according to claim 6 or 7, characterized in that: The discriminator comprises: sequentially stacked convolutional layers and leaky linear rectification units, wherein: the convolutional layer performs convolution processing according to the feature map information of the image to obtain the high-level feature map result of the image, and the linear rectification unit performs linear rectification processing according to the feature map information of the image, introduces nonlinear operations, and enables the neural network to have the ability to fit nonlinear functions; The discriminator is constrained by the adversarial loss function so that the target, i.e., the adversarial loss function L D Get the minimum value, the adversarial loss function Where: I t To synthesize the image, is the mathematical expectation of the data distribution corresponding to the synthetic image, p t is the spatial transformation parameter predicted by the spatial transformer network, ||·||2 is the L2 norm, and D[·] is the output of the discriminator network.

Images