A method, device and storage medium for image synthesis based on relation embedding

Abstract

The present invention discloses a method, device and storage medium for image synthesis based on relation embedding, the method comprising: embedding the inverse relationship between image synthesis and image restoration into an image synthesis model; performing mutual supervision learning on images based on the inverse relationship, so that the image synthesis model learns the objects of the foreground image and the features of the background image; training the image synthesis model with relation embedding to obtain an image synthesizer, the image synthesizer comprising: a generator and a discriminator; training a synthetic image scoring classifier based on a data set of synthetic images to automatically score the synthetic images; performing image synthesis and scoring the synthetic images based on the image synthesizer, the synthetic image scoring classifier, the foreground image, the background image, and the trained relation embedding image synthesis model. The device comprises: a processor and a memory. The image synthesis relationship designed by the present invention enables the foreground and background images to achieve better image synthesis.

Description

Technical Field

[0001] The present invention relates to the field of image synthesis, and in particular to a method, device and storage medium for image synthesis based on relationship embedding. Background Art

[0002] In the early work of image synthesis, researchers used a combination of graphics and mathematical ideas to achieve image synthesis. The most classic one is the application of Poisson fusion for image synthesis. Its idea is based on the general interpolation mechanism for solving the Poisson equation, introducing the image gradient domain, and using color gradient instead of color intensity to produce a more realistic synthesis effect when fusion. Later, with the development of deep learning research, insufficient data is a more serious problem. The model cannot learn valuable representations from the data. Researchers have focused on using deep learning methods to solve image synthesis, thereby solving the problems of insufficient data and time-consuming and labor-intensive manual labeling to a certain extent. GAN (Generative Adversarial Network) helps solve the problem of image generation, DCGAN (Deep Convolutional Generative Adversarial Network) can generate images belonging to a specific class, and LAPGAN (Generative Adversarial Network Based on Laplacian Pyramid) can generate images from coarse to fine using Laplacian pyramid.

[0003] GAN can indeed help the model learn valuable representations, but using GAN alone cannot synthesize two independent images. From the perspective of data synthesis, researchers proposed a simple cut-and-paste method to extract objects under box-level annotations and paste them into possible scenes to obtain new and more realistic synthetic data, but more realistic data does not ensure that the data will definitely enhance the performance of the training model. Therefore, subsequent research combined the two methods of image synthesis and adversarial, proposed a new image synthesis network to learn synthetic images, and then used this method to train a synthesizer to generate useful synthetic samples, thereby helping to improve the performance of the target network. At present, there is relatively little work on image synthesis using the copy-and-paste method. People are more willing to use virtual engines to generate a large amount of synthetic data, and some work is based on virtual image scenes to obtain synthetic data sets. In addition, synthetic data also helps people better understand the real world. Synthetic images can be divided into real and non-real images, and the real synthetic image areas can be recolored, which helps to better understand natural color statistics and color perception. Based on prior knowledge of the real world, people can find that the foreground and background of synthetic images are incompatible, and there will be differences in the data distribution between synthetic data sets and real data sets.

[0004] In general, the following problems exist in the field of image synthesis:

[0005] 1. There is a difference in data distribution between synthetic data and real data, which makes model learning more difficult;

[0006] 2. The relationship between the foreground and background of the synthesis cannot be regularized, and deep learning cannot be used to describe the relationship between the foreground and background;

[0007] 3. Designing specific 3D models is expensive, which makes it expensive to use 3D models to synthesize data;

[0008] 4. It is difficult to evaluate the difference between synthetic data and real data sets, which makes it impossible to quantify the effectiveness of the synthetic model after synthesizing the data;

[0009] 5. It is impossible to automatically synthesize massive images and complete quality assessment. Summary of the invention

[0010] The present invention provides a method, device and storage medium for image synthesis based on relation embedding. The image synthesis relationship designed by the present invention enables foreground and background images to achieve better image synthesis; and based on the training and learning of the image synthesis model based on relation embedding, the consistency of the synthesized image is improved; the image synthesis of the present invention can expand useful data sets as needed, as described below:

[0011] In a first aspect, a method for image synthesis based on relation embedding comprises the following steps:

[0012] The inverse relationship between image synthesis and image restoration is embedded into the image synthesis model;

[0013] Based on the mutual inverse relationship, mutual supervised learning is performed on the images, so that the image synthesis model learns the features of the objects in the foreground image and the background image;

[0014] The image synthesis model of relational embedding is trained to obtain an image synthesizer, wherein the image synthesizer includes: a generator and a discriminator;

[0015] Based on the synthetic image dataset, a synthetic image scoring classifier is trained to automatically score the synthetic images;

[0016] Based on the image synthesizer, the synthetic image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model, the image synthesis work is performed and the synthetic image is scored.

[0017] In one embodiment, the inverse relationship is:

[0018] RSI = Relation(B, B+F)

[0019] Among them, B represents the restored background image, F represents the foreground image required for synthesis, and Relation represents the relationship.

[0020] The input of the image synthesis model includes: background image x, background image y with foreground image, where x, R is the feature space, H is the height of the face image, W is the width, and C is the number of channels.

[0021] In one embodiment, the image synthesis model includes: a generator and a discriminator,

[0022] The generator consists of a background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y )composition;

[0023] Encoder Used to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained With decoder (G x , G y ) to generate a new image, G x To generate the repaired image, G y To generate a synthetic image;

[0024] (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x For the decoder The generated inpainted image is distinguished from the natural image, D y For the decoder The generated synthetic images are distinguishable from natural images.

[0025] Preferably, the image synthesis work includes a repair route method and a synthesis route method:

[0026] The repair route is as follows:

[0027] Extract the target position feature f1 from the target background image B; extract the target background feature B2 from the natural image F+B;

[0028] Take f1 and B2 as the input of the generator and generate a synthetic image of f1+B2;

[0029] The synthetic route is:

[0030] Extract background feature B1 from target background image B; extract foreground feature f2 from natural image F+B;

[0031] Take f2+B1 as the input of the generator and generate a synthetic image of f2+B1.

[0032] Furthermore, the method further comprises: evaluating the quality of image synthesis based on the automatic synthesis score index and the ideal synthesis score index;

[0033] The automatic synthesis score is used to copy and paste the synthesized image, manually perform the appearance, size and position scoring and annotation to complete the training, and automatically score the synthesized image;

[0034] The proportional coefficients of the ideal composite scores are all a maximum value of 1.

[0035] In a second aspect, a device for image synthesis based on relation embedding is provided, the device comprising:

[0036] An embedding module, used to embed the inverse relationship between image synthesis and image restoration into the image synthesis model;

[0037] A learning module, configured to perform mutual supervision learning on the images based on the mutual inverse relationship, so that the image synthesis model learns the features of the objects in the foreground image and the background image;

[0038] An acquisition module, used for training the image synthesis model embedded in the relationship to obtain an image synthesizer, wherein the image synthesizer includes: a generator and a discriminator;

[0039] A training module, used for training a synthetic image scoring classifier based on a synthetic image dataset to automatically score the synthetic images;

[0040] The scoring module is used to perform image synthesis and score the synthesized image based on the image synthesizer, the synthesized image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model.

[0041] In a third aspect, a device for image synthesis based on relational embedding, the device comprises: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any method step described in the first aspect of claim.

[0042] In a fourth aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the method steps described in the first aspect.

[0043] The beneficial effects of the technical solution provided by the present invention are:

[0044] 1. This method can complete image synthesis and image restoration work well; and it is proved that image synthesis and image restoration are reversible;

[0045] 2. This method can learn the position and foreground ratio of image synthesis; improve the consistency of appearance, size and position features of image synthesis;

[0046] 3. This method can be applied to deep learning as a new data enhancement method; and experiments have proved that this method is suitable for application and promotion in the field of image synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of relation definition legend for image synthesis based on relation embedding;

[0048] Figure 2 A graph of image synthesis models based on relational embeddings;

[0049] Figure 3 A logical flow chart of relation embedding for image synthesis based on relation embedding;

[0050] Figure 4 Flowchart for image synthesis using relational embeddings.

[0051] Figure 5 An example diagram of scoring an evaluation of image synthesis based on relational embeddings;

[0052] Figure 6 It is an overall flow chart of image synthesis based on relation embedding;

[0053] Figure 7 A schematic diagram of the structure of a device for image synthesis based on relation embedding;

[0054] Figure 8 Another structural schematic diagram of a device for image synthesis based on relation embedding. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0056] Example 1

[0057] The embodiment of the present invention provides a method for image synthesis based on relationship embedding, see Figure 1-Figure 4 , the method comprises the following steps:

[0058] 101: Embed the inverse relationship between image synthesis and image restoration into the image synthesis model;

[0059] 102: Perform mutual supervision learning on images based on the mutual inverse relationship, so that the image synthesis model learns the features of the objects in the foreground image and the background image;

[0060] 103: Train the image synthesis model for relational embedding to obtain an image synthesizer, wherein the image synthesizer includes a generator and a discriminator;

[0061] 104: Based on the synthetic image dataset, train a synthetic image scoring classifier to automatically score synthetic images;

[0062] 105: Based on the image synthesizer, the synthetic image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model, perform image synthesis and score the synthetic image.

[0063] The input of the image synthesis model in step 102 includes:

[0064] Background image x, background image y with foreground image, where x, R is the feature space, H is the height of the face image, W is the width, and C is the number of channels.

[0065] In one embodiment, the image synthesis model in step 101 and step 102 includes: a generator and a discriminator,

[0066] The generator consists of a background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y )composition;

[0067] Encoder Used to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained With decoder (G x , G y ) to generate a new image, G x To generate the repaired image, G y To generate a synthetic image;

[0068] (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x For the decoder The generated inpainted image is distinguished from the natural image, D y For the decoder The generated synthetic images are distinguishable from natural images.

[0069] In one embodiment, the image synthesis work in step 105 includes: a repair route method and a synthesis route method;

[0070] Among them, the repair route method is:

[0071] Extract the target position feature f1 from the target background image B; extract the target background feature B2 from the natural image F+B;

[0072] Take f1 and B2 as the input of the generator and generate a repaired image of f1+B2;

[0073] The synthetic route is:

[0074] Extract background feature B1 from target background image B; extract foreground feature f2 from natural image F+B;

[0075] Take f2+B1 as the input of the generator and generate a synthetic image of f2+B1.

[0076] In one embodiment, the method further includes: evaluating the quality of image synthesis based on the automatic synthesis score index and the ideal synthesis score index;

[0077] Among them, the automatic synthesis score is used to copy and paste the synthesized image, manually perform the appearance, size and position scoring and annotation to complete the training, and automatically score the synthesized image; the proportional coefficient of the ideal synthesis score is the maximum value of 1.

[0078] In summary, the embodiment of the present invention achieves better synthesis of foreground and background images through the above steps 101 to 105; and improves the consistency of the synthesized image based on the training and learning of the image synthesis model embedded in the relationship.

[0079] Example 2

[0080] The scheme in Example 1 is further introduced below in combination with specific calculation formulas and examples, as described below for details:

[0081] 1. Image Synthesis Model with Relation Embedding

[0082] 1. Definition of Relation Embedding

[0083] The embodiment of the present invention defines the relational embedding here as the relationship between image synthesis and restoration, referred to as RSI (Relationship Between Image Synthesis and Inpainting). To a certain extent, image synthesis and restoration can be regarded as an inverse relationship. Then, the embodiment of the present invention embeds RSI into the synthesis model.

[0084] Therefore, the learning goal of the synthesis model is to learn the inverse relationship between image synthesis and image restoration, so that the model can learn information such as the position and proportion of the synthesized foreground, and improve the appearance of the synthesized image, the size of the foreground image and the consistency in the background image as much as possible.

[0085] The RSI legend is as follows Figure 1 As shown in the figure, the RSI relationship is shown by taking the face without glasses and the face with glasses as examples. The upper row represents the synthesis process in RSI, and the lower row represents the repair process in RSI. The two processes constitute the relationship of RSI, which can be formally expressed as:

[0086] RSI=Relation(B,B+F) (1)

[0087] Among them, B represents the background image after Inpainting, F represents the foreground image required for Synthesis, and Relation represents the relationship.

[0088] 2. Image Synthesis Model with Relational Embedding

[0089] The image synthesis model framework of the relational embedding of the embodiment of the present invention is designed based on VAE (Variational Autoencode) and GAN (Generative Adversarial Network). Figure 2 As shown, the model input has two images, one is the background image, which is defined as x, and the other is the background image with the foreground image, which is defined as y, where x, R is the feature space, H is the height of the face image, W is the width, and C is the number of channels. For the convenience of introduction, the image synthesis model based on relational embedding is called ISRE (Image Synthesizer Based on Relational Embedding). The details of the model are introduced below.

[0090] 2.1. ISRE generator design

[0091] The construction of the generator mainly includes an encoder and a decoder, such as Figure 2 As shown, it is mainly composed of the background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y ). Among them, the encoder The role of is to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained Then with the decoder (G x , G y ) to generate a new image. x Generate the repaired image, G y Generate a synthetic image.

[0092] It should be noted that image x∈X represents the natural background image that does not contain the foreground of the synthetic image, the foreground object o∈O of the synthetic image represents the foreground image of the target synthetic image, image y∈Y represents the natural image that contains the foreground image of the synthetic image, and

[0093] 2.2. ISRE Discriminator Design

[0094] Figure 2 D represents the discriminator, where (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x The purpose is to decode The generated inpainted image is distinguished from the natural image.

[0095] in and As G x The two input parameters Indicates the appearance encoding information of the background image, parameters Indicates the encoding information of the object in the foreground image. y The purpose is to decode The generated synthetic images are distinguished from natural images, where and As G y The two input parameters Represents the background appearance encoding information of the foreground image, It represents the position feature information of the foreground object to be synthesized in the background image.

[0096] 2. Image Synthesis Algorithm for Relational Embedding

[0097] The overall process of the algorithm designed in the embodiment of the present invention is as follows: Figure 3 As shown in the figure, B represents a single background image, and F+B represents a natural image of foreground + background. Based on the RSI definition, two learning routes of the algorithm are designed. The upper route is the restoration route of the algorithm, and the lower route is the synthesis route of the algorithm. In detail:

[0098] 1. Repair route algorithm

[0099] Step 1: Extract the target position feature f1 from the target background image B;

[0100] Step 2: Extract target background features B2 from the natural image F+B;

[0101] Step 3: Take f1 and B2 as the input of the generator to generate a repaired image of f1+B2.

[0102] 2. Synthesis route algorithm

[0103] Step 1: Extract background features B1 from the target background image B;

[0104] Step 2: Extract foreground feature f2 from natural image F+B;

[0105] Step 3: Take f2+B1 as the input of the generator to generate a synthetic image of f2+B1.

[0106] Finally, the goal of the embodiment of the present invention is to synthesize a large number of f2+B1 images and then apply them as data sets to specific engineering scenarios.

[0107] 3. Design of loss function

[0108] In order to make the model learn better and enable the two learning routes to effectively learn the "synthesis-repair" relationship, the embodiment of the present invention designs five loss functions to constrain the target model, which are described in detail as follows:

[0109] 1) Loss function related to f1+B2

[0110] Two loss functions are designed for f1+B2 to constrain its synthesis effect. Loss1 constrains the foreground features of the image synthesized by f1+B2, and loss2 constrains the background features of the image synthesized by f1+B2, which are defined as follows:

[0111]

[0112]

[0113] 2) Loss function related to f2+B1

[0114] Two loss functions are designed for f2+B1 to constrain its synthesis effect. Among them, loss3 constrains the foreground features of the f2+B1 synthesized image, and loss4 constrains the background features of the f2+B1 synthesized image, which are defined as follows:

[0115]

[0116]

[0117] 3) Generate adversarial loss function

[0118] In order to encourage the generated synthetic images to be better, that is, indistinguishable from natural images, the embodiment of the present invention adopts a generative adversarial loss, where G x and G y Trying to generate realistic synthetic images, D x and D yIn an attempt to distinguish natural images from generated synthetic images, the embodiment of the present invention defines the generation adversarial loss as follows:

[0119]

[0120] Among them, B y represents the background characteristics of y, F x represents the foreground feature of x; correspondingly, B x represents the background features of x, F y represents the foreground feature of y, D x (x) and D y (y) represents the discriminator for distinguishing synthetic images from natural images, and E represents the mathematical expectation.

[0121] 4) Target learning function

[0122] The above loss function needs to be used simultaneously when training the generator and the discriminator. Therefore, the embodiment of the present invention defines a complete target learning function as:

[0123]

[0124] Among them, α is the loss weight of the f1+B2 synthetic image, and β is the loss weight of the f2+B1 synthetic image, and their values ​​are between 0 and 1.

[0125] 5) Image synthesis using ISRE model

[0126] After training the ISRE model based on the above steps, you can use the ISRE model for image synthesis. Figure 4 As shown, the embodiment of the present invention uses the street view dataset as an example. After the ISRE model is trained using the street view dataset, the foreground image (such as Figure 4 The car in the image) and the background image (such as Figure 4 The ISRE model then synthesizes the foreground image and the background image, and the final output image is the expected synthesized image. Similarly, the algorithm in the embodiment of the present invention can be used to complete the training in other data sets and can also be used as a synthesis model to perform image synthesis, thereby expanding the data set and solving the problem of insufficient data sets in deep learning.

[0127] 3. Design of New Evaluation Indicators

[0128] In this embodiment, two universal evaluation indicators for image synthesis are proposed. According to research, the evaluation indicator proposed in the embodiment of the present invention is the first universal evaluation indicator in the field of image synthesis. Therefore, it is also one of the important inventions of the embodiment of the present invention. The detailed introduction is as follows:

[0129] 1. Automatic synthesis score ACS

[0130] The Automatic Composite Score (ACS) is designed to evaluate the performance of the automatic synthesis strategy proposed by the relational embedding image synthesis algorithm. i Represents the synthesis result of the i-th synthetic image. The definition of the automatic synthesis score ACS is as follows:

[0131] ACS=λ1a(C i )+λ2s(C i )+λ3p(C i ) (8)

[0132] Among them, a, s, p represent the composite scores of appearance, size and position respectively, and {λ1, λ2, λ3} are the proportional coefficients corresponding to a, s, and p respectively, with values ​​between 0 and 1, indicating the importance in the score calculation. a, s, and p are predicted by the Composite Image Score Classifier (CISC). The Composite Image Score Classifier is obtained through training, specifically:

[0133] A batch of images were pre-synthesized by copying and pasting, and then the three scores were manually performed. An image score includes three parts: appearance, size, and position. When the classifier is trained, a synthetic image is input, and the three scores of a, s, and p of the image will be obtained. The value is between 0 and 1. The closer to 1, the better the synthesis effect. That is:

[0134] {a, s, p} = CISC(C i ) (9)

[0135] 2. Ideal Composite Score ICS

[0136] The ideal composite score ICS (Ideal Composite Score) is used to evaluate the maximum performance that traditional image composite algorithms can achieve through ideal composite strategies. It can be considered as the upper bound of the automatic composite score, that is, the case where all proportional coefficients are taken to the maximum value of 1. It aims to encourage the image composite algorithm with relational embedding to propose better and more effective automatic composite strategies. Let C i Represents the synthesis result of the i-th synthetic image. The main definition of the ideal fusion score ICS is as follows:

[0137] ICS=a(C i )+s(C i )+p(C i ) (10)

[0138] like Figure 5As shown, after the ISRE model is used to complete the model training in the embodiment of the present invention, the test image is scored to obtain the scores of a, s, and p of the corresponding synthetic image. Figure 5 The example shows an ideal synthesis score, and the automatic synthesis score needs to be multiplied by the corresponding coefficient weight.

[0139] In summary, the embodiment of the present invention is an image synthesis method based on relationship embedding. The overall process is as follows: Figure 6 First, the embodiment of the present invention needs to train the ISRE model designed in the embodiment, then use the model to complete the image synthesis work, and finally perform the quality evaluation of the synthesized image.

[0140] Example 3

[0141] A device for image synthesis based on relational embedding, see Figure 7 , the device comprises:

[0142] Embedding module 1, used for embedding the inverse relationship between image synthesis and image restoration into the image synthesis model;

[0143] A learning module 2 is used to perform mutual supervision learning on the images based on the mutual inverse relationship, so that the image synthesis model learns the features of the objects in the foreground image and the background image;

[0144] An acquisition module 3 is used to train the image synthesis model of relation embedding to obtain an image synthesizer, wherein the image synthesizer includes: a generator and a discriminator;

[0145] A training module 4 is used for training a synthetic image scoring classifier to automatically score the synthetic images based on the synthetic image data set;

[0146] The scoring module 5 is used to perform image synthesis and score the synthesized image based on the image synthesizer, the synthesized image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model.

[0147] Among them, the image synthesis model includes: generator and discriminator,

[0148] The generator consists of a background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y )composition;

[0149] Encoder Used to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained With decoder (G x , Gy ) to generate a new image, G x To generate the repaired image, G y To generate a synthetic image;

[0150] (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x For the decoder The generated inpainted image is distinguished from the natural image, D y For the decoder The generated synthetic images are distinguishable from natural images.

[0151] In summary, the embodiments of the present invention achieve better synthesis of foreground and background images through the above modules; and improve the consistency of the synthesized image based on the training and learning of the image synthesis model based on relationship embedding.

[0152] Example 4

[0153] A device for image synthesis based on relational embedding, see Figure 8 The device includes: a processor 6 and a memory 7, wherein the memory 7 stores program instructions, and the processor 6 calls the program instructions stored in the memory 7 to enable the device to execute the following method steps in Example 1:

[0154] The inverse relationship between image synthesis and image restoration is embedded into the image synthesis model;

[0155] Based on the mutual inverse relationship, mutual supervised learning is performed on the images, so that the image synthesis model learns the features of the objects in the foreground image and the background image;

[0156] The image synthesis model of relational embedding is trained to obtain an image synthesizer, wherein the image synthesizer includes: a generator and a discriminator;

[0157] Based on the synthetic image dataset, a synthetic image scoring classifier is trained to automatically score the synthetic images;

[0158] Based on the image synthesizer, the synthetic image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model, the image synthesis work is performed and the synthetic image is scored.

[0159] In one embodiment, the inverse relationship is:

[0160] RSI = Relation(B, B+F)

[0161] Among them, B represents the restored background image, F represents the foreground image required for synthesis, and Relation represents the relationship.

[0162] The input of the image synthesis model includes: background image x, background image y with foreground image, where x, R is the feature space, H is the height of the face image, W is the width, and C is the number of channels.

[0163] In one embodiment, the image synthesis model includes: a generator and a discriminator,

[0164] The generator consists of a background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y )composition;

[0165] Encoder Used to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained With decoder (G x , G y ) to generate a new image, G x To generate the repaired image, G y To generate a synthetic image;

[0166] (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x For the decoder The generated inpainted image is distinguished from the natural image, D y For the decoder The generated synthetic images are distinguishable from natural images.

[0167] Preferably, the image synthesis work includes: a repair route method and a synthesis route method:

[0168] The restoration route is as follows: extract the target position feature f1 from the target background image B; extract the target background feature B2 from the natural image F+B;

[0169] Take f1 and B2 as the input of the generator and generate a synthetic image of f1+B2;

[0170] The synthesis route is as follows: extract background feature B1 from the target background image B; extract foreground feature f2 from the natural image F+B;

[0171] Take f2+B1 as the input of the generator and generate a synthetic image of f2+B1.

[0172] Furthermore, the present invention also includes: evaluating the quality of image synthesis based on the automatic synthesis score index and the ideal synthesis score index;

[0173] Automatic synthesis scoring is used to copy and paste synthetic images, manually annotate the appearance, size and position, and complete the training to automatically score the synthetic images;

[0174] The ideal composite score has a maximum value of 1.

[0175] In summary, the embodiments of the present invention achieve better synthesis of foreground and background images through a processor and a memory; and improve the consistency of the synthesized image based on the training and learning of the image synthesis model based on relationship embedding.

[0176] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0177] The execution subjects of the above-mentioned processor 6 and memory 7 can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiment of the present invention does not limit the execution subject and it is selected according to the needs of actual applications.

[0178] The data signal is transmitted between the memory 7 and the processor 6 via the bus 8, which will not be described in detail in the embodiment of the present invention.

[0179] Example 5

[0180] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0181] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0182] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0183] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0184] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0185] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0186] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0187] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image synthesis based on relation embedding, characterized in that: The method comprises the following steps: The inverse relationship between image synthesis and image restoration is embedded into the image synthesis model; Based on the mutual inverse relationship, mutual supervised learning is performed on the images, so that the image synthesis model learns the features of the objects in the foreground image and the background image; The image synthesis model of relational embedding is trained to obtain an image synthesizer, wherein the image synthesizer includes: a generator and a discriminator; Based on the synthetic image dataset, a synthetic image scoring classifier is trained to automatically score the synthetic images; Based on the image synthesizer, the synthetic image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model, perform image synthesis and score the synthetic image; Among them, the inverse relationship is: RSI=Relation(B,B+F) Among them, B represents the restored background image, F represents the foreground image required for synthesis, and Relation represents the relationship; The image synthesis model includes: a generator and a discriminator, The generator consists of a background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y )composition; Background Image Appearance Encoder Used to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained With decoder (G x , G y ) to generate a new image, G x To generate the repaired image, G y To generate a synthetic image; (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x For the decoder The generated inpainted image is distinguished from the natural image, D y For the decoder The generated synthetic images are distinguishable from natural images; The image synthesis work includes a repair route method and a synthesis route method: The repair route is as follows: Extract the target position feature f1 from the target background image B; extract the target background feature B2 from the natural image F+B; Take f1 and B2 as the input of the generator and generate a repaired image of f1+B2; The synthetic route is: Extract background feature B1 from target background image B; extract foreground feature f2 from natural image F+B; Take f2+B1 as the input of the generator and generate a synthetic image of f2+B1; The loss function of the image synthesis model is: 1) Loss function related to f1+B2 Loss1 constrains the foreground features of the f1+B2 composite image, and loss2 constrains the background features of the f1+B2 composite image: 2) Loss function related to f2+B1 Loss3 constrains the foreground features of the synthesized image f2+B1, and loss4 constrains the background features of the synthesized image f2+B1: 3) Generate adversarial loss function G x and G y For generating realistic synthetic images, D x and D y Used to distinguish natural images from generated synthetic images: Generate adversarial loss is defined as: L adv =E[logD x (x)]+E{log[1-D x (G x (B y ,F x ))]}+E[logD y (y)]+E{log[1-D y (G y (B x ,F y ))]} Among them, B y represents the background characteristics of y, F x represents the foreground feature of x; B x represents the background features of x, F y represents the foreground feature of y, D x (x) and D y (y) represents the discriminator for distinguishing synthetic images from natural images, and E represents the mathematical expectation.

2. The method for image synthesis based on relation embedding according to claim 1, characterized in that: The input of the image synthesis model includes: background image x, background image y with foreground image, where R is the feature space, H is the height of the face image, W is the width, and C is the number of channels.

3. The method for image synthesis based on relation embedding according to claim 1, characterized in that: The method further comprises: evaluating the quality of image synthesis based on the automatic synthesis score index and the ideal synthesis score index; The automatic synthesis score is used to copy and paste the synthesized image, manually perform the appearance, size and position scoring and annotation to complete the training, and automatically score the synthesized image; The proportional coefficients of the ideal composite scores are all a maximum value of 1.

4. A device for image synthesis based on relation embedding, characterized in that: The device comprises: An embedding module, used to embed the inverse relationship between image synthesis and image restoration into the image synthesis model; A learning module, configured to perform mutual supervision learning on the images based on the mutual inverse relationship, so that the image synthesis model learns the features of the objects in the foreground image and the background image; An acquisition module, used for training the image synthesis model embedded in the relationship to obtain an image synthesizer, wherein the image synthesizer includes: a generator and a discriminator; A training module, used for training a synthetic image scoring classifier based on a synthetic image dataset to automatically score the synthetic images; A scoring module, which is used to perform image synthesis and score the synthesized image based on the image synthesizer, the synthesized image score classifier, the foreground image, the background image, and the trained relationship embedding image synthesis model; Wherein, the inverse relationship is: RSI=Relation(B,B+F) Among them, B represents the restored background image, F represents the foreground image required for synthesis, and Relation represents the relationship; The image synthesis model includes: a generator and a discriminator, The generator consists of a background image appearance encoder Foreground Image Object Encoder and decoder (G x , G y )composition; Background Image Appearance Encoder Used to obtain the background appearance feature vector by encoding the background image Similarly, the feature vector of the object in the foreground image is obtained With decoder (G x , G y ) to generate a new image, G x To generate the repaired image, G y To generate a synthetic image; (D x , D y ) are two discriminators for generating repaired images and synthetic images respectively, D x For the decoder The generated inpainted image is distinguished from the natural image, D y For the decoder The generated synthetic images are distinguishable from natural images; The image synthesis work includes a repair route method and a synthesis route method: The repair route is as follows: Extract the target position feature f1 from the target background image B; extract the target background feature B2 from the natural image F+B; Take f1 and B2 as the input of the generator and generate a repaired image of f1+B2; The synthetic route is: Extract background feature B1 from target background image B; extract foreground feature f2 from natural image F+B; Take f2+B1 as the input of the generator and generate a synthetic image of f2+B1; The loss function of the image synthesis model is: 1) Loss function related to f1+B2 Loss1 constrains the foreground features of the f1+B2 composite image, and loss2 constrains the background features of the f1+B2 composite image: 2) Loss function related to f2+B1 Loss3 constrains the foreground features of the synthesized image f2+B1, and loss4 constrains the background features of the synthesized image f2+B1: 3) Generate adversarial loss function G x and G y For generating realistic synthetic images, D x and D y Used to distinguish natural images from generated synthetic images: Generate adversarial loss is defined as: L adv =E[logD x (x)]+E{log[1-D x (G x (B y ,F x ))]}+E[logD y (y)]+E{log[1-D y (G y (B x ,F y ))]} Among them, B y represents the background characteristics of y, F x represents the foreground feature of x; B x represents the background features of x, F y represents the foreground feature of y, D x (x) and D y (y) represents the discriminator for distinguishing synthetic images from natural images, and E represents the mathematical expectation.

5. A device for image synthesis based on relation embedding, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method steps described in any one of claims 1 to 3.

Images