Method, electronic device, and program product for generating image

Abstract

The embodiment of the invention relates to a method for generating an image, electronic equipment and a computer program product. The method includes obtaining a semantic segmentation map by semantic segmentation of a source image. The method further includes acquiring keywords for describing features of a target image to be generated. The method further includes obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keywords. The method further includes generating a target image based on the transformed semantic segmentation map. According to the method provided by the embodiment of the invention, the target image can be generated by using the semantic segmentation map and the keyword of the source image, so that the generated target image has the target features and has semantic consistency with the source image, and thus the high-quality target image can be generated in a controllable and customizable manner.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing, and in particular to a method, an electronic device, and a computer program product for generating an image. Background Art

[0002] As the functions of electronic devices increase day by day, image processing has been widely used in various applications of electronic devices, such as computer vision, autonomous driving, facial recognition, and medical imaging.

[0003] For some applications, there are high requirements for the accuracy of image processing. In order to improve the accuracy of image processing, these applications usually use image processing models to process images. However, as user requirements increase, these image processing models may face increasingly severe challenges in terms of accuracy. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for generating an image.

[0005] According to a first aspect of the present disclosure, a method for generating an image is provided. The method includes obtaining a semantic segmentation map by semantically segmenting a source image. The method also includes obtaining a keyword for describing the features of a target image to be generated. The method also includes obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keyword. The method also includes generating a target image based on the transformed semantic segmentation map.

[0006] According to a second aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, the instructions causing the device to perform an action when executed by the at least one processor, the action including obtaining a semantic segmentation map by semantically segmenting a source image. The action also includes obtaining a keyword for describing the features of a target image to be generated. The action also includes obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keyword. The action also includes generating a target image based on the transformed semantic segmentation map.

[0007] According to a third aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a non-volatile computer-readable medium and comprises machine-executable instructions, which when executed cause a machine to perform the steps of the method in the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0009] Figure 1 A schematic diagram illustrating an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented is illustrated.

[0010] Figure 2 A flowchart of a method for generating an image according to an embodiment of the present disclosure is illustrated.

[0011] Figure 3 A schematic diagram of a predetermined feature space according to an embodiment of the present disclosure is illustrated.

[0012] Figure 4 A schematic diagram of a trained neural network model according to an embodiment of the present disclosure is illustrated.

[0013] Figure 5 A schematic diagram of generating and using a target image according to an embodiment of the present disclosure is illustrated.

[0014] Figure 6 A block diagram of an electronic device for generating an image according to an embodiment of the present disclosure is illustrated.

[0015] Figure 7 A schematic block diagram of one example of an example device suitable for implementing embodiments of the present disclosure is illustrated.

[0016] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] With the development of electronic devices, more and more applications in electronic devices use image processing models to perform image processing. For example, semantic segmentation models have been widely used in various applications of electronic devices, such as computer vision, autonomous driving, facial recognition, and medical imaging. The semantic segmentation model is trained based on a training data set so that it can be used to perform image processing, such as semantic segmentation of images. The accuracy of the semantic segmentation model depends on the size and quality of the training data set. The training data set usually includes real-world images obtained from the real world and associated annotation information. However, due to environmental conditions or due to time or labor costs, it is usually difficult to obtain sufficient real-world images from the real world. For example, for semantic segmentation models used in the field of autonomous driving, it is usually difficult to obtain sufficient real-world images under various lighting conditions, weather conditions, and other environmental or driving conditions to establish a training data set. Therefore, it is necessary to generate new images based on real-world images that have been obtained (e.g., "source images" in existing training data sets) to enhance the training data set.

[0020] Typically, in the process of enhancing a training dataset, a new image is generated by randomly flipping, rotating, and scaling the already obtained source image. However, only a limited number of images can be generated in this way. For example, when the source image is an image captured indoors, it is impossible to generate a new image corresponding to outdoor lighting conditions. In addition, the images generated in this way may not conform to the real-world image type under the complex semantic pattern of the semantic segmentation model, which may cause the training dataset enhanced using the generated images to have an offset, thereby resulting in poor performance and low accuracy of the semantic segmentation model trained by the training dataset with the offset.

[0021] In addition, although with the development of artificial intelligence (AI), some drawing applications can also generate images, these applications are usually designed to generate images for entertainment or cultural creation. These generated images usually have large differences from the source images (for example, they have random background transformations or image content transformations relative to the source images) and cannot be used to enhance the training data set of the semantic segmentation model.

[0022] At least to solve the above and other potential problems, an embodiment of the present disclosure provides a method for generating an image. The method includes obtaining a semantic segmentation map by semantically segmenting a source image. The method also includes obtaining keywords for describing the features of a target image to be generated. The method also includes obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keywords. The method also includes generating a target image based on the transformed semantic segmentation map. According to the method of an embodiment of the present disclosure, a target image can be generated by using a semantic segmentation map and keywords of a source image, so that the generated target image has target features and has semantic consistency with the source image, thereby enabling a high-quality target image to be generated in a controllable and customizable manner. This enables the generated target image to effectively enhance a training data set including a source image, thereby improving the accuracy and robustness of a semantic segmentation model trained using the enhanced training data set.

[0023] The embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. Figure 1 A schematic diagram of an example environment 100 is illustrated in which devices and / or methods according to embodiments of the present disclosure may be implemented according to embodiments of the present disclosure. Figure 1 The example environment 100 shows a device 110 that can implement the device and / or method according to the embodiments of the present disclosure. The device 110 can be any electronic device capable of executing the method according to the present disclosure, for example, the device 110 can be an electronic device capable of running a trained neural network model to execute the method according to the present disclosure, such as a computer, a portable electronic device, and a vehicle-mounted terminal.

[0024] In some embodiments, the input 120 of the device 110 may receive a source image 121 and a keyword (KEY) 122. The source image 121 may serve as a basis for the image generation process, and the keyword 122 may describe the characteristics of the target image 130 to be generated, such as "fogging" or "adding noise", etc. In some embodiments, the device 110 may perform semantic segmentation on the source image 121 to obtain semantic segmentation maps 111 and 112, etc. (although Figure 1 Only two semantic segmentation maps are shown, but in actual applications, any number of semantic segmentation maps can be generated as needed). In some embodiments, the device 110 can transform the semantic segmentation map 111 or 112, etc. by using a keyword, thereby generating a target image 130 using the transformed semantic segmentation map. The generated target image 130 has features corresponding to the keyword (for example, it has been "fogged") and is similar to the source image 121, which will be further described in the examples below.

[0025] In some embodiments, the generated target image 130 can be used to expand the training data set 140 for training a predetermined semantic segmentation model, that is, can be included in the training data set 140. In some embodiments, the transformed semantic segmentation map used to generate the target image 130 can be used as the annotation information of the target image 130. In some embodiments, the source image 121 received by the device 110 comes from the training data set 140. The training data set 140 may include multiple images 140-1, 140-2, ..., 140-M (M is an integer), and may also include annotation information corresponding to each image.

[0026] In the above manner, the target image 130 can be generated by using the semantic segmentation map of the source image 121 and the keyword 122, so that the generated target image 130 has target features and has semantic consistency with the source image 121, so that a high-quality target image can be generated in a controllable and customizable manner. This enables the generated target image 130 to effectively enhance the training data set 140 including the source image, thereby improving the accuracy and robustness of the semantic segmentation model trained using the enhanced training data set 140.

[0027] Figure 2 A flow chart of a method 200 for generating an image according to an embodiment of the present disclosure is illustrated. Figure 2 The method can be Figure 1 The device 110 shown in FIG. Figure 2 As shown, at block 202, by processing a source image (e.g., Figure 1 The source image 121 shown in FIG. 1 is semantically segmented to obtain a semantic segmentation map (eg, Figure 1 In some embodiments, the source image 121 is a training data set (e.g., Figure 1 The training dataset 140 is used for training a predetermined semantic segmentation model. At block 204, a target image to be generated (e.g., Figure 1 The keywords of the features of the target image 130 shown (for example, Figure 1 In some embodiments, the keyword 122 may be, for example, "atomization" or "adding noise", etc. It should be understood that the keyword 122 may also be any other keyword input by the user.

[0028] At block 206, the semantic segmentation map 111 or 112 is transformed by using the keyword 122 to obtain a transformed semantic segmentation map. At block 208, a target image (e.g., Figure 1In some embodiments, the method 200 according to the present disclosure may further include: including the target image 130 and the transformed semantic segmentation map in the training dataset 140 to enhance the training dataset 140. In this case, the transformed semantic segmentation map obtained at block 206 may be used as the annotation information of the target image 130. In some embodiments, the source image 121 may include at least one image. Accordingly, in some embodiments, the target image 130 may include at least one image corresponding to the source image 121.

[0029] By using the method 200 according to the present disclosure, a target image can be generated by using a semantic segmentation map and keywords of a source image, so that the generated target image has target features and is semantically consistent with the source image, thereby being able to generate a high-quality target image in a controllable and customizable manner.

[0030] In addition, in order to further improve the quality of the generated target image, the generated target image can be made to have sufficient distinction from the source image, and at the same time have authenticity and maintain similarity with the source image. To this end, in some embodiments, the method 200 according to the present disclosure may also include mapping the source image 121 and the keyword 122 to a predetermined feature space (e.g., a feature space represented by a vector). In some embodiments, in the predetermined feature space, the distance between the mutually matching image and the keyword may be less than the first predetermined distance, and the distance between the mutually non-matching image and the keyword may be greater than the second predetermined distance. In this case, in some embodiments, at box 208, the generated target image 130 can be made to have a distance between the target image 130 and the keyword 122 when it is mapped to the predetermined feature space, and the distance between the target image 130 and the keyword 122 and the source image 121 is greater than the second predetermined distance. Thus, the target image can be made to have sufficient distinction from the source image.

[0031] Figure 3 FIG. 3 is a schematic diagram of a predetermined feature space 300 according to an embodiment of the present disclosure. Figure 3 As shown, the distance D1 between the generated target image 130 and the keyword 122 is less than the first predetermined distance P1, and the distance D2 between the target image 130 and the keyword 122 and the source image 121 is greater than the second predetermined distance P2. In some embodiments, the first predetermined distance P1 and the second predetermined distance P2 can be set according to actual needs. For example, when the above mapping is performed by a neural network model, the first predetermined distance P1 and the second predetermined distance P2 can be determined according to the training degree of the neural network model. Although Figure 3Only one source image 121 , one target image 130 , and one keyword 122 are shown, but it should be understood that the predetermined feature space 300 may include more source images, target images, and keywords.

[0032] Return to reference Figure 2 In order to make the generated target image authentic and similar to the source image, in some embodiments, at block 208, the difference between the generated target image 130 and the real-world image may be smaller than a predetermined difference threshold. Thus, the target image may conform to the real-world image type in the complex semantic mode of the semantic segmentation model.

[0033] In addition, at block 208, the similarity between the generated target image 130 and the source image 120 may be greater than a predetermined similarity threshold. For example, in the process of generating the target image, only the semantic segmentation map related to the keyword may be transformed (e.g., "fogging") without changing other semantic segmentation maps, so that the generated target image has semantic consistency and a high similarity with the source image. Figure 1 As shown, relative to the source image 121 , only the background portion of the target image 130 is blurred, while the content in the target image remains consistent with the source image 121 .

[0034] In the above case, when the target image 130 as above is included in the training data set for use as training data, since the target image 130 has different features from the source image 121 but has semantic consistency, the training data set can include more types of data without causing deviation, which in turn makes the semantic segmentation model trained using the training data set have higher accuracy and higher robustness.

[0035] In some embodiments, in order to generate images more efficiently and accurately, the above method 200 can be performed by a trained neural network model. Figure 4 FIG. 4 is a schematic diagram of a trained neural network model 400 according to an embodiment of the present disclosure. Figure 4 As shown, in some embodiments, the trained neural network model 400 may include a first sub-network model 410, a second sub-network model 420, and a third sub-network model 430. In some embodiments, the first sub-network model 410 may be used to map the source image 121 and the keyword 122 to a predetermined feature space 413 (for example, it may be Figure 3 In some embodiments, the second sub-network model 420 can be used to obtain a semantic segmentation map 421 (for example, as shown in FIG. 1 ) by performing semantic segmentation on the source image 121. Figure 1In some embodiments, the third sub-network model 430 can be used to transform the semantic segmentation map 421 using the keyword 122 to obtain a transformed semantic segmentation map and generate the target image 130.

[0036] In some embodiments, in order to obtain the above trained neural network model 400, the method 200 according to the present disclosure may also include: acquiring the first sub-network model 410 by training the first neural network model 411 and the second neural network model 412. In some embodiments, the first neural network model 411 can be used to map an image into an image feature space, for example, mapping an image into a vector. In some embodiments, the second neural network model 412 can be used to map the keyword 122 into a text feature space, for example, mapping the text into a vector. For example, the first neural network model 411 and the second neural network model 412 can be any embedded neural network model. In this case, in some embodiments, training the first neural network model 411 and the second neural network model 412 may include: jointly training the first neural network model 411 and the second neural network model 412 so that the trained first neural network model 411 and the second neural network model 412 jointly map the input image and the input keyword into the above predetermined feature space 413.

[0037] In some embodiments, the method 200 of the present disclosure may further include: obtaining a trained semantic segmentation model as a second sub-network model 420. In some embodiments, the trained semantic segmentation model 420 may be different from a predetermined semantic segmentation model trained using a training data set to be enhanced. In some embodiments, the method 200 of the present disclosure may further include: obtaining a third sub-network model 430 by training a third neural network model. In some embodiments, the third neural network model may be based on a generative adversarial network (GAN, Generative Adversarial Network) architecture. For example, the third neural network model may be based on a GAN prior embedded network (GPEN, GAN Prior Embedded Network) architecture. In some embodiments, the third neural network model may include a generator model 431 and a discriminator model 432. The generator model 431 may be used to generate an output image based on an input semantic segmentation map and an input keyword. The discriminator model 432 may be used to determine whether an image is a real-world image.

[0038] In some embodiments, training the third neural network model may include: jointly training the generator model 431 and the discriminator model 432, so that the trained discriminator model 432 determines an image whose difference with the real-world image is less than a predetermined difference threshold as a real-world image, and the trained generator model 431 generates an output image that satisfies a predetermined condition. In some embodiments, the predetermined condition may include: the output image generated by the trained generator model 431 is determined to be a real-world image by the trained discriminator model 432. In addition, in some embodiments, the predetermined condition may also include: in a predetermined feature space 431 (for example, Figure 3 In the predetermined feature space 300 shown in FIG. 1 , the distance between the output image generated by the trained generator model 431 and the input keyword is less than the first predetermined distance P1. In some embodiments, the predetermined condition may also include: in the predetermined feature space 431 (or 300), the distance between the output image generated by the trained generator model 431 and the input keyword and the input image is greater than the second predetermined distance P2. In some embodiments, the predetermined condition may also include: the similarity between the output image generated by the trained generator model 431 and the input image is greater than the above-mentioned predetermined similarity threshold.

[0039] The following will show example constraints in the training process in the form of mathematical expressions. In some embodiments, the first neural network model 411 and the second neural network model 412 can be jointly trained using a contrast loss function as defined in the following equation (1).

[0040]

[0041] In equation (1), L contrastive represents the contrast loss function, E θ,I represents the first neural network model 411, E θ,T represents the second neural network model 412, I i Indicates input to E θ,I The image of T i and T j Indicates the corresponding input to E θ,T Keywords. i Belongs to the image training dataset I, T i and T j Belongs to the keyword training data set T. I i and T i Match each other, and I i and T j The image training data set I may include N images (I = I1, I2, ..., I N), N is an integer. Accordingly, the keyword training data set T may include N keywords (T = T1, T2, ..., T N ). In equation (1), if the input image I i and T i If they match each other, then y i =0, if the input image I i and T j If they do not match each other, then y i = 1. m represents the margin hyperparameter used to determine whether the input image I and keyword T match.

[0042] In some embodiments, the first neural network model E can be adjusted based on the above equation (1) as follows: θ,I Parameter θ of 411 I , and the second neural network model E θ,T Parameter θ of 412 T : In y i = 0, make the latter term in the above equation (1) as large as possible; i = 1, the first term in the above equation (1) is made as small as possible. Thus, in the common predetermined feature space, the distance between the mutually matching images and keywords is small (for example, smaller than the first predetermined distance), and the distance between the mutually non-matching images and keywords is large (for example, larger than the second predetermined distance).

[0043] In some embodiments, the first and second neural network models E can also be conditioned using the prior embedded loss function defined in the following equation (2): θ,I and E θ,T Conduct joint training.

[0044]

[0045] In the above equation (2), L prior represents the prior embedding loss function, I ori represents the source image, I gen represents the generated output image, y represents the input keyword, and Margin represents the boundary condition set as needed. In some embodiments, the first neural network model E can be adjusted based on the above equation (2) as follows: θ,I Parameter θ of 411 I , and the second neural network model E θ,T Parameter θ of 412 T : Make the first term in the above equation (2) as large as possible, and make the second term in the above equation (2) as small as possible. Thus, the distance between the generated output image and the keyword and the source image in the predetermined feature space can be large, for example, greater than the second predetermined distance.

[0046] In some embodiments, for Figure 4 The third neural network model of the third subnetwork 430 can use a joint loss function as defined in the following equation (3) to jointly train the generator model 431 and the discriminator model 432 included in the third neural network model.

[0047]

[0048] In equation (3), L represents the joint loss function, L adv represents the adversarial loss function, L prior Denotes another prior embedding loss function, L recon represents the reconstruction loss function, and λ1 and λ2 represent weight coefficients set according to actual needs.

[0049] L adv It can be expressed by the following equation (4):

[0050]

[0051] In equation (4), G represents the generator model 431, D represents the discriminator model 432, represents the expected value for the real-world image x, represents the expected value for the generated output image z. Thus, the trained discriminator model D can determine an image whose difference with the real-world image is less than a predetermined difference threshold as a real-world image (for example, the predetermined difference threshold can be a threshold determined according to the state (for example, resolution) of the discriminator model D at the end of training), and the output image generated by the trained generator model G can be determined by the trained discriminator model D as a real-world image. L prior It can be expressed by the following equation (5):

[0052]

[0053] Equation (5) can be used to make the distance between the output image generated by the trained generator model G and the input keyword and the input image greater than a second predetermined distance in a predetermined feature space (feature space 413 of the first subnetwork 410).

[0054] L recon It can be expressed by the following equation (6):

[0055]

[0056] Equation (6) can be used to make the similarity between the output image generated by the trained generator model G and the input image greater than a predetermined similarity threshold. It should be understood that the first predetermined distance, the second predetermined distance, the predetermined similarity threshold, etc. can all be determined by the state (e.g., accuracy) of the trained neural network model 410 at the end of training. In addition, in some embodiments, when adjusting the above model E based on the above equations (1)-(6), θ,I and E θ,T The parameter θ I and θ T , as well as the parameters of the generator model G and the discriminator model D, the stochastic gradient descent (SGD) method can be used for adjustment.

[0057] After the trained neural network model 410 is obtained through training in the above manner, the following Figure 2 The method 200 shown is used to generate a target image to enhance the training data set 140 for a predetermined semantic segmentation model. In addition, in some embodiments, when there is a pre-acquired semantic segmentation map set (e.g., a semantic segmentation map pool), only the trained third sub-network 430 can be used to generate the target image.

[0058] Figure 5 A schematic diagram 500 of generating and using a target image according to an embodiment of the present disclosure is illustrated. Figure 5 As shown, the trained third sub-network 506 (e.g., Figure 4 The third subnetwork 430 in the training data set 506 may receive the semantic segmentation map and the keywords from the semantic segmentation map set 502 and the keyword set 504. The trained third subnetwork 506 may generate a target image based on the received semantic segmentation map and the keywords. The generated target image and the transformed semantic segmentation map used to generate the target image may be included in the training data set 508 to enhance the training data set 508. Then, the enhanced training data set 508 may be used for training a predetermined semantic segmentation model.

[0059] In some embodiments, for example, the original training data set 508 may be composed of D ori =(x i ,y i )i=1 nori To indicate that x i represents the source image, y i represents the corresponding annotation (e.g., the semantic segmentation map of the source image), i represents the index value, and i is an integer. The enhanced training data set 508 can be composed of D aug =(x j ,y j )j=1 naug For example, naug =n ori +n gen , n aug represents the number of images in the enhanced training dataset 508, n ori represents the number of images in the original training dataset 508, n gen Indicates the number of target images generated.

[0060] In some embodiments, the training of the predetermined semantic segmentation model 510 may be performed by minimizing the objective function represented by the following equation (7):

[0061]

[0062] In equation (7), f(x j ) represents the input image x j The output of the predetermined semantic segmentation model 510, l(f(x j ), y j ) represents the loss function for the predetermined semantic segmentation model 510.

[0063] Figure 6 A block diagram of an electronic device for generating an image according to an embodiment of the present disclosure is illustrated. Figure 6 The electronic device 600 shown may be Figure 1 The device 110 shown or any electronic device capable of executing the method according to the present disclosure. The electronic device 600 may include at least one processor 602 and a memory 604. The memory 604 is coupled to the at least one processor 602 and has instructions stored thereon. The instructions, when executed by the at least one processor 602, cause the electronic device 600 to perform an action. In some embodiments, the action includes obtaining a semantic segmentation map by semantically segmenting the source image. In some embodiments, the source image is an image in a training data set, and the training data set is used for training a predetermined semantic segmentation model. In some embodiments, the action may also include obtaining a keyword for describing the features of the target image to be generated. In some embodiments, the action may also include obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keyword. In some embodiments, the action may also include generating a target image based on the transformed semantic segmentation map. In some embodiments, the source image may include at least one image, and the target image may include at least one image corresponding to the source image. In some embodiments, the action may also include including the target image and the transformed semantic segmentation map in the training data set to enhance the training data set. In some embodiments, the transformed semantic segmentation map may be used as annotation information for the target image. In some embodiments, the action may also include mapping the source image and the keyword into a predetermined feature space.

[0064] In some embodiments, in a predetermined feature space, the distance between a mutually matching image and a keyword may be less than a first predetermined distance, and the distance between a mutually non-matching image and a keyword may be greater than a second predetermined distance. In some embodiments, generating a target image based on a transformed semantic segmentation map may include: making the distance between the generated target image and the keyword less than a first predetermined distance when the generated target image is mapped into a predetermined feature space, and the distance between the target image and the keyword and the source image is greater than a second predetermined distance. In some embodiments, generating a target image based on a transformed semantic segmentation map may also include: making the difference between the generated target image and the real-world image less than a predetermined difference threshold, and making the similarity between the generated target image and the source image greater than a predetermined similarity threshold.

[0065] In some embodiments, the electronic device 600 may perform the action through a trained neural network model. In some embodiments, the trained neural network model may include a first sub-network model, a second sub-network model, and a third sub-network model. In some embodiments, the first sub-network model may be used to map the source image and the keyword into a predetermined feature space. In some embodiments, the second sub-network model may be used to obtain a semantic segmentation map by semantically segmenting the source image. In some embodiments, the third sub-network model may be used to transform the semantic segmentation map using the keyword to obtain a transformed semantic segmentation map and generate a target image.

[0066] In some embodiments, the actions performed by the electronic device 600 may further include: acquiring a first sub-network model by training the first neural network model and the second neural network model. In some embodiments, the first neural network model may be used to map an image into an image feature space. In some embodiments, the second neural network model may be used to map keywords into a text feature space. In some embodiments, training the first neural network model and the second neural network model may include: jointly training the first neural network model and the second neural network model so that the trained first neural network model and the second neural network model jointly map the input image and the input keyword into a predetermined feature space.

[0067] In some embodiments, the action may further include obtaining a trained semantic segmentation model as a second sub-network model. In some embodiments, the trained semantic segmentation model may be different from the predetermined semantic segmentation model. In some embodiments, the action may further include training a third neural network model to obtain a third sub-network model. In some embodiments, the third neural network model may be based on a generative adversarial network architecture. In some embodiments, the third neural network model may include a generator model and a discriminator model. In some embodiments, the generator model may be used to generate an output image based on an input semantic segmentation map and an input keyword. In some embodiments, the discriminator model may be used to determine whether the image is a real-world image. In some embodiments, training the third neural network model may include: jointly training the generator model and the discriminator model so that the trained discriminator model determines an image whose difference with the real-world image is less than a predetermined difference threshold as a real-world image, and causes the trained generator model to generate an output image that satisfies a predetermined condition. In some embodiments, the predetermined condition may include that the output image is determined to be a real-world image by the trained discriminator model. In some embodiments, the predetermined condition may also include: in a predetermined feature space, the distance between the output image and the input keyword is less than a first predetermined distance. In some embodiments, the predetermined condition may further include: in a predetermined feature space, the distance between the output image and the input keyword and the input image is greater than a second predetermined distance. In some embodiments, the predetermined condition may further include: the similarity between the output image and the input image is greater than a predetermined similarity threshold.

[0068] According to the electronic device of the embodiment of the present disclosure, it is possible to generate a target image by using a semantic segmentation map and keywords of a source image, so that the generated target image has target features and is semantically consistent with the source image, thereby being able to generate a high-quality target image in a controllable and customizable manner.

[0069] Figure 7 A schematic block diagram of one example of an example device suitable for implementing embodiments of the present disclosure is illustrated.

[0070] Figure 7 A schematic block diagram of an example device 700 that may be used to implement embodiments of the present disclosure is shown. Figure 1 or Figure 6The electronic device of the present invention can be realized by using the device 700. As shown in the figure, the device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0071] A number of components in the device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc., an output unit 707, such as various types of displays, speakers, etc., a storage unit 708, such as a disk, an optical disk, etc., and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0072] The various processes and processing described above, such as method 200, may be performed by processing unit 701. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by CPU 701, one or more actions of method 200 described above may be performed.

[0073] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0074] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive lists) of computer readable storage medium include portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or convex structures in grooves on which instructions are stored, and any suitable combination thereof. Computer readable storage medium used here is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses by optical fiber cables), or electrical signals transmitted by wires.

[0075] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0076] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0077] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0078] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0080] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0081] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating an image, comprising: Obtain a semantic segmentation map by performing semantic segmentation on the source image; Obtaining keywords for describing features of a target image to be generated; Obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keyword; and Based on the transformed semantic segmentation map, the target image is generated.

2. The method according to claim 1, wherein the source image is an image in a training data set, and the training data set is used for training a predetermined semantic segmentation model.

3. The method according to claim 2, further comprising: Including the target image and the transformed semantic segmentation map in the training dataset to enhance the training dataset, The transformed semantic segmentation map serves as annotation information of the target image.

4. The method according to claim 2, further comprising: Mapping the source image and the keyword into a predetermined feature space, wherein in the predetermined feature space, the distance between the mutually matching images and keywords is less than a first predetermined distance, and the distance between the mutually non-matching images and keywords is greater than a second predetermined distance, and The generating of the target image based on the transformed semantic segmentation map comprises: making the generated target image, when mapped into the predetermined feature space, have a distance between the target image and the keyword smaller than the first predetermined distance, and a distance between the target image and the keyword and the source image larger than the second predetermined distance.

5. The method according to claim 4, wherein generating the target image based on the transformed semantic segmentation map further comprises: The difference between the generated target image and the real-world image is made smaller than a predetermined difference threshold, and the similarity between the generated target image and the source image is made larger than a predetermined similarity threshold. The method of claim 5 , wherein the method is performed by a trained neural network model.

7. The method according to claim 6, wherein the trained neural network model comprises a first sub-network model, a second sub-network model and a third sub-network model, The first sub-network model is used to map the source image and the keyword into the predetermined feature space, The second sub-network model is used to obtain the semantic segmentation map by performing semantic segmentation on the source image, and The third sub-network model is used to transform the semantic segmentation map by using the keyword to obtain the transformed semantic segmentation map and generate the target image.

8. The method according to claim 7, further comprising: The first sub-network model is obtained by training a first neural network model and a second neural network model, wherein the first neural network model is used to map an image into an image feature space, and the second neural network model is used to map a keyword into a text feature space; Acquire a trained semantic segmentation model as the second sub-network model, wherein the trained semantic segmentation model is different from the predetermined semantic segmentation model; and The third sub-network model is obtained by training a third neural network model, wherein the third neural network model is based on a generative adversarial network (GAN) architecture.

9. The method according to claim 8, wherein training the first neural network model and the second neural network model comprises: The first neural network model and the second neural network model are jointly trained so that the trained first neural network model and the second neural network model jointly map the input image and the input keyword into the predetermined feature space.

10. The method according to claim 8, wherein the third neural network model comprises a generator model and a discriminator model, and The generator model is used to generate an output image based on an input semantic segmentation map and an input keyword, and the discriminator model is used to determine whether the image is a real-world image.

11. The method according to claim 10, wherein training the third neural network model comprises: The generator model and the discriminator model are jointly trained so that the trained discriminator model determines an image whose difference with a real-world image is less than a predetermined difference threshold as a real-world image, and the trained generator model generates an output image that satisfies a predetermined condition, The predetermined condition includes that the output image is determined to be a real-world image by the trained discriminator model.

12. The method according to claim 11, wherein the predetermined condition further comprises: In the predetermined feature space, the distance between the output image and the input keyword is less than the first predetermined distance, In the predetermined feature space, the distance between the output image and the input keyword and the input image is greater than the second predetermined distance, and The similarity between the output image and the input image is greater than the predetermined similarity threshold.

13. The method according to claim 1, wherein the source image comprises at least one image, and the target image comprises at least one image corresponding to the source image.

14. An electronic device comprising: at least one processor; as well as a memory coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the electronic device to perform actions, the actions comprising: Obtain a semantic segmentation map by performing semantic segmentation on the source image; Obtaining keywords for describing features of a target image to be generated; Obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keyword; and Based on the transformed semantic segmentation map, the target image is generated. 15 . The electronic device according to claim 14 , wherein the source image is an image in a training data set, and the training data set is used for training a predetermined semantic segmentation model.

16. The electronic device according to claim 15, wherein the actions further comprise: Including the target image and the transformed semantic segmentation map in the training dataset to enhance the training dataset, The transformed semantic segmentation map serves as annotation information of the target image.

17. The electronic device according to claim 15, wherein the actions further comprise: Mapping the source image and the keyword into a predetermined feature space, wherein in the predetermined feature space, the distance between the mutually matching images and keywords is less than a first predetermined distance, and the distance between the mutually non-matching images and keywords is greater than a second predetermined distance, and The generating of the target image based on the transformed semantic segmentation map comprises: making the generated target image, when mapped into the predetermined feature space, have a distance between the target image and the keyword smaller than the first predetermined distance, and a distance between the target image and the keyword and the source image larger than the second predetermined distance.

18. The electronic device according to claim 17, wherein generating the target image based on the transformed semantic segmentation map further comprises: The difference between the generated target image and the real-world image is made smaller than a predetermined difference threshold, and the similarity between the generated target image and the source image is made larger than a predetermined similarity threshold.

19. The electronic device of claim 18, wherein the electronic device performs the action through a trained neural network model.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions which, when executed, cause a machine to: Obtain a semantic segmentation map by performing semantic segmentation on the source image; Obtaining keywords for describing features of a target image to be generated; Obtaining a transformed semantic segmentation map by transforming the semantic segmentation map using the keyword; and Based on the transformed semantic segmentation map, the target image is generated.

Images