Image generation device, image generation method, and program

Abstract

To provide a technique which can generate an edited image from a given image and an editing instruction without using a text representing a state before editing.SOLUTION: An image generation device according to one embodiment of the present disclosure includes: an input unit which is configured to receive a text representing an editing instruction for an image to be edited; and an editing information generation unit which is configured to generate editing information for generating an image resulting from editing the image to be edited in accordance with the editing instruction on the basis of relationship between given reference image information and the text.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device, an image generation method, and a program. [Background technology]

[0002] A method called StyleCLIP is known that can edit input images using natural language (for example, Non-Patent Document 1). StyleCLIP combines a method called CLIP, which can encode images and natural language into the same embedding space, with a method called StyleGAN, which is a type of generative adversarial network (GAN).

[0003] In StyleCLIP, the editing direction of an image in the embedding space is determined from the text representing the editing instructions (also called the target text) and the text representing the state before editing (also called the neutral text), and the image is edited using StyleGAN using that editing direction.In addition, when determining the editing direction, the vector representing the direction from the neutral text to the target text in the embedding space is determined as the editing direction. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Or Patashnik et al. Styleclip: Text-driven manipulation of stylegan imagery. In CVPR, pages 2085-2094, 2021. Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, StyleCLIP requires neutral text in addition to target text when determining the editing direction. However, in StyleCLIP, neutral text is a kind of tuning parameter, and it is often difficult to properly set the text that can reflect the editing instructions.

[0006] The present disclosure has been made in consideration of the above points, and aims to provide a technology that can generate an edited image from a given image and editing instructions without using text that represents the state before editing. [Means for solving the problem]

[0007] An image generating device according to one aspect of the present disclosure has an input unit configured to input text representing editing instructions for an image to be edited, and an editing information generating unit configured to generate editing information for generating an image edited from the image to be edited in accordance with the editing instructions based on the relationship between given reference image information and the text. [Effects of the Invention]

[0008] A technique is provided that can generate an edited image from a given image and editing instructions without using text that represents the state before editing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image generating apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an image generating apparatus according to the present embodiment. [Figure 3] 10 is a flowchart illustrating an example of an image generation process according to the present embodiment. [Figure 4] FIG. 10 is a diagram illustrating a modified example of the functional configuration of the image generating device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will be described below. In the following embodiment, an image generating device 10 will be described that, when given an image and editing instructions (target text), can generate an edited image by editing the image in accordance with the editing instructions. Note that, hereinafter, the image given to the image generating device 10 (the image before editing) will also be referred to as the "input image," and the edited image will also be referred to as the "output image."

[0011] <Example of hardware configuration of image generating device 10> An example of the hardware configuration of an image generation device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the image generation device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0012] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the image generating device 10 does not necessarily have to include at least one of the input device 101 and the display device 102.

[0013] The external I / F 103 is an interface with an external device such as a recording medium 103a. The image generating device 10 can read from and write to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0014] The communication I / F 104 is an interface for connecting the image generating device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0015] The image generating device 10 according to this embodiment has the hardware configuration shown in Fig. 1 and is therefore capable of realizing the image generation process described below. Note that the hardware configuration shown in Fig. 1 is merely an example, and the hardware configuration of the image generating device 10 is not limited to this. For example, the image generating device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware components other than the hardware shown in the figure.

[0016] <Example of functional configuration of image generating device 10> FIG. 2 shows an example of the functional configuration of an image generating device 10 according to this embodiment. As shown in FIG. 2, the image generating device 10 according to this embodiment includes an input unit 201, a language encoding unit 202, an image encoding unit 203, an image search unit 204, an editing direction determination unit 205, an image generating unit 206, and an output unit 207. Each of these units is realized, for example, by a process in which one or more programs installed in the image generating device 10 are executed by a processor 108. The image generating device 10 according to this embodiment also includes an image corpus storage unit 208. The image corpus storage unit 208 is realized, for example, by the auxiliary storage device 107. Note that the image corpus storage unit 208 may also be realized, for example, by a storage device such as a database server connected to the image generating device 10 via a communication network.

[0017] The input unit 201 inputs a given input image and target text. Here, the target text is a text that expresses an editing instruction in a natural language. The input image is an image that includes an object to be edited according to the editing instruction expressed by the target text (i.e., an image that shows the editing object). The editing object is not limited to a specific object, but examples include a human face, an animal such as a dog or cat, or a machine such as a vehicle or ship. In the following, the target text is referred to as t target Let us denote the input image by x.

[0018] The language encoding unit 202 encodes the target text t target Given the input, the target text t target and the text feature e text The language encoding unit 202 generates and outputs the text encoder CLIP (Contrastive Language-Image Pre-training) described in Reference 1, for example. text Using e text =CLIP text (t target ) to target text t target is encoded and the text feature e text However, this is just an example, and the language encoding unit 202 may generate and output CLIP text The target text t is obtained by an encoder other than target may be encoded.

[0019] The image encoding unit 203 receives an input image x, encodes the input image x, and generates and outputs an image feature s representing the feature of the input image x. The image encoding unit 203 may use, for example, an e4e (Encoder for Editing) encoder described in Reference 2 to encode the input image x using the e4e encoder, and generate and output the image feature s. However, this is just one example, and the image encoding unit 203 may also encode the input image x using an encoder other than the e4e encoder. For example, the image encoding unit 203 may use an image encoder CLIP for CLIP described in Reference 1. image Using s=CLIP image (x) may encode the input image x. The image encoder is CLIP image and e4e encoders, but not limited to CLIP image By using encoders such as the e4e encoder, it is possible to expect to generate more accurate output images.

[0020] Both CLIP and e4e encoders are pre-trained to reduce the distance (or increase the similarity) between the text features of a given text and the image features of that image when they are semantically similar. The multimodal embedding space (feature space) generated by such pre-training is also called CLIP space.

[0021] The image search unit 204 searches for the text feature e text is used as input, the image corpus storage unit 208 is searched, and information consisting of a set of positive example image features representing the features of positive example images and a set of negative example image features representing the features of negative example images is output as reference image information.

[0022] Here, a positive example image is an image included in a certain image set (hereinafter referred to as a reference image set) whose feature is equal to or exceeds the text feature e text On the other hand, a negative example image is an image in the reference image set whose feature is not the text feature e textIt is an image that is not in the vicinity. For example, when the reference image set is X = {x i |i = 1, ···, N} (where N is the number of images included in the reference image set X), for each i = 1, ···, N, when the similarity (e.g., cosine similarity, etc.) between CLIP image (x i ) and the text feature vector e text is obtained respectively, the top N1 (where N1 < N) images with high similarity are positive example images, and the bottom N2 images (where N2 < N and N1 + N2 ≤ N) are negative example images. However, this is just an example and not limited to this. For example, images with similarity above a certain predetermined threshold can be used as positive example images, and images with similarity below a certain predetermined threshold can be used as negative example images. More generally, for example, using two thresholds θ1, θ2 (where θ1 ≥ θ2), images with similarity above threshold θ1 can be used as positive example images, and images with similarity below θ2 can be used as negative example images. Note that the similarity is just an example and not limited to this. It is also possible to use other evaluation indicators (e.g., distance, etc.) that represent the relationship with the text feature vector e text .

[0023] Hereinafter, the set of image feature vectors of each image x i included in the reference image set X is S = {CLIP image (x i )|i = 1, ···, N}. Also, the set of positive example images is X + = {x i |x i is a positive example image} ⊂ X, and the set of negative example images is X - = {x i |x i is a negative example image} ⊂ X. Furthermore, the set of image feature vectors of positive example images is S + = {CLIP image (x i )|x i is a positive example image} ⊂ S, and the set of image feature vectors of negative example images is S - = {CLIP image (x i )|x i is a negative example image} ⊂ S. Thus, the reference image information is represented as (S + , S - ).

[0024] The editing direction determination unit 205 determines the reference image information (S + ,S - ) as input, generate and output an edit vector. The edit vector is a vector that indicates the edit direction of the image generated by the image generation unit 206. For example, the image generation unit 206 generates an image by using the Synthesis network of StyleGAN described in Reference 3. Also, let us say that the image generated by the image generation unit 206 from the image feature amount s is G(s), and this image G(s) is the target text t. target The result edited according to the above formula is defined as G(s+αΔs). In this case, αΔs is the edit vector. Here, Δs is the direction of change in StyleGAN's Style space, and α is a hyperparameter that represents the intensity of that change. Style space is a space that represents the multidimensional attributes of a target, such as age, gender, facial expression, eye color, hair color, etc. (Each of these multidimensional attributes is called a channel or style channel, etc.). Below, the number of channels in the Style space is represented by C, and each channel is represented by c (1≦c≦C). In StyleGAN, s is also called style or style information, etc., and represents a value in the Style space (i.e., a C-dimensional vector).

[0025] The value of the change direction Δs in channel c (1≦c≦C) (hereinafter referred to as Δs(c)) is the channel change direction Δi c and the text change direction Δt in CLIP space. Δs, Δi c The calculation method of Δt and Δi will be described later. In StyleCLIP (Non-Patent Document 1), the text change direction Δt is calculated from the target text and the neutral text, but in this embodiment, since the neutral text is not given, the text change direction Δt is calculated from the reference image information without using the neutral text. On the other hand, the channel change direction Δi c can be calculated in the same way as in StyleCLIP (Non-Patent Document 1).

[0026] The image generation unit 206 receives the image feature s and the edit vector αΔs as input, and generates and outputs an output image G(s+αΔs). The image generation unit 206 may generate the output image G(s+αΔs) using, for example, a synthesis network of StyleGAN described in Reference 3.

[0027] The output unit 207 outputs the output image G(s+αΔs) to a predetermined output destination, such as the display device 102, the auxiliary storage device 107, or another device or equipment connected via a communication network.

[0028] The image corpus storage unit 208 stores a set of reference images given in advance and the feature amount (image feature amount) of each image included in the set of reference images. i |i=1,···,N} and its image feature set S={CLIP image (x i )|i=1, ,N} is stored. However, the image corpus storage unit 208 does not necessarily need to store the reference image set X, and may store only the image feature set S. The reference image set X or the image feature set S, or both, may be referred to as "reference information" or the like. Note that the reference image set X and its image feature set S (or the reference image set X) are similar to corpora used in natural language processing and the like, and are therefore sometimes referred to as "image corpora" or the like.

[0029] <Image generation processing> The image generation process according to this embodiment will be described below with reference to FIG.

[0030] The input unit 201 receives an input image x and a target text t target is input (step S101).

[0031] Next, the language encoding unit 202 generates the target text t target is used as input, and the text feature etext is generated and output (step S102).

[0032] Furthermore, the image encoding unit 203 receives the input image x as an input, and generates and outputs an image feature amount s (step S103).

[0033] Following step S102, the image search unit 204 searches for the text feature e text is input, and the image feature set S of the positive example image is selected from the image feature set S stored in the image corpus storage unit 208. + and the set of image features of negative examples S - Search for the reference image information (S + ,S - ) (step S104). The image search unit 204 outputs, for example, the text feature e text The top N1 image features with the highest similarity (for example, cosine similarity) to the positive example image are searched from the image feature set S, and the image feature set S + Similarly, the image search unit 204 may use, for example, the text feature e text Search the image feature set S for the bottom N2 image features with the lowest similarity to the negative example image, and - This can be done as follows.

[0034] However, the above search method is an example, and the image search unit 204 may use other search methods to search for the image feature set S of positive example images. + and the set of image features of negative examples S - For example, the image search unit 204 may search for an image from the reference image set X using a general image search method (for example, the target text t target The images are searched for in the reference image set X using the query, and the images are sorted in order of their search scores. The top N1 images are taken as positive examples, and the bottom N2 images are taken as negative examples. The set of image features corresponding to the positive examples is then denoted as S + , the set of image features corresponding to negative example images is S - The search score is a score that indicates the similarity to the query.

[0035] Next, the editing direction determination unit 205 determines the reference image information (S + ,S - ) is input, an edit vector αΔs is generated and output (step S105). Here, the change direction Δs is calculated by the following steps 1-1 and 1-2.

[0036] Step 1-1: The editing direction determination unit 205 determines |Δi c ·If Δt|≧β, Δs(c)=Δi c Calculate each Δs(c) by Δt, otherwise Δs(c) = 0. Here, β is a hyperparameter introduced to ignore channels that are weakly related to the text change direction Δt. This hyperparameter makes it possible to manipulate only the attributes (channels) you want to edit. Also, Δi c Δt is Δi c and Δt. Note that the channel change direction Δi in the CLIP space c The calculation method of the text change direction Δt in the CLIP space will be described later.

[0037] Step 1-2: The editing direction determination unit 205 normalizes Δs(c) (1≦c≦C) so that the maximum value is 1. That is, the editing direction determination unit 205 normalizes Δs(c)←Δs(c) / max{Δs(c)|c=1, ,C} for each channel c (1≦c≦C). This obtains Δs having Δs(c) as the value of channel c.

[0038] ·Channel change direction Δi in CLIP space c How to calculate Hereinafter, the channel change direction Δi in the CLIP space c This calculation method is the same as that of StyleCLIP (Non-Patent Document 1), so please refer to Non-Patent Document 1 as needed. Below, we will calculate the channel change direction Δi cThis is calculated by the following steps 2-1 to 2-3.

[0039] Step 2-1: The image generator 206 generates an image pair x(s) by applying a perturbation (one standard deviation of channel c) only to channel c of the image G(s). c + =G(s+αΔs c ) and x c - =G(s-αΔs c ) where Δs c is a vector that takes the value of one standard deviation of channel c only for channel c, and takes 0 for other channels.

[0040] Step 2-2: Next, the image encoding unit 203 encodes the image feature pair e c + =CLIP image (x c + ) and e c - =CLIP image (x c - )

[0041] Step 2-3: After repeating the above steps 2-1 and 2-2 for a predetermined number of images G(s) (for example, 100 images), the editing direction determination unit 205 determines e c + -e c - The average of Δi c , that is, Δi c =Ave(e c + -e c - )

[0042] The above steps 2-1 to 2-3 are executed for all channels c (1≦c≦C). As a result, for each channel c (1≦c≦C), the channel change direction Δi c is obtained.

[0043] ·How to calculate the text change direction Δt in CLIP space A method for calculating the text change direction Δt in the CLIP space will be described below. In this embodiment, positive example images and negative example images are separated in the CLIP space (i.e., S + and S - The normal to the boundary surface (separating hyperplane) that separates the text is calculated as the text change direction Δt. Such text change direction Δt is calculated by the following steps 3-1 to 3-2.

[0044] Step 3-1: The editing direction determination unit 205 separates positive example images and negative example images in the CLIP space (S + and S - The boundary surface that separates the two is calculated using a linear SVM (Support Vector Machine).

[0045]

number

[0046]

number

[0047] Step 3-2: Then, the editing direction determination unit 205 sets the normal n=^w to the boundary surface shown in the above equation 1 as Δt. That is, the editing direction determination unit 205 calculates the text change direction Δt using Δt=^w.

[0048] Here, the boundary surface and its normal shown in the above equation 1 are obtained as a solution to the following optimization problem.

[0049]

number

[0050] This makes it possible to obtain the text change direction Δt without using neutral text, unlike StyleCLIP (Non-Patent Document 1).

[0051] Following the above steps S103 and S105, the image generation unit 206 receives the image feature amount s and the edited vector αΔs as input, and generates and outputs an output image G(s+αΔs) (step S106).

[0052] Then, the output unit 207 outputs the output image G(s+αΔs) to a predetermined output destination (step S107). target The edited output image G(s+αΔs) is obtained according to the following equation.

[0053] <Modification> A modified example of this embodiment will be described below. In the above description, it has been assumed that the image corpus storage unit 208 stores at least the image feature set S. However, for example, there may be cases where the image feature set S is not stored and only the reference image set X is stored. Therefore, in this modified example, a case where only the reference image set X is stored in the image corpus storage unit 208 will be described.

[0054] An example of the functional configuration of the image generating device 10 in this modified example is shown in Fig. 4. As shown in Fig. 4, the image search unit 204 of the image generating device 10 in this modified example uses the text feature e text is input, the image corpus storage unit 208 is searched, and a set of positive example images X + and the negative image set X - Then, the image encoding unit 203 obtains the positive example image set X + Each positive example image included in + and the negative example image set X -Each negative example image included in - Then, the image search unit 204 obtains the image feature set S + and S - Please refer to the image information (S + ,S - )

[0055] In the above example, the positive example image set X + and the negative image set X - For example, the image search unit 204 acquires a reference image set X from the image corpus storage unit 208, and also acquires each image x included in the reference image set X. i and text feature e text In this case, the editing direction determination unit 205 (or the image encoding unit 203) may calculate some score (for example, a score representing the distance or similarity) between each image x included in the reference image set X. i The images can be classified into positive and negative examples based on the scores.

[0056] <Evaluation experiment> Below, we will explain the experiments conducted to evaluate the image generation device 10 according to this embodiment and the results of those experiments. Hereinafter, the image generation process executed by the image generation device 10 according to this embodiment will be referred to as the "proposed method."

[0057] <Experimental Settings> Dataset In the proposed method, we used FlickrFaces-HQ (FFHQ) described in Reference 3 as the reference image set X. FFHQ is a dataset consisting of 70,000 face images used for training StyleGAN. FFHQ is characterized by its high resolution, including facial images of various ages and ethnicities, and a wide range of accessories such as glasses and hats.

[0058] In this experiment, we used Multi-Modal-CelebA-HQ, described in Reference 4, as the development dataset for determining hyperparameters and the test dataset for evaluating the proposed method and the comparative method. Multi-Modal-CelebA-HQ is a set of 30,000 high-resolution face images from CelebA-HQ, described in Reference 5, with 10 captions automatically generated for each image using a probabilistic context-free grammar based on facial attribute labels. In this experiment, 100 images were randomly selected from the training dataset of Multi-Modal-CelebA-HQ, and the dataset consisting of these 100 images was used as the development dataset. In addition, 50 images were randomly selected from the test dataset of Multi-Modal-CelebA-HQ, and the dataset consisting of these 50 images was used as the test dataset. Target text t target set a restriction of at least 10 words and randomly selected 10 items for each of the development and test datasets.

[0059] Hyperparameters To determine the appropriate α and β, we compare the images in the development dataset with the target text t target The images were edited using the proposed method and the comparative method. At this time, a grid search was performed for α between 2.0 and 6.0 in increments of 0.5, and for β between 0.1 and 0.2 in increments of 0.05. The generated image (output image) in CLIP space and t target We adopted (α,β) which maximizes the cosine similarity with

[0060] In both the proposed method and the comparative method, α = 6.0 and β = 0.1. Other hyperparameters conform to Non-Patent Document 1.

[0061] Comparative method We used StyleCLIP as the baseline model and compared it with other methods. neutralWe used three types of text: (a) "face," (b) "male face" if the face in the input image was male, and "female face" if it was female, and (c) a detailed caption of the input image. (c) The detailed caption of the input image was randomly selected from the captions of each image in Multi-Modal-CelebA-HQ.

[0062] Qualitative evaluation ·About the influence of neutral text In StyleCLIP, neutral text t neutral The results changed significantly by changing the content of the neutral text. neutral This resulted in stable image generation without the need for a separate image sensor.

[0063] -The influence of hyperparameters In StyleCLIP, excessive editing was observed when α was increased or β was decreased, making it difficult to tune the generated results using α and β. On the other hand, there were few such cases with the proposed method.

[0064] <Subjective evaluation> The results of the subjective evaluation are shown in Table 1 below.

[0065] [Table 1] Note that Ours represents the proposed method.

[0066] In the subjective evaluation, the target text t target Two annotators evaluated the binary classification of whether the gender of the generated image (output image) matched the editing instructions indicated by the target text t, and whether the generated image had a natural face (whether the facial shape and hair were natural). targetIf the subject mentioned gender, that gender was used as the correct answer. If not, the gender of the original face image (input image) was used as the correct answer. The agreement between the two subjects' gender and naturalness annotations (Cohen's kappa coefficient) was 0.92 and 0.63, respectively.

[0067] The proposed method shows higher scores for both gender and naturalness compared to StyleCLIP. target It has excellent editing capabilities according to the model and robust facial image generation capabilities. neutral Since the evaluation results of StyleCLIP used for this are poor, we are looking for an appropriate neutral text to reflect the editing instructions. neutral It turns out that choosing is not easy.

[0068] <Summary> As described above, the image generating device 10 according to this embodiment can generate neutral text t required in conventional methods such as StyleCLIP. neutral Without using the target text t target The input image can be edited naturally according to the above. Note that the image generating device 10 according to this embodiment is configured with pre-trained StyleGAN and CLIP, and therefore does not require training using training data (correct answer data).

[0069] The image generation device 10 according to the present embodiment provides certain improvements over conventional approaches such as StyleCLIP, and represents an advancement in various technical fields where natural language image editing techniques are used.

[0070] Furthermore, the image generating device 10 according to this embodiment can be applied to various technical fields in which image editing technology using natural language is used. For example, in technical fields such as computer vision, the image generating device 10 can be applied to industrially important services such as avatar creation in virtual spaces and communication support.

[0071] The following additional notes are provided regarding the above-described embodiments. (Appendix 1) Memory and at least one processor coupled to said memory; Including, The processor: Enter the text that represents the editing instructions for the image to be edited, an image generating device that generates editing information for generating an image by editing the image to be edited in accordance with the editing instructions, based on given reference image information and a relationship with the text; (Appendix 2) the reference image information is an image set composed of reference images, The processor: The image generating device described in Appendix 1 generates the editing information based on the relationship between the reference image information and the text when, among the reference images included in the image set, reference images that are near the text in feature space are considered positive examples and reference images that are not near the text are considered negative examples. (Appendix 3) The processor: 3. The image generating device according to claim 2, wherein the editing information is generated as a normal vector of a hyperplane that separates the reference images included in the image set into positive examples and negative examples. (Appendix 4) The processor: 4. The image generating device according to claim 2 or 3, wherein, of the reference images included in the image set, a first predetermined number of reference images having a high similarity to the text in the feature space or reference images having a similarity to the text equal to or greater than a first threshold are regarded as positive examples, and a second predetermined number of reference images having a low similarity to the text or reference images having a similarity to the text less than a second threshold are regarded as negative examples. (Appendix 5) the reference image information is a feature set composed of image feature values ​​representing feature values ​​of the reference image, The processor: The image generating device described in Appendix 1 generates the editing information based on the relationship between the reference image information and the text when, among the image features included in the feature set, the reference image having image features that are near the text in feature space is considered a positive example, and the reference image having image features that are not near the text is considered a negative example. (Appendix 6) The processor: 6. The image generating device according to claim 5, wherein the editing information is generated as a normal vector of a hyperplane that separates the image features of the reference image determined as a positive example from the image features of the reference image determined as a negative example. (Appendix 7) Memory and at least one processor coupled to said memory; Including, The processor: An image generating device that receives reference image information for editing an image to be edited as an input and generates editing information for generating an image obtained by editing the image to be edited based on the reference image information. (Appendix 8) The processor: the image to be edited and text representing editing instructions for the image to be edited are input, and a first image that is close to the text and a second image that is not close to the text are acquired from a predetermined image set based on the text, as the reference image information; 8. The image generating device according to claim 7, wherein the acquired reference image information is used as an input and the editing information is generated based on the reference image information. (Appendix 9) Enter the text that represents the editing instructions for the image to be edited, A generation method in which a computer executes a process to generate editing information for generating an image by editing the image to be edited in accordance with the editing instructions, based on the relationship between given reference image information and the text. (Appendix 10) A non-transitory storage medium storing a program that causes a computer to execute the generation method described in Appendix 9.

[0072] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0073] [References] Reference 1: Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021 Reference 2: Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. arXiv:2102.02766 [cs.CV] Reference 3: Tero Karras et al. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019. Reference 4: Weihao Xia et al. Tedigan: Text-guided diverse face image generation and manipulation. In CVPR, pages 2256-2265, 2021. Reference 5: Tero Karras et al. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. [Explanation of symbols]

[0074] 10 Image generation device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Input section 202 Language encoding section 203 Image Encoding Unit 204 Image Search Section 205 Editing direction determination unit 206 Image Generation Unit 207 Output section 208 Image Corpus Storage Unit 209 Editing Information Generation Unit

Claims

1. an input unit configured to input text representing editing instructions for the image to be edited; an editing information generation unit configured to generate reference image information, which is a set of images configured of reference images that are positive examples for the text and reference images that are negative examples for the text, and editing information for generating an image obtained by editing the image to be edited in accordance with the editing instructions, based on a relationship with the text; An image generating device having:

2. The editing information generation unit 2. The image generating device according to claim 1, wherein, among the reference images included in the image set, reference images that are near the text in a feature space are defined as positive examples, and reference images that are not near the text are defined as negative examples, the image generating device is configured to generate the editing information based on a relationship between the reference image information and the text.

3. The editing information generation unit The image generating device according to claim 2 , configured to generate, as the editing information, normal vectors of a hyperplane that separates the reference images included in the image set into positive examples and negative examples.

4. The editing information generation unit 4. The image generation device according to claim 2 or 3, wherein, of the reference images included in the image set, a first predetermined number of reference images having a high similarity to the text in the feature space or reference images having a similarity to the text equal to or greater than a first threshold are treated as positive examples, and a second predetermined number of reference images having a low similarity to the text or reference images having a similarity to the text less than a second threshold are treated as negative examples.

5. the reference image information is a feature set composed of image feature values ​​representing feature values ​​of the reference image, The editing information generation unit 2. The image generating device according to claim 1, wherein the reference image having an image feature that is near the text in a feature space among the image features included in the feature set is defined as a positive example, and the reference image having an image feature that is not near the text is defined as a negative example, and the image generating device is configured to generate the editing information based on a relationship between the reference image information and the text.

6. The editing information generation unit The image generating device according to claim 5, configured to generate, as the editing information, a normal vector of a hyperplane that separates image features of the reference image determined as a positive example from image features of the reference image determined as a negative example.

7. an editing information generation unit configured to receive as input reference image information, which is a set of images including reference images that are positive examples for text representing editing instructions for the image to be edited and reference images that are negative examples for the text, and to generate editing information for generating an image obtained by editing the image to be edited based on the reference image information; An image generating device having:

8. an acquisition unit configured to receive the image to be edited and the text as input, and acquire, from a predetermined image set based on the text, a first image that is close to the text and a second image that is not close to the text as the reference image information; The editing information generation unit The image generating device according to claim 7 , configured to receive the reference image information acquired by the acquisition unit as an input and generate the editing information based on the reference image information.

9. an input step of inputting text representing editing instructions for the image to be edited; an editing information generation step of generating reference image information, which is a set of images including reference images that are positive examples for the text and reference images that are negative examples for the text, and editing information for generating an image obtained by editing the image to be edited in accordance with the editing instructions, based on a relationship with the text; A computer-implemented image generation method.

10. an input step of inputting text representing editing instructions for the image to be edited; an editing information generation step of generating reference image information, which is a set of images including reference images that are positive examples for the text and reference images that are negative examples for the text, and editing information for generating an image obtained by editing the image to be edited in accordance with the editing instructions, based on a relationship with the text; A program that causes a computer to execute the following.

Images