Image generation method

Abstract

To generate an image that closely resembles the image intended by the user, without burdening the user. [Solution] The program of the present invention causes a computer to perform the following steps: receiving instruction information for generating an image; acquiring an image generated based on control information corresponding to the input instruction information; calculating the similarity between the feature quantities of the generated image and the feature quantities of the control information for each of a plurality of evaluation axes; correcting the control information if the calculated similarity for each evaluation axis is less than a predetermined value; and acquiring an image generated based on the corrected control information.

Description

Background Art

[0001] In recent years, with the development of deep learning technology, image generation AI technology has made significant progress. Image generation AI is based on various generation models such as GAN (Generative Adversarial Network), VAE (Variational Autoencoder), and Diffusion Model, each having different characteristics.

Prior Art Documents

Non-Patent Documents

[0002]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, in any of the generation models, it is still very difficult to accurately capture the user's intention. For example, when a user generates an image of the head with AI to show the hairstyle they envision to a hairdresser, even if they devise text information (prompt) by themselves to generate the image, it is currently quite difficult to obtain an image that matches their vision. Moreover, apart from whether the image matches the vision, it is not uncommon for the generated image to feel unnatural. This is thought to be because it is difficult to accurately reflect the user's senses in the prompt, and since the face and the surrounding hairstyle are things that are frequently seen in daily life, from the perspective of image data, even a slight difference from the real object (such as a real photographic image) in the details of the shapes, colors, textures, etc. that are normally visually recognized tends to be perceived by the user as a strong sense of discomfort.

[0004] Typically, if the generated image is unsatisfactory, the user would try modifying the prompt text through trial and error and generate the image again. However, for example, with hairstyle images, due to the circumstances described above, it is often the case that a satisfactory image cannot be generated even after repeated attempts. In fact, if such effort does not result in an improved output image, users often give up on generating the image altogether. On the other hand, it is also possible to generate multiple images at once and present them to the user, allowing them to select the image that best matches their image. However, even if multiple images are generated indiscriminately, there is no guarantee that the generated image will be closer to the user's image, as it is impossible to objectively evaluate whether the generated image is appropriate (whether it reflects the user's intention).

[0005] The present invention aims to generate images that closely resemble the user's intended image without burdening the user. [Means for solving the problem]

[0006] In one embodiment, the present invention provides a program for causing a computer to perform the following steps: receiving instruction information for generating an image; acquiring an image generated based on control information corresponding to the input instruction information; calculating the similarity between the feature quantities of the generated image and the feature quantities of the control information for each of a plurality of evaluation axes; correcting the control information if the calculated similarity for each evaluation axis is less than a predetermined value; and acquiring an image generated based on the corrected control information. In a preferred embodiment, the instruction information and the evaluation axis relate to hairstyles, and the acquired image is an image representing a human head. In a preferred embodiment, the evaluation axis includes at least one of the following: naturalness, volume, color, texture, and realism. In a preferred embodiment, the instruction information is text information, and at least a portion of the plurality of evaluation axes are automatically generated based on the text information. In a preferred embodiment, if the similarity for each evaluation axis is less than a predetermined value, the control information is modified and the image is reacquired repeatedly until the calculated similarity satisfies a predetermined convergence condition. In a preferred embodiment, if the calculated similarity does not converge even after the correction of the control information and the reacquisition of the image are repeated a predetermined number of times, the image with the highest overall similarity evaluation for each evaluation axis is output. In a preferred embodiment, in modifying the control information, the evaluation axis to be adjusted is determined from among the plurality of evaluation axes based on the relative similarity or convergence state of each evaluation axis. In a preferred embodiment, if the similarity of one evaluation axis falls within a predetermined range for a predetermined number of consecutive times, that evaluation axis is excluded from the adjustment target. [Effects of the Invention]

[0007] According to the present invention, it is possible to generate an image that closely resembles the image intended by the user without burdening the user. [Brief explanation of the drawing]

[0008] [Figure 1] Functional block diagram of the information processing device 10 [Figure 2] Example of operation in the information processing device 10 [Figure 3A] Detailed explanation of the CLIP evaluation process [Figure 3B] Detailed explanation of the convergence determination process [Figure 4A] Examples of evaluation criteria [Figure 4B] Examples of evaluation criteria [Figure 4C] Examples of evaluation criteria [Figure 4D] Examples of evaluation criteria [Figure 4E] Examples of evaluation criteria [Figure 5] Examples of information entered by the user [Figure 6] Example of a prompt when generating an evaluation axis based on control information from a user [Figure 7] Example of a generated evaluation axis [Figure 8] Schematic diagram of an information processing system 1 according to a modified example [Figure 9] Functional block diagram of the information processing system 1 [Mode for Carrying Out the Invention]

[0009] Hereinafter, modes for carrying out the present invention will be described in detail. Here, an example is described in which the information processing apparatus 10 is installed in a beauty salon and operated by a customer and / or a hairdresser, and the customer or the hairdresser inputs text information using the information processing apparatus 10 in order to show the hairdresser or the customer an image of the hairstyle after cutting that the customer or the hairdresser imagines, and the information processing apparatus 10 is used to generate an image of hair (only the hair portion, or the face, head, upper body, etc. including the hair) based on this information. The above-mentioned "image" is a concept that includes moving images in addition to still images, and there is no limitation on the image format or the like.

[0010] FIG. 1 shows a functional block diagram of the information processing apparatus 10. The information processing apparatus 10 is implemented as a computer device such as a general-purpose desktop PC, server, tablet terminal, or mobile terminal. It includes a processor (CPU, GPU, etc.), a memory, and devices for inputting and outputting information (a keyboard, a touch panel, a mouse, a display, a communication interface). The memory stores programs that are executed by the processor to implement the functions of the information processing apparatus 10 described later, as well as algorithms and parameters used in various functional units.

[0011] As shown in FIG. 1, the system of the present invention is functionally configured from a condition input unit 170, an initial setting unit 110, an image generation unit 120, a region detection unit 130, an evaluation unit 140, a convergence determination unit 150, and an adjustment unit 160.

[0012] The condition input unit 170 is implemented as a user interface including an input device. The input device is used for inputting instructions and data from a user, and includes a keyboard, a mouse, a touch panel, and the like. Figure 5 is an example of information input by a user. Using the condition input unit 170, the user can input text information such as a text sentence devised by the user regarding the hairstyle the user envisions (such as "hairstyle that looks young and suits a round face") and one or more keywords (such as "40s", "female", "volume feeling"). Alternatively, a plurality of input items and options for each item are predefined in the information processing apparatus 10, and the condition input unit 170 presents these to the user using a user interface of a pull-down format or the like, and may allow the user to select one from the plurality of options for each item. In short, any system is acceptable as long as the user specifies (selects or inputs) text-based (in other words, expressed as character information) conditions.

[0013] The initial setting unit 110 defines a plurality of evaluation axes, and sets conditions and parameters used in evaluation of a generated image and image generation and regeneration (described later), such as a target value, importance, and convergence threshold for each evaluation axis. Specifically, parameters used when initially generating an image corresponding to the information input by the user based on a predetermined algorithm (information obtained by converting text information into a format processable by an image generation AI, such as a prompt sentence or a conditional vector), parameters used for evaluating the generated image, parameters used when determining whether to execute image regeneration, and parameters used when finely adjusting control information when performing image regeneration are determined.

[0014] Examples of parameters are shown below. ·Evaluation axes (e.g., "naturalness", "volume feeling", "tone", "texture") ·Target and importance (priority, weighting value) for each evaluation axis ·Convergence threshold for each evaluation axis (e.g., when expressed as a numerical value ranging from 0 (completely dissimilar) to 1 (completely similar), such as 0.85 or 0.9) • The maximum number of iterations for image generation (e.g., "10 times"; to prevent infinite loops and ensure that image regeneration is not repeated indefinitely). • Initial value of the step coefficient related to the adjustment of the conditional vector (to adjust the convergence speed) Figures 4A to 4E show examples of evaluation axes and corresponding text representations. Here, five evaluation axes are generated, but the content and number of evaluation axes are arbitrary. Furthermore, when determining the evaluation axes, one or more axes that match the information entered by the user may be selected from a pre-registered set of evaluation axes, or the evaluation axes may be freely determined without such constraints. In short, at least some of the multiple evaluation axes are automatically generated based on the aforementioned text information.

[0015] The method for generating evaluation criteria based on user input is optional. For example, it may use LLM (Large-Scale Language Model) or other machine learning techniques, or it may be rule-based. Figure 6 shows an example of prompts used when generating evaluation axes and text representations (positive and negative text representations) for each evaluation axis based on control information from the user, using LLM. Figure 7 shows examples of the generated evaluation axes and textual representations.

[0016] The control information generation unit 121 generates information (such as conditional vectors (prompts or vector representations); hereinafter referred to as "control information") as commands for controlling the image generation unit 120, based on the information supplied from the initial setting unit 110. The generated control information is supplied to the image generation unit 120. The generated control information may or may not be presented to the user.

[0017] The image generation unit 120 generates an image based on control information supplied from the control information generation unit 121, according to a predetermined image generation model. Various models can be used as the generation model, such as GAN (Generative Adversarial Network), VAE (Variational Autoencoder), and Diffusion Model. When using a Diffusion Model, it is preferable to fix the noise used in each of the multiple image generation processes. The image generation unit 120 can also cooperate with an external image generation service as needed. For example, control information (and adjusted control information) may be sent to an existing image generation AI server via a communication network, and the generated (regenerated) image may be obtained from the image generation AI server via the communication network.

[0018] The region detection unit 130 detects a specific region (in this example, the hair region) from the generated image. Specifically, known region detection techniques such as segmentation models can be used.

[0019] The evaluation unit 140 calculates the similarity between the generated image features and the target values ​​for each evaluation axis. Specifically, it can use feature extraction techniques such as CLIP (Contrastive Language-Image Pre-training) and similarity calculation methods such as cosine similarity. CLIP is a learning model trained to associate text and images, and it learns to place images and their corresponding text descriptions close together in the feature space using image-text pair data. Specifically, it takes an image as input, extracts a feature vector representing its content, takes text as input, extracts a feature vector representing its semantic content, and calculates the similarity between the extracted image features and text features in the feature space to quantitatively evaluate the relationship between the image and text.

[0020] The convergence determination unit 150 determines, for each evaluation axis, whether the similarity exceeds a predetermined threshold according to a predetermined determination algorithm. Specifically, for each evaluation axis, if the similarity is below the threshold set for that axis, it is determined that the data has not converged. If the similarity exceeds the threshold for all evaluation axes, it is determined that the data has converged overall (i.e., the target image has been obtained). Furthermore, the convergence determination unit 150 determines whether all evaluation axes satisfy the convergence conditions or whether the preset maximum number of iterations has been reached.

[0021] The adjustment unit 160 adjusts the control information (conditional vector) for each evaluation axis that has not converged, according to a predetermined adjustment algorithm. The adjusted control information is output to the image generation unit 120, and image generation is performed again.

[0022] (1) Setting the evaluation axis vector First, the evaluation axis vector

number

[0023] (2) Aggregation of text vectors Next, the text vectors are aggregated as follows.

number

number

[0024] (3) Adjustment of the condition vector Examples of adjusting the condition vector are as follows: (i) Adjustment method 1 (Multi-axis linear interpolation adjustment)

number

[0025] (ii) Adjustment method 2 (similarity-based adaptive adjustment)

number

[0026] (iii) Adjustment method 3 (threshold-based selective adjustment)

number

[0027] In any of the three methods described above, in a preferred embodiment, only evaluation axes with a similarity below a predetermined threshold are subject to adjustment. That is, based on the relative similarity of each evaluation axis, it is determined whether to include it in the adjustment process (exclude it from adjustment) or to determine the priority (order) of the adjustments. Furthermore, the update history of each evaluation axis may be stored, and based on the convergence status (similarity history) of multiple past convergences, it may be determined whether to include or exclude the evaluation axis from adjustment. For example, an evaluation axis that has converged for a predetermined number of consecutive times in the past generation of control information (adjustment of evaluation axes) (e.g., the similarity of that evaluation axis falls within a predetermined numerical range) is judged to be stably converged and is excluded from adjustment thereafter. This is because it is not expected that the similarity of such evaluation axes will improve significantly even if they are adjusted. In this way, by introducing the concept of convergence stability as a concept separate from the magnitude of similarity, and by setting a priority for the evaluation axis adjustments based on this stability, efficient adjustment of evaluation axes is performed, the convergence speed for all evaluation axes is improved, and as a result, the time required to output the final image is shortened.

[0028] <Operation> Figure 2 shows an example of operation in the information processing device 10.

[0029] <Initial setup process (S101)> In the condition input unit 170 and the initial setting unit 110, multiple evaluation axes are defined, and target values, importance levels, convergence thresholds, etc., are set for each evaluation axis.

[0030] <Image generation and image evaluation processing (S102)> In the image generation unit 120, processes S301 and S302 are executed to generate an image using control information (conditional vectors) supplied from the control information generation unit 121. Next, in the region detection unit 130, process S303 is executed to detect a specific region (e.g., hair region) from the generated image. Note that S303 may be omitted. Next, in the evaluation unit 140, processes S304 and S305 are executed to calculate the similarity between the feature quantities of the generated image and the target values ​​for each evaluation axis. Specifically, this is done as follows.

[0031] Figure 3A shows a specific example of image evaluation using CLIP in S304 and S305. In S601, one evaluation axis is selected. In S602 and S603, the positive and negative text groups are determined for each selected evaluation axis. In S604 and S605, text encoding processing is performed. In S606 and S607, vector aggregation is performed for each positive and negative text group. In S608 and S609, a similarity function is applied to calculate the similarity, and in S610 and S611, these are integrated to calculate the result for the selected evaluation axis. This series of processes is performed for all selected evaluation axes.

[0032] <Convergence determination process (S103)> In the convergence determination unit 150, in S103, it is determined whether the similarity exceeds the convergence threshold for each evaluation axis. A concrete example will be explained using Figure 3B. First, for each evaluation axis, it is determined whether or not it has converged based on a predetermined algorithm (S401), and its convergence state (similarity) is stored (S402). Next, it is determined whether or not that evaluation axis is in a state of stable convergence (S403). Then, it is determined whether or not all evaluation axes are in a state of stable convergence (S405). If all evaluation axes are in a state of stable convergence (S405: YES), the process proceeds to S505; if at least one evaluation axis is not in a state of stable convergence (S405: NO), the process proceeds to S501.

[0033] <Adjustment process (S104)> If there are evaluation axes that have not converged (S103:NO), the adjustment unit 160 selects the evaluation axis that has not converged (S501) and adjusts the conditional vector related to the selected evaluation axis (S502). The adjustment unit 160 uses the adjusted conditional vector to generate an image and evaluate that image again (S502). Subsequently, the convergence determination unit 150 determines whether the number of adjustments to the conditional vector has reached the maximum number of iterations (S503). On the other hand, if the maximum number of iterations has not been reached (S503: NO), image generation is performed again using the adjusted condition vector (S504). If the maximum number of iterations has been reached (S503: YES) or if all evaluation axes satisfy the convergence conditions (S103: YES), one condition vector (and one image generated based on it) is selected from among the multiple generated condition vectors (and multiple images generated based on it) and output (S505). Here, if convergence is achieved, the most recently generated condition vector (and the image generated based on it) is selected. However, if convergence has not been achieved even after reaching the maximum number of iterations, the image with the best overall evaluation of the similarity of each evaluation axis (an index obtained by performing predetermined statistical processing on the mean and other multiple similarity metrics) is selected from among the multiple generated condition vectors (and the corresponding generated images). This is because, if convergence has not been achieved, the last generated image is not necessarily the most desirable (i.e., has the highest similarity).

[0034] According to the above embodiment, control information is generated based on conditions specified by the user, and an image is automatically generated using this conditional vector as control information. Then, the feature quantities of the generated image are calculated, the similarity to the target value is determined from the perspective of evaluation axes, and the conditional vector is automatically adjusted and the image playback is repeated until the similarity meets a certain level. This eliminates the burden on the user of having to check each generated image and revise the prompt text. Furthermore, it increases the probability that the image will be modified in a direction that is objectively close to the user's intention. In other words, it is expected that an image close to the user's intention will be generated.

[0035] Each functional unit of the information processing device 10 shown in Figure 1 may be distributed and implemented across multiple nodes (hardware). In that case, each node is connected to each other via a network so that they can communicate with one another, and the whole system functions as a single information processing system.

[0036] Figure 8 is a schematic diagram of the modified information processing system 1. The information processing system 1 consists of a user terminal 20, an information processing device 10A, an image generation AI server 30, and an evaluation axis generation AI server 40, all connected via a communication network 90. Figure 9 is a functional block diagram of the information processing system 1. As shown in the figure, the user terminal 20 is a tablet-type information processing terminal operated by a user, for example, and is equipped with an input unit, a display unit, and a communication unit, and has a condition input unit 170. The evaluation axis generation AI server 40 has the function of a control information generation unit 121. The image generation AI server 30 has the function of an image generation unit 120.

[0037] Furthermore, in this invention, the content of the generated images, their purpose, and their uses are arbitrary. For example, in the construction industry, it could be used to generate images of the interior or exterior of a planned building that a user envisions, in order to confirm the interior or exterior before construction begins. Alternatively, it can be applied to generating images to visualize the finished state when designing and coordinating not only hairstyles but also clothing.

[0038] In short, the information processing system according to the present invention, comprising one or more information processing devices, is sufficient if it includes the steps of: receiving input instruction information for generating an image; acquiring an image generated based on control information corresponding to the input instruction information; calculating the similarity between the feature quantities of the generated image and the feature quantities of the control information for each of a plurality of evaluation axes; correcting the control information if the calculated similarity for each evaluation axis is less than a predetermined value; and acquiring an image generated based on the corrected control information. [Explanation of symbols]

[0039] 1: Information Processing System 10, 10A: Information Processing Device 20: User terminal 30: Image generation AI server 40: AI Server for Generating Evaluation Axis 90: Communication Network 110: Initial setting section 120: Image generation unit 121: Control Information Generation Unit 130: Region detection unit 140: Evaluation Department 150: Convergence determination unit 160: Adjustment section 170: Condition Input Section

Claims

1. On the computer, A step of receiving input of instruction information for generating an image, The steps include: acquiring an image generated based on control information corresponding to the input instruction information; The steps include calculating the similarity between the features of the generated image and the features of the control information for each of the multiple evaluation axes, If the similarity for each of the calculated evaluation axes is less than a predetermined value, the control information is modified. The steps include obtaining an image generated based on the modified control information and A program to execute.

2. The aforementioned instruction information and evaluation axis relate to hairstyles. The image to be acquired is an image representing a human head. The program according to claim 1.

3. The aforementioned evaluation axis includes at least one of the following: naturalness, volume, color, texture, and realism. The program according to claim 2.

4. The aforementioned instruction information is text information. At least some of the aforementioned evaluation axes are automatically generated based on the text information. The program according to claim 1.

5. If the similarity for each evaluation axis is less than a predetermined value, the control information is modified and the image is reacquired until the calculated similarity satisfies a predetermined convergence condition. The program according to claim 1.

6. If the calculated similarity does not converge even after the correction of the control information and the reacquisition of the image are repeated a predetermined number of times, the image with the highest overall similarity evaluation for each evaluation axis is output. The program according to claim 1.

7. In modifying the control information, the evaluation axis to be adjusted is determined based on the relative similarity or convergence state of each evaluation axis among the multiple evaluation axes. The program according to claim 5.

8. If the similarity of one evaluation axis falls within a predetermined range for a predetermined number of consecutive times, that evaluation axis is excluded from the adjustment process. The program according to claim 7.

9. A means for receiving input of instruction information for generating an image, Means for acquiring an image generated based on control information corresponding to the input instruction information, A means for calculating the similarity between the features of the generated image and the features of the control information for each of the multiple evaluation axes, If the similarity for each of the calculated evaluation axes is less than a predetermined value, means for correcting the control information, Means for acquiring an image generated based on the modified control information and An information processing system having

10. In one or more information processing devices that constitute an information processing system, A step of receiving input of instruction information for generating an image, The steps include: acquiring an image generated based on control information corresponding to the input instruction information; The steps include calculating the similarity between the features of the generated image and the features of the control information for each of the multiple evaluation axes, If the similarity for each of the calculated evaluation axes is less than a predetermined value, the control information is modified. The steps include obtaining an image generated based on the modified control information and An image generation method having the following characteristics.

Images