Information processing program, information processing method, and information processing apparatus

Abstract

To provide an information processing program, an information processing method, and an information processing apparatus configured to shorten the time required for image generation.SOLUTION: A program causes a computer to execute a process including: selecting some modules from a plurality of modules to be applied to a trained machine learning model that performs image generation by removing noise from random noise up to a final stage out of a plurality of stages; generating a first image by synthesizing the selected modules and performing noise removal from predetermined random noise to a middle stage before the final stage; and generating a second image by performing noise removal from the first image a predetermined number of times for each module included in the plurality of modules.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Image generation using a diffusion model is attracting attention as an image generation technology using AI (Artificial Intelligence). A diffusion model is an image generation model that generates an image by denoising an image of random noise according to prompts, which are conditional text for image generation.

[0003] Diffusion models are large-scale models, so fine-tuning (FT) of the entire diffusion model requires high-performance hardware and a large amount of computational resources. Therefore, it is necessary to reduce the size of the model to be fine-tuned. Therefore, instead of tuning the entire original diffusion model, research is being conducted on methods that prepare specific or additional layers of the diffusion model (Parameter-Efficient Fine-Tuning (PEFT)). PEFT methods include Adapter and Low-Rank Adaptation (LoRA). These specific or additional layers are called modules.

[0004] Modules are trained for specific applications; for example, there are modules that have been trained to process paintings in a colored style, such as watercolors or anime paintings. By replacing modules, the diffusion model can acquire generative capabilities that correspond to the modules for specific applications. When the modules are returned to their original state, the diffusion model's generative capabilities for the specific task return to their original capabilities.

[0005] However, once a module has been fine-tuned for a specific application, the influence of the prompts learned in the original diffusion model is reduced, making it much more difficult to control with prompts. However, users often do not know which module is most suitable for their needs. Therefore, when fine-tuning a diffusion model using PEFT, users search for the module that can produce the output closest to the desired image. Therefore, it is possible to group modules with high similarity based on user feedback on the output of each module when multiple modules are used. Once appropriate grouping is achieved, it is possible to prioritize and present modules from the group according to the user's needs for new subsequent tasks.

[0006] As a technique for training an image generation model, a technique has been proposed in which an image generation model is trained using SSIM (Structural Similarity) together with mean square error. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2023-7107 Summary of the Invention [Problem to be solved by the invention]

[0008] However, it takes a certain amount of time to generate an image using a diffusion model by replacing modules. Calculating similarity also requires a lot of time and effort. Therefore, selecting highly similar modules takes a lot of time, and simply grouping highly similar modules based on user feedback on the output of multiple modules is difficult to achieve. Therefore, it is difficult to improve the learning efficiency of a diffusion model, and as a result, it is difficult to reduce the time required for image generation.

[0009] Furthermore, the technology that uses SSIM together with mean squared error to train an image generation model does not take into account grouping of models based on similarity, making it difficult to improve the learning efficiency of the diffusion model and reduce the time required for image generation.

[0010] The disclosed technology has been made in view of the above, and aims to provide an information processing program, an information processing method, and an information processing device that reduce the time required for image generation work. [Means for solving the problem]

[0011] In one aspect of the information processing program, information processing method, and information processing device disclosed in the present application, a computer is caused to select several modules from a plurality of modules applied to a trained machine learning model that generates an image by performing noise removal from random noise up to a final stage among a plurality of stages, combine the selected modules, and perform noise removal from predetermined random noise up to an intermediate stage before reaching the final stage to generate a first image, perform noise removal from the first image a predetermined number of times for each module included in the plurality of modules to generate a second image, and classify the modules included in the plurality of modules based on the second image for each module. [Effects of the Invention]

[0012] In one aspect, the present invention can reduce the time required for image generation work. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating image generation using a diffusion model. [Figure 3] FIG. 3 is a diagram illustrating learning by adding a module. [Figure 4]FIG. 4 is a diagram for explaining the similarity of the outputs. [Figure 5] FIG. 5 is a flowchart of the module clustering process. [Figure 6] FIG. 6 is a flowchart of the module proposal process. [Figure 7] FIG. 7 is a diagram illustrating the hardware configuration of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of an information processing program, an information processing method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the information processing program, the information processing method, and the information processing device disclosed in the present application are not limited to the following embodiments. [Example]

[0015] Fig. 1 is a block diagram of an information processing device according to an embodiment. As shown in Fig. 1, the information processing device 1 according to this embodiment includes a synthetic diffusion model generation unit 11, a data storage unit 12, a first image generation unit 13, and a module application unit 14. Furthermore, the information processing device 1 includes a second image generation unit 15, an FID (Frechet Inception Distance) calculation unit 16, a clustering unit 17, an input / output device 18, and a module proposal unit 19.

[0016] The data storage unit 12 is a storage device and stores a module group 201 and a base model 200.

[0017] The base model 200 is a trained diffusion model that is the basis for fine-tuning using modules.

[0018] Here, the diffusion model will be described. Figure 2 is a diagram showing image generation using the diffusion model. The diffusion model is given conditioning text for image generation in advance. The diffusion model is a machine learning model that repeatedly performs denoising to remove noise from random noise P, ultimately generating a desired image 22.

[0019] Processing based on the diffusion model involves a diffusion process in the direction of arrow D2 and a de-diffusion process in the direction of arrow D1. The de-diffusion process is the image generation process.

[0020] The diffusion process gradually adds Gaussian noise to the image 22. In the diffusion process, noise is simply added to the image 22. The de-diffusion process repeats the de-noising process, gradually removing the noise, to finally produce a clear image 22.

[0021] The diffusion model learns which noises to remove and reverses the diffusion process. The trained diffusion model predicts the image that will be generated for new random noise P and prompts, generating the final image 22.

[0022] Returning to Figure 1, the explanation will continue. The module group 201 includes a large number of modules. Modules intended to perform processing for specific purposes are collected in the module group 201. Examples of processing for specific purposes include processing for coloring in accordance with a painting style in image generation, and processing for adding new concepts such as people and objects.

[0023] A module is a layer of a diffusion model. A module is created using PEFT (Parameter-Efficient Fine-Tuning), which tunes the parameters of a layer of the diffusion model that is the target of the module. Among the methods for fine-tuning a diffusion model using PEFT, there are several methods for inserting a module into a diffusion model and applying it to a new task. In this embodiment, a method called Low-Rank Adaptation (LoRA) is used. Other methods for fine-tuning a diffusion model using PEFT include Adapter, Parallel Bottleneck Adapter, and (IA)^3.

[0024] LoRA is a resource-saving technique that uses a low-rank matrix to represent the weight matrix. In LoRA, the weight matrix is ​​decomposed into two low-rank matrices and a scaling hyperparameter. This decomposition can be applied to specific parameter groups. For example, it is applied to the linear projection part of the attention of each transformer layer in a diffusion model. The final model is then obtained by adding the weights of the low-rank matrix to the weights of the original model (pretrained weights) in parallel. The weights of the original model are also called model parameters. The process of adding the weights of the low-rank matrix to the weights of the original model in parallel corresponds to adding a module. Furthermore, training the weights of the low-rank matrix corresponds to training a module. This reduces the computational load while maintaining the performance of the diffusion model.

[0025] FIG. 3 illustrates learning by adding a module. Here, a LoRAs repository 101 contains multiple LoRAs that add various concepts, such as a bag, a toy chick, and Mr. P. When a user fine-tunes a diffusion model, the user selects a LoRA module 102 from the LoRAs repository 101 that adds Mr. P as the desired concept. The user then fine-tunes the diffusion model by adding the weights of the low-rank matrix represented by module 102 to the weights of the original diffusion model. The fine-tuned diffusion model can generate various images 104 that include the new concept Mr. P.

[0026] The composite diffusion model generation unit 11 acquires a plurality of modules from the module group 201 from the data storage unit 12. The composite diffusion model generation unit 11 also acquires the base model 200 from the data storage unit 12.

[0027] The combined diffusion model generation unit 11 then combines the acquired multiple modules and applies the combined diffusion model to the base model 200 to generate a combined diffusion model. Here, the combined diffusion model generation unit 11 combines the weights of the modules to combine the modules. Specifically, the combined diffusion model generation unit 11 calculates the average of the sum of the weights of each module and generates a combined diffusion model using the calculated value as the weight. In the case of LoRA, the combined diffusion model generation unit 11 averages the weights attached to the edges between LoRA neurons.

[0028] However, it is unclear whether the size and rank of each module are uniform. If the size and rank of each module are not uniform, it is difficult to calculate the sum of weights. Therefore, it is preferable that the combined diffusion model generation unit 11 combines modules after making the size and rank of each module uniform, for example, by adding up the most numerous modules of the same size or rank. Alternatively, the combined diffusion model generation unit 11 may generate a combined diffusion model by combining modules of the same size as much as possible.

[0029] In this embodiment, a case where a plurality of modules are combined is exemplified, but instead of combining a plurality of modules, one module may be selected and processed.

[0030] The composite diffusion model generation unit 11 outputs the generated composite diffusion model to the first image generation unit 13.

[0031] The first image generation unit 13 receives an input of the synthetic diffusion model from the synthetic diffusion model generation unit 11. Then, the first image generation unit 13 generates a first image by performing denoising on random noise with a fixed seed a predetermined number of times using the synthetic diffusion model. If an image that satisfies the requirements can be generated by 50 denoising attempts, the first image generation unit 13 may perform denoising 45 to 47 times, for example. Thereafter, the first image generation unit 13 outputs the generated first image to the second image generation unit 15.

[0032] Here, denoising at the beginning of the diffusion process against random noise makes it difficult to extract the characteristics of each module. In contrast, denoising at the end of the diffusion process brings the image closer to the final one, making the differences in module characteristics clearer. Therefore, if the goal is simply to measure the similarity between modules and perform clustering, denoising near the final step of the diffusion process is important.

[0033] Therefore, the first image generation unit 13 generates an initial image using a synthetic diffusion model including the characteristics of various modules, and performs denoising on the first image as the initial image to generate a second image, making it easier to obtain the characteristics of each module. Note that the first image does not necessarily have to be the initial image.

[0034] Furthermore, the first image generation unit 13 generates first images using random noise with different seeds for multiple seeds, and outputs the multiple first images generated from the random noise with different seeds to the second image generation unit 15.

[0035] The module application unit 14 acquires the base model 200 from the data storage unit 12. Next, it selects and acquires one module from the module group 201 and applies it to the base model 200 to generate a diffusion model. Thereafter, the module application unit 14 outputs the diffusion model to which the selected module has been applied to the second image generation unit 15.

[0036] The module application unit 14 selects modules one by one from the module group 201, applies the modules to the base model 200 described above, and outputs the diffusion models to the second image generation unit 15 in order.

[0037] The second image generation unit 15 receives input of a plurality of first images generated from random noise of different seeds from the first image generation unit 13. The second image generation unit 15 also receives input of a diffusion model to which a selected module has been applied from the module application unit 14.

[0038] Then, the second image generation unit 15 repeats denoising a predetermined number of times using the acquired diffusion model on the first image generated from random noise of a specific seed to generate a second image as a final image. For example, the second image generation unit 15 performs denoising two or three times to generate the second image. Note that the second image does not necessarily have to be the final image.

[0039] In other words, for each module included in the plurality of modules, noise removal from the first image is performed a predetermined number of times to generate the second image. Alternatively, noise removal from the first image may be performed up to the final stage to generate the second image.

[0040] The second image generation unit 15 receives an input of a diffusion model to which other modules have been applied from the module application unit 14, and generates a second image from the first image generated from random noise of the same specific seed. At this time, the second image generation unit 15 uses the same numerical value for each module as a prompt. As a result, the second image generation unit 15 obtains, for all modules included in the module group 201, respective second images obtained from the first image generated from random noise of the specific seed.

[0041] Here, the second image generation unit 15 can generate the second image using modules other than the modules used to generate the synthetic diffusion model, because the de-diffusion process can be applied to the first image in the middle of the de-diffusion process, regardless of its size, etc.

[0042] Furthermore, the second image generation unit 15 similarly performs the same processing on the first images generated from random noise with different seeds, thereby acquiring second images generated from the diffusion models to which all the modules included in the module group 201 are applied. Thereafter, the second image generation unit 15 outputs, for each first image with different seeds, the second images obtained from the diffusion models to which each module is applied to the FID calculation unit 16.

[0043] In this embodiment, the second image is generated from the first image, so the quality of the final product does not reach the quality of the final product generated by each module individually from initial random noise. However, similarity calculations require that the generated images for each module are different enough to be clusterable. Therefore, even a second image of lower quality can be used for similarity calculations. Specifically, if the similarity calculation using the Fréchet distance between Gaussian distributions described below can be performed and the de-diffusion process can be advanced to the extent that the minimum distance between clusters is equal to or greater than a threshold, that second image can be used. If such a second image can be obtained, the goal of classifying modules can be achieved even if clustering is performed using an image different from the final product.

[0044] For each first image having a different seed, the FID calculation unit 16 receives input of each second image obtained from the diffusion model to which each module is applied from the second image generation unit 15. Then, the FID calculation unit 16 selects a pair of modules and calculates the FID from the second image when each module is applied.

[0045] FID is a method for measuring the distance between two data distributions in a generative model. The FID calculation unit 16 calculates the FID by performing the following calculation steps.

[0046] The FID calculation unit 16 extracts features from each of the second images corresponding to one module and the other module using an Inception network, which is a classification network. Next, the FID calculation unit 16 fits a multivariate Gaussian distribution to each data set (the second image corresponding to one module and the second image corresponding to the other module) from the extracted features to calculate the mean and covariance matrix of the feature quantities. Next, the FID calculation unit 16 calculates the Fréchet distance between the two Gaussian distributions using the mean vector and covariance matrix of the two multivariate Gaussian distributions in the following equation (1). This Fréchet distance corresponds to the FID.

[0047]

number

[0048] where μ x and Σ x denotes the mean and covariance matrix of the data distribution of one of the second images. g and Σ g denotes the mean and covariance matrix of the data distribution of the other second image. Also, "|| ||" denotes the Euclidean norm. Also, Tr denotes the trace, which is the sum of the diagonal elements of the matrix.

[0049] Thereafter, the FID calculation unit 16 outputs the calculated FIDs between the modules to the clustering unit 17.

[0050] The clustering unit 17 receives input of the FIDs between modules from the FID calculation unit 16. Then, the clustering unit 17 performs clustering of the modules using the FIDs, and clusters the modules. Here, if the FIDs are close, it indicates that the second images between the modules are close, and it can be considered that the similarity is high. For example, the clustering unit 17 can perform clustering using the k-means method.

[0051] FIG. 4 is a diagram illustrating output similarity. Finally, as shown in FIG. 4, when there are modules 111 to 113 used in LoRA, the clustering unit 17 provides an index indicating which of the images 121 to 123 created by each of the modules are similar and which are different. For example, the clustering unit 17 classifies modules 112 and 113 into the same cluster because images 122 and 123 are similar, and classifies module 111 into a different cluster because image 121 is different from the others. This shows that when modules 112 and 113 are applied to the diffusion model, similar images are generated, but when module 111 is applied, an image different from when modules 112 and 113 are applied is generated.

[0052] The input / output device 18 includes input devices such as a keyboard and a mouse, and an output device such as a monitor. Using the input / output device 18, a user can check information provided by the information processing device 1 and can input commands, information, etc.

[0053] The module proposal unit 19 acquires information about each cluster of the clustered modules from the clustering unit 17. Next, the module proposal unit 19 extracts a central module of each cluster. Then, the module proposal unit 19 acquires the extracted module from the module group 201 and the base model 200 from the data storage unit 12. Then, for each central module of each cluster, the module proposal unit 19 applies the module to the base model 200 to generate a diffusion model, and generates an image using the generated model. Next, the module proposal unit 19 outputs an image of each central module of each cluster to the input / output device 18 to display and present it to the user.

[0054] Thereafter, the module suggestion unit 19 receives input of information about an image selected from the images for each central module of each cluster as the image closest to the image the user wants to create from the input / output device 18. Next, the module suggestion unit 19 acquires several modules that are close to and similar to the module that generated the selected image from the module group 201 in the data storage unit 12. Then, the module suggestion unit 19 generates an image for each module using modules that are close to and similar to the module that generated the selected image, and presents the images to the user.

[0055] Thereafter, the module suggestion unit 19 receives input of information about an image that is reselected from the images for each module that is at a similar proximity to the module that generated the selected image as the image that is closest to the image that the user wants to create, from the input / output device 18. Here, the module suggestion unit 19 asks the user using the input / output device 18 whether or not to accept the selected image as the final result. If the user accepts, the selected image is treated as the final result and information about the module that was used to generate that image is provided to the user using the input / output device 18.

[0056] 5 is a flowchart of the module clustering process. Next, the flow of the module clustering process performed by the information processing device 1 according to this embodiment will be described with reference to FIG.

[0057] The composite diffusion model generation unit 11 acquires a plurality of modules from the module group 201 from the data storage unit 12. The composite diffusion model generation unit 11 also acquires the base model 200 from the data storage unit 12. Then, the composite diffusion model generation unit 11 combines the acquired plurality of modules and applies it to the base model 200 to generate a composite diffusion model (step S1).

[0058] The first image generation unit 13 generates a first image by repeating denoising a predetermined number of times from random noise of the set seed using a synthetic diffusion model (step S2). Here, the first image generation unit 13 receives in advance the setting of the seed to be used initially.

[0059] The module application unit 14 selects one module from among the unselected modules in the module group 201 (step S3).

[0060] Next, the module application unit 14 applies the selected module to the base model 200 to generate a diffusion model (step S4).

[0061] The second image generating unit 15 generates a second image by using the diffusion model generated by the module application unit 14 for the first image generated by the first image generating unit 13 (step S5).

[0062] Next, the second image generation unit 15 determines whether or not second images have been generated for all modules included in the module group 201 (step S6). If there is a module for which second images have not been generated (step S6: No), the clustering process returns to step S3.

[0063] On the other hand, if second images have been generated for all modules included in the module group 201 (step S6: Yes), the second image generation unit 15 determines whether generation of second images for a predetermined number of random noises with different seeds has been completed (step S7). If there is a random image with a seed for which second images have not been generated (step S7: No), the synthetic diffusion model generation unit 11 changes the seed of the random noise to another unused seed (step S8). Then, the clustering process returns to step S2.

[0064] On the other hand, if the generation of second images for the predetermined number of random noises with different seeds is completed (step S7: Yes), second image generation unit 15 outputs multiple images generated from multiple random noises with different seeds for each module to FID calculation unit 16. FID calculation unit 16 calculates each FID between modules (step S9).

[0065] The clustering unit 17 clusters the modules using the FIDs calculated by the FID calculation unit 16 (step S10).

[0066] 6 is a flowchart of the module proposal process. Next, the flow of the module proposal process performed by the information processing device 1 according to this embodiment will be described with reference to FIG.

[0067] The module proposal unit 19 extracts a module that exists at the center of each cluster of modules generated by the clustering unit 17 and obtains it from the module group 201. Next, the module proposal unit 19 applies each extracted module to the base model 200 to generate respective images and displays them on the input / output device 18. In this way, the module proposal unit 19 presents to the user each image when the extracted modules are used (step S11).

[0068] Next, the module proposing unit 19 receives, from the input / output device 18, input of information on an image selected by the user from among the images when the extracted module is used (step S12).

[0069] Next, the module proposal unit 19 applies the neighboring modules of the selected module to the base model 200 to generate respective images and displays them on the input / output device 18. In this way, the module proposal unit 19 presents to the user respective images when the neighboring modules of the selected module are used (step S13).

[0070] Next, the module proposing unit 19 receives an input of information about an image selected by the user from among the images obtained when the neighboring modules of the selected module are used, from the input / output device 18 (step S14).

[0071] Next, the module proposing unit 19 uses the input / output device 18 to confirm whether or not the selected image is to be accepted as the final result (step S15).

[0072] Then, the module proposal unit 19 determines whether or not the user has accepted the input from the input / output device 18 (step S16). If the user has not accepted the input (step S16: No), the module proposal unit 19 returns to step S13.

[0073] On the other hand, if the user accepts (step S16: Yes), the module proposing unit 19 provides the user with information about the module used to generate the selected image, and ends the module proposing process.

[0074] As described above, the information processing device according to this embodiment generates a composite diffusion model by combining appropriate modules and applying them to a base model. Then, using the generated composite diffusion model, noise is removed from a predetermined random noise through multiple noise removal stages, up to an intermediate stage before the final stage, to generate a first image. The information processing device then performs noise removal from the first image a predetermined number of times using the diffusion model to which each module is applied, to generate each second image. The information processing device then performs clustering of the modules using the generated second image, and suggests desired modules to the user using the clusters of the modules.

[0075] A second image can be created from the first image model by performing denoising processing several times, which can shorten the step of generating a second image for each module. Therefore, the overall FID calculation between modules can be accelerated, and clustering of modules according to their characteristics can be performed in a short time. Using these clusters, it is easy to suggest appropriate modules to users. Therefore, appropriate modules can be suggested to users in a short time, and the time required for image generation can be shortened.

[0076] For example, in a typical inverse diffusion process schedule, denoising using a diffusion model that applies each module to a first image generated using a composite diffusion model is sufficient to calculate similarity two or three times. In contrast, to generate a second image using a diffusion model that applies each module from initial random noise, denoising is performed approximately 40 times. In other words, this is expected to speed up FID calculations by approximately 20 times, significantly reducing the calculation cost of similarity between modules.

[0077] On the other hand, the cost of each calculation added to realize the functions of the information processing device according to this embodiment is very low. Specifically, this is the calculation cost for calculating the weights of the synthesis module and the calculation cost for obtaining the first image during the despreading process using the synthesis module. The calculation for obtaining the first image during the despreading process using the synthesis module is performed only once, and the greater the number of target modules, the greater the cost reduction effect when using the information processing device according to this embodiment, making it advantageous overall.

[0078] For example, when measuring the weight distance between modules, the calculation speed is high, but the size and rank of the modules must be exactly the same, making it unrealistic to actually use this distance for clustering. Furthermore, when generating a second image from a first image using a diffusion model to which all modules are applied and calculating the FID, the calculation is possible, but it requires a very large number of calculations, making it difficult to implement. In contrast, clustering using the information processing device according to this embodiment is possible as long as it can output an image when using each module, regardless of the size and rank of the module, and the calculation can be performed quickly.

[0079] Another method for selecting a module is for the user to directly specify a module from among a group of modules. For example, one method is to specify a specific color for a range area by color specification. This is achieved by the user directly specifying RGB and imposing restrictions on the color. Another method is to specify the color by prompt. This allows the user to directly specify the color using words. Another method is to list and tag learned modules. This makes them searchable by listing the module generation results and manually tagging them.

[0080] However, the method of coloring a specific range of colors by specifying a color takes time because the desired output is obtained by repeatedly obtaining the generated results. Also, the user must specify the color from scratch each time the module is used, which makes it difficult to do efficiently. Furthermore, although it is possible to get close to the color desired by the user, it is difficult to specify complex colors.

[0081] Even with the method of specifying color tone using prompts, it takes time to obtain the desired output by repeatedly obtaining the generated results. Furthermore, with this method, the user must specify the color tone from scratch every time the module is used, making it difficult to perform efficiently. Another problem is that it is difficult to select the prompt for which you want to improve the quality.

[0082] The method of listing and tagging learned modules can improve efficiency by allowing users to reuse previously used listings and tags. However, it takes a lot of time to obtain the output of all modules in advance. It is also not possible to compare modules used with the same prompt, making it unclear whether the module that meets the user's needs will be obtained. Furthermore, if tags are not properly assigned, users will have to choose from a large number of modules, making it difficult to find the appropriate one.

[0083] In contrast, when the information processing device 1 according to the present embodiment is used, unnecessary calculations can be eliminated and an appropriate module can be easily obtained by selecting a module from the generated cluster. Also, by using a cluster that has already been generated, it is possible to efficiently propose modules. Furthermore, since user feedback can be reflected immediately, it is possible to propose an appropriate module that is more in line with the user's wishes, and the user experience is improved.

[0084] (Hardware configuration) 7 is a hardware configuration diagram of the information processing device. Next, an example of a hardware configuration for realizing each function of the information processing device 1 will be described with reference to FIG.

[0085] 7, the information processing device 1 includes, for example, a CPU (Central Processing Unit) 91, a memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0086] The network interface 94 is an interface for communication between the information processing device 1 and an external device.

[0087] The hard disk 93 is an auxiliary storage device. The hard disk 93 realizes the function of the data storage unit 12 illustrated in FIG. 1. The hard disk 93 also stores various programs including the following programs. For example, the hard disk 93 stores programs for realizing the functions of the synthetic diffusion model generation unit 11, the first image generation unit 13, the module application unit 14, and the second image generation unit 15 illustrated in FIG. 1. The hard disk 93 also stores programs for realizing the functions of the FID calculation unit 16, the clustering unit 17, the input / output device 18, and the module proposal unit 19 illustrated in FIG. 1.

[0088] The memory 92 is a main storage device and may be, for example, a dynamic random access memory (DRAM).

[0089] The CPU 91 reads out various programs from the hard disk 93, expands them into the memory 92, and executes them. As a result, the CPU 91 realizes the functions of the synthetic diffusion model generation unit 11, the first image generation unit 13, the module application unit 14, the second image generation unit 15, the FID calculation unit 16, the clustering unit 17, the input / output device 18, and the module proposal unit 19, all of which are illustrated in FIG. [Explanation of symbols]

[0090] 1. Information processing equipment 11. Synthetic diffusion model generation unit 12 Data storage section 13 First image generation unit 14 Module application section 15 Second image generation unit 16 FID calculation section 17 Clustering Department 18 Input / Output Devices 19 Module Proposal Department 200 base model 201 Modules

Claims

1. Select several modules from the multiple modules that apply to a trained machine learning model that performs noise removal from random noise up to the final stage of multiple stages to generate images, The selected modules are combined to remove predetermined random noise up to an intermediate stage before the final stage to generate a first image; generating a second image by performing noise removal from the first image a predetermined number of times for each module included in the plurality of modules; Classifying the modules included in the plurality of modules based on the second image for each module. An information processing program that causes a computer to execute a process.

2. 2. The information processing program according to claim 1, wherein the synthesis is performed by calculating an average of the weights of the modules and setting the calculated value as the weight.

3. 2. The information processing program according to claim 1, wherein the classification of the modules includes a process of calculating a distance between the modules based on the second image and performing classification based on the calculated distance.

4. The information processing program according to claim 3, characterized in that the process of generating the second image includes a process of removing noise from the first image the number of times that the shortest distance between the classifications based on the distance between the modules calculated based on the second image is greater than or equal to a threshold.

5. The information processing program according to claim 1, further causing the computer to execute a process of selecting one module from each of the categories, generating an image based on specific random noise for each selected module using the machine learning model to which the module is applied, and presenting to a user a plurality of images generated for each selected module.

6. receiving input of information on a selected image selected by a user from the presented plurality of images, and reselecting a module that is close to the module used to generate the selected image; For each of the reselected modules, an image is generated based on specific random noise using the machine learning model to which the module is applied, and a plurality of images generated for each of the selected modules are presented to the user.

6. The information processing program according to claim 5, further causing the computer to execute a process.

7. Select several modules from the multiple modules that apply to a trained machine learning model that performs noise removal from random noise up to the final stage of multiple stages to generate images, The selected modules are combined to remove predetermined random noise up to an intermediate stage before the final stage to generate a first image; generating a second image by performing noise removal from the first image a predetermined number of times for each module included in the plurality of modules; Classifying the modules included in the plurality of modules based on the second image for each module. An information processing method characterized by executing processing.

8. Select several modules from the multiple modules that apply to a trained machine learning model that performs noise removal from random noise up to the final stage of multiple stages to generate images, a first image generating unit that generates a first image by combining the selected modules and performing noise removal from predetermined random noise up to an intermediate stage before reaching the final stage; a second image generating unit that generates a second image by performing noise removal from the first image a predetermined number of times for each module included in the plurality of modules; a clustering unit that classifies modules included in the plurality of modules based on the second image for each module; An information processing device comprising:

Images