Information processing system, information processor, information processing method

Abstract

To more easily obtain a desired image using a diffusion model.SOLUTION: An image processing part generates an output image by repeating a noise removal step for removing noise components from an input image, for a plurality of times. A user interface part outputs a noise-removed image acquired in the middle of the plurality of times of noise removal steps, and acquires parameters related to the removal of noise components. An image processing part executes the noise removal steps using the acquired parameters. The present embodiment can be realized in any form, such as an information processing system, information processor, or information processing method.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present embodiment relates to an information processing system, an information processing device, and an information processing method. [Background technology]

[0002] In recent years, applications using AI (Artificial Intelligence) technology have been developed to generate images from other types of information such as text, and to convert the characteristics of existing images. Attempts have been made to execute such applications to obtain new images or edit existing images without relying on photography, drawing, or retouching techniques. These applications have been applied to the generation or conversion of photorealistic images similar to photographs, non-photorealistic images such as illustrations, and the like.

[0003] In this regard, Patent Document 1 describes an automatic line drawing coloring program, an automatic line drawing coloring device, and a program for a graphical user interface for coloring a line drawing image. According to the automatic line drawing coloring program, a coloring process is performed on reduced line drawing data obtained by reducing the acquired line drawing data of the original size, and a second-stage coloring process is performed using the obtained colored reduced data and the line drawing data of the original size to obtain a colored image, thereby making it possible to appropriately perform coloring process on data of a large size. In addition, based on each of sample data groups in which at least a part of the sample data is different from each other, a plurality of first trained models used in the first coloring process and a plurality of second trained models used in the second coloring process are trained and stored in advance, and a coloring process is performed using one of the plurality of first trained models and one of the plurality of second trained models, thereby making it possible to perform coloring processes with various coloring tendencies.

[0004] Non-Patent Document 1 also describes an image generation method using a stable diffusion model. The stable diffusion model includes a diffusion model, a variational autoencoder, and a text encoder, and generates an output image corresponding to an input text. The diffusion model gradually removes noise components from an image containing noise, and generates an image mainly composed of signal components. The variational autoencoder converts an input image into a latent representation with a small size, and converts the converted latent representation into an output image of the original size. The text encoder converts the input text into a vector. In the diffusion model, the correspondence between the converted vector and the latent representation is learned. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-197948 A [Non-patent literature]

[0006] [Non-Patent Document 1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models”, [online], Jul. 27, 2022, GitHub, Inc., [Retrieved June 1, 2023], Internet<URL: https: / / github.com / CompVis / latent-diffusion> Summary of the Invention [Problem to be solved by the invention]

[0007] However, according to the above method, the user has only one opportunity to set the conditions for the desired output image before execution. If the generated output image is different from the desired image, it is necessary to reset the conditions for obtaining the desired image and repeat the execution. In addition, it is difficult to give detailed conditions all at once before execution. For example, when generating an output image based on text, it may be difficult to know what information should be specified in the text to obtain the desired image without repeated experience. Some users may record the relationship between the desired image and the text in detail as a rule of thumb. The present application has been made in consideration of the above-mentioned problems, and has as its object to provide an information processing system, an information processing device, and an information processing method for more easily acquiring a desired image using a diffusion model. [Means for solving the problem]

[0008] The information processing system according to the first aspect includes an image processing unit that generates an output image by repeating a noise removal step that removes noise components from an input image a plurality of times, and a user interface unit that outputs a noise-removed image obtained during the plurality of repetitions of the noise removal step and obtains parameters related to the removal of the noise components, and the image processing unit executes the noise removal step using the parameters.

[0009] An information processing device according to a second aspect includes an image processing unit that generates an output image by repeating a noise removal step of removing noise components from an input image a plurality of times, and a user interface unit that outputs a noise-removed image obtained midway through the plurality of times of the noise removal step and obtains parameters related to the removal of the noise components, and the image processing unit executes the noise removal step using the parameters.

[0010] The information processing method according to the third aspect includes: An information processing method in an information processing system, wherein the image processing unit executes an image generation procedure in which an output image is generated from input data by repeating a noise removal step multiple times to remove noise components from a noise removed image of a previous step to obtain a noise removed image of a current step, and the user interface unit executes a user interface procedure in which the noise removed image obtained during the multiple repetitions of the noise removal step is output and parameters related to the removal of the noise components are obtained, and the noise removal step is executed using the parameters in the image generation procedure. Effect of the Invention

[0011] According to the present application, a desired image can be obtained more easily using a diffusion model. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic block diagram illustrating an example of a functional configuration of an information processing device according to an embodiment of the present invention. [Diagram 2] 1 is a schematic block diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Diagram 3] FIG. 4 is a diagram showing an example of a display screen according to the embodiment. [Figure 4] FIG. 11 is a diagram showing another example of the display screen according to the embodiment. [Diagram 5] 5 is a flowchart illustrating an example of image information processing according to the embodiment. [Figure 6] 4 is a schematic block diagram illustrating an example of the functional configuration of a generation unit according to the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Next, an embodiment of the present application will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing an example of a functional configuration of an information processing device 10 according to this embodiment. The information processing device 10 generates an output image by repeating a noise removal step of removing noise components from an input image multiple times. The input image is, for example, a seed image in which real values ​​forming a pseudorandom number sequence are expressed as signal values ​​for each pixel. The output image may be any of a realistic image such as a photograph, a non-realistic image such as a cartoon or illustration, and the like. The information processing device 10 is capable of outputting a noise-removed image acquired during multiple iterations of the noise removal step.

[0014] The information processing device 10 includes an image processing unit 110, a user interface unit 120, and a storage unit 130. Each unit of the information processing device 10 includes dedicated hardware. The information processing device 10 may include a general-purpose computer system to realize the functions of each unit. That is, the information processing device 10 includes at least a processor, executes a predetermined program, and realizes the functions of an image processing unit 110, a user interface unit 120, and a storage unit 130 in cooperation with a storage medium and other hardware.

[0015] The processor has, for example, a CPU (Central Processor Unit). The processor may further include a GPU (Graphic Processing Unit). The storage medium has a RAM (Random Access Memory), a ROM (Read-only Memory), etc. The storage medium may further include an auxiliary storage device such as an HDD (Hard-disk Drive) or an SSD (Solid State Drive). In this application, executing a process instructed by an instruction written in a program may be referred to as "executing a program" or "running a program", etc.

[0016] The image processing unit 110 includes a generation unit 112, a conversion unit 114, and a feature extraction unit . The generating unit 112 repeats a noise removal step for removing noise components from a target image to be processed a plurality of times to generate an output image. The number of times T that the noise removal step is repeated may be determined in advance. The generating unit 112 may execute the noise removal step by applying a constraint condition. For example, an image, an image feature, text, or other information that can indicate the features of an image may be applied as the constraint condition.

[0017] The generating unit 112 acquires an input image to be processed. The generating unit 112 generates a pseudo-random number sequence using a known pseudo-random number generation method based on a certain seed value, for example. The generating unit 112 determines each pseudo-random number value constituting the generated pseudo-random number sequence as a signal value for each pixel of the input image. The generating unit 112 sets the generated input image as a target image in a first noise removal step. In a certain noise removal step, the generating unit 112 subtracts a noise component estimated from the target image using a model previously learned from the target image from the target image to obtain a noise-removed image. In this application, the "model" mainly refers to a mathematical model, that is, a calculation processing method or procedure for obtaining an output from an input. The generating unit 112 sets the noise-removed image obtained by the noise removal step of that time as a target image in the noise removal step of the next time. The generating unit 112 outputs output image data indicating, as an output image, a noise-removed image obtained by the final noise removal step or a noise removal step instructed by operation data from the receiving unit 126 (described later), to the user interface unit 120. The generating unit 112 may output the output image data to the storage unit 130 for storage.

[0018] In the noise removal step, the generating unit 112 may adjust the amount of the estimated noise component and subtract the adjusted noise component from the target image. When adjusting the amount of the estimated noise component, the generating unit 112 multiplies the noise component by a step width, for example. The step width is also called a step size and corresponds to a gain. The generating unit 112 may include a classifier that estimates the probability of each type (class) of image for the target image, estimate class-specific noise components for each class, and estimate a weighted sum of the class-specific noise components as the noise components in the noise removal step. The image class may be applied to subjects or their arrangement, background color or brightness distribution, etc. The classifier is trained in advance with a model that receives a known image as input and outputs the class of the image. The known image used for training may be associated with text of a phrase expressing the characteristics of the image or class. This text may be called a prompt. The generating unit 112 derives class-specific weighting coefficients used in calculating the weighted sum of the class-specific noise components from the model trained in advance. The generating unit 112 may update the weighting coefficient for the class to a weighted average of a multiplication value obtained by multiplying the weighting coefficient derived for each class by a first coefficient and a multiplication value obtained by multiplying a uniform random number by a second coefficient. The sum of the first coefficient and the second coefficient may be normalized to be 1. The generating unit 112 may estimate a weighted sum of noise components by class based on the updated weighting coefficient as the noise component in the noise removal step. This prevents the output image from being biased toward a specific class. Adjusting the degree to which the output image is biased toward a specific class is sometimes called CFG (Classifier Free Guidance) scaling. The second coefficient corresponds to a parameter for indicating the diversity of the output image. In this application, the second coefficient is sometimes called diversity.

[0019] The generating unit 112 can use a DDIM (Denoising Diffusion Implicit Model) as a model related to the removal of noise components. The DDIM is a type of diffusion model. The diffusion model is a mathematical model for estimating noise components contained in a target image and removing the estimated noise components from the target image. The DDIM is mainly used for generating an output image. The DDIM has a forward diffusion process and a reverse diffusion process. The forward diffusion process is a process for adding noise components to a target image. The forward diffusion process is a process for generating a noise-added image by adding Gaussian noise to a known image as an input. The forward diffusion process has multiple noise addition steps. The more the number of repetitions of the noise addition step (also referred to as a step number or time in the present application) increases, the more the composition ratio of the noise components contained in the noise-added image increases. The generating unit 112 learns in advance the parameters of a prediction model for predicting the noise components contained in the noise-added image from the target image to be processed at each time in the forward diffusion process. The de-diffusion process is a process for removing the noise components from the noise-added image. The de-diffusion process has a plurality of noise removal steps. In each noise removal step, the generating unit 112 predicts the noise components from the target image to be processed using the learned prediction model, and subtracts the predicted noise components from the target image to obtain a noise-removed image. The generating unit 112 sets the obtained noise-removed image as the target image for the noise removal step at the next time. In this way, by repeating the noise removal step for removing the noise components from the target image to obtain the noise-removed image, an output image in which the noise components have been removed from the input image is estimated.

[0020] The generating unit 112 may use PIDM (Palette Image-to-Image Diffusion Model) in removing the noise components. PIDM is mainly used for converting an input image. PIDM has a forward diffusion process and a reverse diffusion process similar to DDIM. However, in PIDM, an image to be converted is applied as a constraint condition image as a constraint condition for predicting a noise component in the reverse diffusion process. A model of the reverse diffusion process in PIDM is learned using training data including a plurality of data sets in which a known noise-added image as an input, combined data obtained by combining a known noise-added image and a constraint condition image as a constraint condition in parallel, and an image to which no noise is added and having a predetermined characteristic as an output are associated with each other. The pixels of the condition image to be converted and the pixels of the output image after conversion are spatially associated with each other. PIDM may be applied to coloring, inpainting, and the like. Coloring is realized by estimating a spatial distribution of colors as a predetermined characteristic. The restoration is realized by using, as a predetermined feature, any one of, or any combination of, an estimation of a pattern before damage for a damaged portion, an estimation of a pattern before deterioration for a deteriorated portion, and an estimation of a pattern before loss for a lost portion.

[0021] The generating unit 112 may apply, for example, a latent diffusion model (LDM). The LDM is a mathematical model that is the basis of stable diffusion. Stable diffusion is a model for generating an output image corresponding to an input text. An application example of the LDM will be described later.

[0022] The conversion unit 114 executes image conversion on the input image by repeating a noise removal step for removing noise components from the target image to be processed under certain constraint conditions multiple times. In this application, certain constraint conditions applied in image conversion may be referred to as "fixed conditions." Constraint conditions used to control a generated image, whether in image conversion or image generation, may be referred to as "control conditions."

[0023] The conversion unit 114 is common to the generation unit 112 in that a new output image is obtained by repeating the noise removal step multiple times. However, the conversion unit 114 differs from the generation unit 112 in that the conversion unit 114 always uses image features or image data as a certain constraint condition. The model of the conversion unit 114 receives combined data obtained by combining the target image and the constraint condition in parallel, and outputs a noise-removed image. Image conversion can be applied to, for example, coloring, super-resolution, restoration, or a combination thereof. The image data used as the constraint condition may be, for example, data indicating the color signal value of each pixel (for example, an RGB image) or data indicating the type (class label) of each pixel (for example, a semantic image). The image data used as the constraint condition may be, for example, an image of a person or character having a specific posture, facial expression, or a combination thereof as the main subject, a scene including the background, or the like. In addition, text indicating a phrase expressing the posture (pose, gesture, etc.), facial expression, or background characteristics may be applied to the constraint condition. The image feature amount may be an image feature amount obtained by the feature extraction unit 116 .

[0024] The feature extraction unit 116 extracts features by performing a predetermined arithmetic process on the input data. The extracted features are used as constraint conditions in the generation unit 112 or the conversion unit 114. For example, three-dimensional image features or two-dimensional features are acquired as features. The three-dimensional image features can be used for image conversion as fixed conditions in the conversion unit 114. The three-dimensional image features include element values ​​for each set of width, height, and channel. The two-dimensional features include element values ​​for each set of words and channels. These features can be used as control conditions for image generation in the generation unit 112 or image conversion in the conversion unit 114, for example.

[0025] The feature extraction unit 116 uses, for example, a model to extract the above feature amount as output data from time series data as input data. As the model, for example, a neural network such as a transformer is used. The input time series data may be, for example, image data (for example, RGB data, semantic data) at each time, text, or a combination thereof. The feature extraction unit 116 outputs the extracted feature amount to the generation unit 112 or the conversion unit 114.

[0026] The user interface unit 120 includes a display unit 122 , a display control unit 124 , and a reception unit 126 . The display unit 122 displays a display image according to display data input thereto. As the display image, for example, various display screens indicated in the display data acquired from the display control unit 124 are displayed. The display unit 122 includes, for example, a display device. The display device may be, for example, any of an LCD (Liquid Crystal Display) and an OLED (Organic Light Emitting Diode) display.

[0027] The display control unit 124 controls the display of an image on the display unit 122 or the image to be displayed on the display unit 122. The display control unit 124 cooperates with the display unit 122 and the reception unit 126 to configure a user interface. The display control unit 124 generates various display screens to be displayed on the display unit 122, and outputs display data showing the generated display screens to the display unit 122. The display control unit 124 controls the processing of the image processing unit 110 in accordance with operation data input from the reception unit 126. While the display screen is being displayed, the display control unit 124 executes, in accordance with the operation data, whether or not a noise removal step needs to be performed, selection of either image generation or image conversion, setting of constraint conditions, and other parameters.

[0028] The display control unit 124 configures a display screen including, for example, a noise-removed image and guidance information. The display control unit 124 acquires the noise-removed image in the middle of a plurality of noise removal steps executed by the generation unit 112 or the conversion unit 114. The guidance information includes information for guiding the provision of constraint conditions related to the removal of noise components. The display control unit 124 acquires operation data indicating the constraint conditions from the reception unit 126. The display control unit 124 causes the generation unit 112 or the conversion unit 114 to start the execution of the noise removal step using the constraint conditions set by the operation data. The display control unit 124 identifies a step at which the execution of the noise removal step is to be resumed according to the operation data, and resumes the execution of the noise removal step from the identified step. The display control unit 124 may acquire a newly acquired noise-removed image from the generation unit 112 or the conversion unit 114, and configure the display screen to include the acquired noise-removed image. An example of the functional configuration of the display control unit 124 will be described later.

[0029] The reception unit 126 acquires operation data and outputs the acquired operation data to the display control unit 124. The reception unit 126 includes, for example, an input device that accepts a user's operation. The input device may be any of a mouse, a pointing device such as a touch sensor, a keyboard, and the like.

[0030] The storage unit 130 temporarily or continuously stores data used in the information processing device 10, data acquired, and programs for implementing the functions. The stored data includes parameter sets of models used in the generation unit 112, conversion unit 114, and feature extraction unit 116. The generation unit 112, conversion unit 114, and feature extraction unit 116 read out their respective parameter sets from the storage unit 130 when starting to operate. The storage unit 130 includes an auxiliary storage device such as an HDD or SSD.

[0031] Next, an example of the functional configuration of the display control unit 124 will be described. The display control unit 124 may configure a display image including a setting area for setting a fixed condition for multiple noise removal steps. In the present application, this setting area may be referred to as a "fixed condition selection unit." The display control unit 124 may include a screen component for selecting whether to execute image conversion or image generation according to operation data, in the display screen. The display control unit 124 may execute image conversion when a fixed condition is set or when a request to set a fixed condition is instructed. When executing image conversion, the display control unit 124 applies the fixed condition to the conversion unit 114 to execute the noise removal step. When a fixed condition is not set or when a request not to set a fixed condition is instructed, the display control unit 124 may determine to execute image generation. When determining to execute image generation, the display control unit 124 causes the generation unit 112 to execute the noise removal step.

[0032] The display control unit 124 configures a display screen including, for example, the noise-removed images obtained by the generation unit 112 or the conversion unit 114 through a plurality of different noise removal steps and screen components. The screen components are used to select the noise-removed images obtained by any of the steps in accordance with the operation data. The display control unit 124 arranges, for example, the noise-removed images obtained by the plurality of different noise removal steps on the display screen in ascending or descending order of the step numbers of the noise removal steps. In this application, the area presenting the noise-removed images may be referred to as a "preview display unit."

[0033] The display control unit 124 may configure the display screen by arranging one denoised image at a time in a preview display unit that is a part of the display screen. The display control unit 124 may update the denoised images to be arranged in ascending or descending order of the step numbers of the noise removal steps at regular intervals (e.g., 1 / 120 to 1 / 12 seconds) with denoised images acquired by different noise removal steps. In this way, the denoised images acquired at each noise removal step are expressed as a moving image that changes with the passage of time. The display control unit 124 may display the denoised images in association with the step numbers or the progress rate (the step numbers are normalized by the total number of steps).

[0034] The display control unit 124 may configure the display screen including a setting area for setting the control condition and the application period of the constraint condition according to the operation data input from the reception unit 126. In the present application, such a setting area may be referred to as a "control condition setting unit." The display control unit 124 sets the constraint condition and the application period in association with each other according to the operation data input from the reception unit 126. The display control unit 124 causes the generation unit 112 or the conversion unit 114 to execute a noise removal step using the set control condition in the set application period. The display control unit 124 may acquire a noise removed image obtained by executing the noise removal step from the generation unit 112 or the conversion unit 114, and update an existing noise removed image within the application period arranged on the display screen to the newly acquired noise removed image. A display example of the display screen will be described later.

[0035] The display control unit 124 may store in the storage unit 130, for each trial, history data including a noise-removed image acquired by executing a noise removal step, a constraint condition used in the noise removal step, and an application period of the constraint condition. The display control unit 124 may refer to the history data stored in the storage unit 130 and generate, for each trial, a history display in which a noise-removed image acquired by a different noise removal step, a constraint condition, and an application period of the constraint condition are arranged in association with each other. The display control unit 124 may configure a display screen by juxtaposing the generated history displays between different trials, and display the display screen on the display unit 122.

[0036] In the above description, a case where an information terminal device used mainly by a specific user realizes the functions of an information processing device 10, which is one form of an information processing system, is taken as an example. Typically, the functions of the information processing device 10 are realized by a processor constituting a computer system of the information terminal device executing a predetermined application program (sometimes referred to as an "app" in the present application) in cooperation with a memory and other hardware. The information processing device 10 according to the present embodiment is not limited to this, and may be realized as a server device such as a web server or a cloud server. Such a server device provides the functions of the information processing device 10 in response to access from a user terminal device 20 of an unspecified or multiple users via a communication network. The user terminal device 20 may be any type of information terminal device equipped with a computer system, such as a personal computer (PC), a tablet terminal device, or a mobile phone.

[0037] This embodiment may be realized as an information processing system S1. The information processing system S1 includes an information processing device 10 and one or more user terminal devices 20. In the example of FIG. 2, the number of user terminal devices 20 in the information processing system S1 is two, but it may be one or three or more. In this application, unless otherwise specified, functions and configurations common to two or more user terminal devices 20 will be described. The information processing device 10 is connected to each user terminal device 20 via a communication network NW so as to be able to transmit and receive various data wirelessly or wired.

[0038] The user terminal device 20 may include a display unit 222 and a reception unit 226 (not shown) and may realize a functional configuration similar to that of the display unit 122 and the reception unit 126 of the information processing device 10. In this case, inputs and outputs similar to those of the display unit 122, the reception unit 126, and the display control unit 124 occur between the user terminal device 20 and the information processing device 10. In addition, the display unit 122 and the reception unit 126 may be omitted in the information processing device 10.

[0039] The user terminal device 20 may further include a display control unit 224 (not shown) and may realize a functional configuration similar to that of the display control unit 124 of the information processing device 10. In this case, input and output similar to that between the display control unit 124 and the generation unit 112 or the conversion unit 114 occurs between the user terminal device 20 and the information processing device 10. In the information processing device 10, the display control unit 124 may be omitted.

[0040] Next, an example of the display screen will be described together with the related functions of the display control unit 124. Fig. 3 is a diagram showing an example of the display screen according to this embodiment. The display screen exemplified in Fig. 3 is used for controlling the image generation or conversion process, presenting the noise removal steps acquired during the process, setting constraint conditions, etc. The display screen exemplified in Fig. 3 includes a noise attenuation control unit A, a fixed condition selection unit B, a control condition selection unit C, and a preview display unit D.

[0041] The noise attenuation control unit A is a display area for instructing a function related to noise removal according to operation data by the display control unit 124. In the example of FIG. 3, the noise attenuation control unit A has two slider bars, a seed change button, a generate button, an Undo button, and a Redo button arranged therein. The slider bar is a screen component for the display control unit 124 to set a parameter value according to the position of the knob instructed by the operation data. The two slider bars make it possible to set diversity and step width. As described above, diversity means the degree of possibility that a noise-removed image having various characteristics is generated. The higher the diversity, the higher the possibility that a noise-removed image having various characteristics is generated, but there is a risk that the characteristics of the noise-removed image will vary each time the noise removal step is performed, and an output image having the desired characteristics will not be obtained. The optimal diversity may be set in consideration of the balance between the stability of the noise-removed image and the quality of the generated output image. As the seed value, diversity, and step width, preset setting values ​​may be used.

[0042] As described above, the step width is the gain for the noise components removed in each noise removal step. The step width is also called the step size. In general, the smaller the step width, the higher the image quality of the output image, but the longer the processing time tends to be. The optimal step width should be set in consideration of the balance between the performance of the information processing device 10 and the quality of the generated output image. The seed change button is a button for instructing a change of the seed value by pressing it. The seed value is a value used for generating a pseudo-random number sequence that constitutes an input image. The display control unit 124 generates a separate pseudo-random number value in response to pressing the seed change button, and sets the generated pseudo-random number value as the seed value. In this application, "pressing" refers to not only an actual pressing, but also generating operation data indicating a position within the display area, or acquiring the operation data.

[0043] The generate button is a button for instructing the start of processing relating to the generation of an output image, that is, the execution of a noise removal step, when pressed. The Undo button is a button for canceling an instruction given by a previous operation when pressed, that is, for stopping a specified process or releasing a specified state. For example, if application of a constraint condition image (described later) was specified immediately before, pressing the Undo button releases the application of the constraint condition image, and the constraint conditions applied previously are applied again. At that time, the display control unit 124 may erase the display of the noise-removed image obtained based on the released constraint condition image, and re-display the noise-removed image obtained based on the constraint conditions applied previously. The Redo button is a button for canceling the cancellation caused by pressing the Undo button, that is, for instructing the restart of the stopped process or the reapplication of the canceled state. For example, if the application of the constraint condition text (described later) is canceled by pressing the Undo button, the constraint condition text is reapplied by pressing the Redo button. At that time, the display control unit 124 may erase the display of the noise-removed image obtained based on the constraint conditions reapplied by pressing the Undo button, and may redisplay the noise-removed image obtained based on the reapplied constraint condition text.

[0044] The fixed condition selection section B is an area for the display control section 124 to specify whether or not a fixed condition needs to be set, and, if a fixed condition needs to be set, a constraint condition image to be applied as the fixed condition, according to the operation data. The fixed condition selection section B has two radio buttons, a display area for the constraint condition image, and a reference button. One radio button is a button for specifying that there is no constraint condition image when pressed. The other radio button is a button for specifying that there is a constraint condition image. Directly below the other radio button, a display area in which the selected constraint condition image is displayed is allocated. The reference button is a button for displaying a directory of the file system when pressed and selecting the location of image data showing a desired constraint condition image. When it is specified that a fixed condition needs to be set, the display control section 124 disables the function of the generation section 112 and enables the function of the conversion section 114. When it is instructed that the fixed condition is not set, the display control unit 124 enables the function of the generation unit 112 and disables the function of the conversion unit 114.

[0045] The control condition setting section C is an area for setting the control condition by the display control section 124 according to the operation data. The control condition is a constraint condition applied to a part of the multiple noise removal steps or to an instructed application period. The control condition setting section C includes a preview section and a condition setting section. The preview section is a display area for displaying a noise removed image obtained in a selected step according to the operation data. The noise removed image to be previewed is selected from among the multiple noise removed images displayed in the preview display section D according to the operation data. The condition setting section includes a display area for displaying the selected image (corresponding to the above-mentioned "constraint condition image"), a reference button, and a text input section. The reference button is a button that is arranged directly below the constraint condition image and is used to display a directory of the file system by pressing the button, and to select the location of image data showing the desired constraint condition image. The text input section is a display area for setting an input text (corresponding to the above-mentioned "constraint condition text") to be applied as a constraint condition according to the operation data.

[0046] In the preview display section D, a preview image display section, a scroll bar, and a step interval setting section are arranged. In the preview image display section, a noise-removed image obtained by a noise removal step executed by the generating section 112 or the converting section 114 is arranged in association with the step number of the noise removal step for each step interval (display interval) specified in the step interval setting section. The display control section 124 sets an application section for each of the constraint condition image and constraint condition text set in the control condition setting section C according to the operation data. For example, the display control section 124 selects the constraint condition detected as pressed from the control condition setting section C as the setting target for the application section. After that, the display control section 124 determines the start point and the end point of the application section for the constraint condition. In the preview display section D, the display control section 124, for example, detects a drag operation and specifies the start point and the end point of the coordinates specified. The display control unit 124 can specify the step numbers of the noise images closest to the specified starting point and end point, respectively, and set the smaller and larger of the specified step numbers as the starting point and end point of the application section, respectively. The set application section is displayed by the display control unit 124 using a double arrow in association with each of the constraint condition image and the constraint condition text. The scroll bar is a screen component for changing the position of the knob specified by the operation data to change the noise-removed image of the noise removal step to be arranged in the preview image display field. In the example of FIG. 3, by moving the position of the knob to the left, a noise-removed image related to a noise removal step that is later in time is displayed. The display control unit 124 sets the numerical value specified by the operation data in the step interval setting field as the step interval. In the example of FIG. 3, one image and one character string are applied as the control condition, but this is not limited to the above. The number of images and the number of character strings applied may be two or more. One of the image and the character string may be set, and the other may not be set. The operation to be detected when setting the application section is not limited to a drag operation. For example, multiple click operations may be used. A screen component may be further provided for each constraint condition set in the control condition setting unit C. The display control unit 124 detects a screen component for which the start point and the end point of the application section are specified based on the operation data, and specifies the constraint condition corresponding to the detected screen component. Then, the display control unit 124 can specify the specified application section as the application section for that constraint condition. For example, an input field may be used as the screen component. In that case, the display control unit 124 specifies the numerical value set in the input field as the step number of the start point and the end point. In addition, a slider bar may be used as the screen component. In that case, the display control unit 124 can specify the step numbers of the start point and the end point corresponding to the position of the slider specified by the operation.

[0047] The display control unit 124 causes the generating unit 112 or the converting unit 114, whichever has its function set to be enabled (sometimes referred to as the "enabled processing unit" in this application), to execute the noise removal step by applying the constraint conditions from the start step to the end step for each set application section. The display control unit 124 may obtain a noise-removed image obtained by executing the noise removal step from the enabled processing unit, and update the display screen by replacing a preview image in the application section with the obtained noise-removed image. This allows the user to check the transition of the noise-removed image according to the set constraint conditions and application section while looking at the preview image.

[0048] The display control unit 124 sequentially stores in the storage unit 130 the instruction notified by the operation data, the setting information, the operating state of the image processing unit 110 according to the instruction, the noise-removed image generated for each noise removal step, and the like. Various information stored in the storage unit 130 is accumulated to form history data. The display control unit 124 stores, as instructions, the start of execution (pressing the generate button), the change of at least one parameter, the application and release of at least one constraint condition, the cancellation of the previous instruction by pressing the Undo button, and the cancellation of the previous cancellation by pressing the Redo button. As for the stop, the step number of the noise removal step that has been completed before the instruction to stop is associated and recorded. The display control unit 124 manages, for example, the information stored by a series of processes from the start of execution to the next instruction to start execution as history data related to one trial. The display control unit 124 counts the number of times the instruction to start execution is issued as the number of trials, and the history data is saved in the storage unit 130 for each trial.

[0049] The setting information may include a seed value used to generate the input image, the diversity and step width used in the noise removal step, the presence or absence of a fixed condition, constraint condition information indicating a certain constraint condition when the fixed condition is present, and the constraint condition and application period as the control condition. The display control unit 124 stores the noise-removed image obtained by the noise removal step in association with the step number of the noise removal step.

[0050] FIG. 4 is a diagram showing another example of the display screen according to the present embodiment. The display screen exemplified in FIG. 4 is an editing history screen. For example, when detecting a press on an editing history display button (not shown) displayed on the display unit 122, the display control unit 124 configures an editing history screen and displays the configured editing history screen on the display unit 122. When configuring the editing history screen, the display control unit 124 refers to the history data for each trial, and configures a history display by correspondingly arranging a noise-removed image acquired by a different noise removal step, a constraint condition, and an application period of the constraint condition. The display control unit 124 configures the editing history screen by juxtaposing the history displays configured for each trial. In the example of FIG. 4, the history displays for each trial are arranged in each row. Each history display includes information of the same items as those in the preview display section D exemplified in FIG. 3. The history display exemplified in the second row of FIG. 4 is the same as the display information in the preview display section D of FIG. 3. A user who comes into contact with the displayed information can easily understand the noise-removed images that are generated according to the set constraint conditions and the application period of those conditions, as well as the differences in the transitions between those images.

[0051] Next, an example of image information processing according to this embodiment will be described. Fig. 5 is a flowchart showing an example of image information processing according to this embodiment. The processing in Fig. 5 is premised on being started when the display screen exemplified in Fig. 3 is displayed on the display unit 122. (Step S102) The display control unit 124 determines whether to execute image conversion or image generation based on the operation data. If image conversion is to be executed (step S102 YES), the display control unit 124 enables the operation of the conversion unit 114 and proceeds to the process of step S104. If image generation is to be executed (step S102 NO), the display control unit 124 enables the operation of the generation unit 112 and proceeds to the process of step S106.

[0052] (Step S104) The display control unit 124 selects, as a fixed condition, image data designated by the operation data or an image feature amount output from the feature extraction unit 116 using the image data as input. The display control unit 124 sets the selected fixed condition in the conversion unit 114 as a certain constraint condition. (Step S106) The display control unit 124 sets the diversity value designated by the operation data to the activation processing unit, which is either the generation unit 112 or the conversion unit 114, whose function has been activated. (Step S108) The display control unit 124 sets the value of the step width instructed by the operation data in the validity processing unit.

[0053] (Step S110) The display control unit 124 waits for operation data indicating pressing of the generate button. When the display control unit 124 detects pressing of the generate button (YES in step S110), the display control unit 124 proceeds to processing in step S112. When the display control unit 124 does not detect pressing of the generate button (NO in step S110), the display control unit 124 repeats the processing in step S110. (Step S112) The validity processing unit executes a noise removal step, and outputs the acquired noise-removed image to the display control unit 124. The display control unit 124 causes the display unit 122 to display a display image including the noise-removed image input from the validity processing unit for each step interval.

[0054] (Step S114) The display control unit 124 judges whether or not the execution of all the noise removal steps is completed. The display control unit 124 may wait for operation data indicating the stop of the noise removal steps. The stop of the noise removal steps is instructed, for example, by pressing a stop button (not shown). The user judges whether or not the desired image is obtained while looking at the noise-removed image for each noise removal step displayed in the preview display unit D. If the desired image is obtained, the user presses the stop button. When the noise removal steps are completed or instructed to be stopped (step S114 YES), the display control unit 124 stores an output image showing the noise-removed image at that time as an output image in association with the number of steps at that time in the storage unit 130. If the execution of the noise removal steps is not completed, the display control unit 124 instructs the validity processing unit to stop the noise removal steps. Thereafter, the process of FIG. 5 is terminated. When the noise removal steps are not completed and the stop of the noise removal steps is not instructed (step S114 NO), the process proceeds to step S116.

[0055] (Step S116) The display control unit 124 uses the control condition selection unit C to acquire setting data including one or both of image data and text according to the operation data. The setting data indicates the characteristics of the desired image. The image characteristics may indicate the distribution of component colors, brightness, etc., the shape of a specific pattern, as well as psychological characteristics such as texture and emotion. The display control unit 124 outputs the acquired setting data to the feature extraction unit 116. The feature extraction unit 116 converts the setting data input from the display control unit 124 into two-dimensional feature amounts, and outputs the acquired two-dimensional feature amounts to the display control unit 124. The display control unit 124 acquires the two-dimensional feature amounts input from the feature extraction unit 116 as constraint conditions. The display control unit 124 sets an application period of the obtained constraint condition according to the operation data. The display control unit 124 causes the validity processing unit to execute a noise removal step using the obtained constraint condition during the application period that has been set. Then, the process returns to step S114.

[0056] Therefore, while checking the noise-removed image displayed on the display unit 122, the user can repeat the task of applying one or more types of constraint conditions to the noise removal step for any period of time until the desired image is obtained. Unlike the existing diffusion model, different constraint conditions can be applied to each step without waiting for the end of a series of image processing. Since it is possible to give more detailed instructions for adjustment of the details according to the obtained noise-removed image, it is possible to more reliably obtain an output image close to the desired image.

[0057] In addition, by recording the seed value, the constraint conditions used in the processing, and the information on the application interval for each constraint condition in the history data, the user can refer to them, trace back to any step, and specify the noise attenuation image obtained at that time as the application interval or the output point of the output image. Then, the user can change the constraint conditions from the start point of the application interval and resume the noise removal step. This also contributes to obtaining an output image close to the desired image.

[0058] Next, an example of the functional configuration of the generation unit 112 according to this embodiment will be described. Fig. 6 is a schematic block diagram showing an example of the functional configuration of the generation unit 112 according to this embodiment. In the example of Fig. 6, the generation unit 112 implements an LDM. The generation unit 112 includes an encoder 112a, a spreading unit 112b, a despreading unit 112c, and a decoder 112d.

[0059] The encoder 112a converts image data representing an input image x, which indicates a signal value for each pixel, into a latent representation z. The encoder 112a outputs the obtained latent representation to the diffusion unit 112b. The input image x is, for example, three-dimensional data indicating the signal value of each pixel distributed on a two-dimensional plane stretched in the horizontal and vertical directions for each color. For example, RGB image data is applied as the image data of the input image x. The latent representation is, for example, three-dimensional data that has been downsampled so that the number of elements in the horizontal and vertical directions is less than that of the original input image x. The number of elements in the horizontal and vertical directions of the latent representation z is, for example, 1 / 2m of the number of elements in the input image x. m is an integer equal to or greater than 1.

[0060] The decoder 112d reconstructs an input image x' from the latent representation z. The amount of data of the reconstructed input image x' is equal to the amount of data of the original input image x input to the encoder 112a. That is, in the example of Fig. 6, a latent representation with a smaller amount of data than the image data is processed in the diffusion process and the de-diffusion process (latent space). The encoder 112a and the decoder 112d function as an autoencoder. The model of the autoencoder is, for example, a variational autoencoder (VAE). The model of the autoencoder is trained in advance so that the difference between the original input image x and the reconstructed input image x' is minimized.

[0061] The diffusion unit 112b performs a diffusion process on the latent representation z input from the encoder 112a to obtain the latent representation z T The noise component is generated as follows. The diffusion process is a process of repeatedly executing a noise addition step in which a noise component is added to the data to be processed to generate noise-added data. T indicates the number of steps, that is, the number of times the noise addition step is repeated. The number of steps T is an integer equal to or greater than 2. The number of steps T is typically about 1000. The noise component is, for example, Gaussian noise. The latent representation z TThe average value and variance relating to the distribution of the sample values ​​constituting the noise component added at each step are preset so that the noise component can be obtained as

[0062] The despreading unit 112c demultiplexes the latent representation z T A de-diffusion process is performed on the data to generate a latent representation z corresponding to the image data. The de-diffusion process is a process of repeatedly performing a noise removal step T times to remove noise components from the data to be processed and generate noise-removed data. The de-diffusion unit 112c includes T de-diffusion calculation units 112c,t (t is an integer between 1 and T), and each de-diffusion calculation unit 112c,t performs the noise removal step for step t. t The denoised data is treated as the data to be processed, and the latent representation z t Here, the initial value of t is T, and the denoised data obtained for the first denoised data is the latent representation z. The model of the dediffusion calculation unit 112c,t is the latent expression z t The despreading calculation unit 112c,t uses the predicted noise component as the latent representation z t The noise-removed data is subtracted from the latent representation z t-1 Derive:

[0063] If the constraint condition is not set, the feature quantity is not input from the feature extraction unit 116 to the despreading unit 112c. In that case, the model of the despreading calculation unit 112c,t is trained so that the noise component to be removed as the output for the input noise component is set as the target value, and the estimated value for the input approaches the target value. As the target value, the noise component added in a step selected using a uniform random number among the noise addition steps executed by the despreading calculation unit 112c,t is used. As a loss function indicating the degree of approximation of the estimated value to the target value, for example, the mean square error (MSE) between the estimated value and the target value is used. A plurality of input images are used as training data. The MSE is the average value of the squared value of the difference between the estimated value and the target value for each input image.

[0064] The model of the despreading calculation unit 112c,t includes a plurality of hierarchical calculation blocks. Output data obtained from a certain layer is applied as input data to the next layer. The model of the despreading calculation unit 112c,t is, for example, U-Net. U-Net is a kind of autoencoder, and has an encoder 112e and a decoder 112f, which are connected in series in that order. The plurality of layers may include an attention layer. The attention layer may be two or more layers (multi-head attention). Each attention layer has a model that performs calculations on the output data from the previous layer and the feature amount τ input from the feature extraction unit 116, and outputs the obtained output data to the next layer.

[0065] In the example of FIG. 6, each attention layer calculates three types of components using the output data from the previous layer and the feature amount τ input from the feature extraction unit 116. The attention layer calculates a weight matrix W Q The matrix obtained by multiplying these is calculated as a query Q. The attention layer calculates a weight matrix W K , W V The attention layer multiplies the matrix by K and the resulting matrix by V (value). T The softmax function of the matrix obtained by multiplying the query by the query and dividing it by the square root of the length d of the predefined embedding vector is multiplied by the value V to obtain the softmax(QK TAttention, which indicates the function value of √d / √dV, is calculated. The softmax function is a function in which the output function value increases monotonically as the input real value increases, and the minimum and maximum function values ​​are 0 and 1, respectively. Therefore, the higher the correlation between the output data from the immediately preceding layer and the feature amount, the larger the positive value of attention becomes. Therefore, the degree of association between the constraint condition and each form of output data is estimated for each attention layer. Note that the attention layer is not limited to this, as long as it has a model that outputs a larger value as the correlation between the output data from the immediately preceding layer and the feature amount increases.

[0066] The model of the despreading calculation unit 112c,t may not include an attention layer. In that case, the despreading calculation unit 112c,t receives the latent representation z t The feature amount τ is input from the feature extraction unit 116, and the latent expression z t-1 The de-diffusion calculation unit 112c,t outputs the latent representation z t-1 The model for estimating the latent representation z t The de-diffusion calculation unit 112c,T receives the combined data obtained by combining the feature amount τ and the latent representation z T The de-diffusion calculation unit 112c,1 receives the latent expression z 0 In this case, too, the constraint conditions are applied to the model of the de-diffusion calculation units 112c,t that execute the individual noise removal steps, so that the constraint conditions are applied on a step-by-step basis, and the noise removal process is executed under the applied constraint conditions.

[0067] When the constraint conditions are set, the feature amount is input from the feature extraction unit 116 to the despreading unit 112c. In this case, the estimated value output from the model of the despreading calculation unit 112c,t in response to the input may differ depending on the constraint conditions. Therefore, training data including a plurality of different data sets including combinations of noise components and constraint conditions to be input in learning the model of the despreading calculation unit 112c,t is used. When, for example, the MSE of the estimated value and the target value is used as the loss function, the MSE is the average value of the squared value of the difference between the estimated value and the target value for each pair of the input image and the constraint conditions.

[0068] Note that the order of steps t in the de-diffusion process illustrated in FIG. 6 is reversed from the order of steps t in FIGS. 3 and 4 in that it is in descending order over time. T corresponds to the latent representation corresponding to the input image according to the above-mentioned FIGS. 1 to 5. The generation unit 112 illustrated in FIG. 6 generates a latent representation z T In this case, the encoder 112a may be omitted. The reconstructed input image x' shown in FIG. 6 corresponds to the output image described above with reference to FIGS. Furthermore, if the model of the despreading unit 112c has already been trained, the spreading unit 112b may be omitted. The conversion unit 114 may also have a model similar to that of the generation unit 112 illustrated in FIG. 6, or one or both of the encoder 112a and the diffusion unit 112b may be omitted.

[0069] As described above, the information processing system S1 or the information processing device 10 according to this embodiment includes an image processing unit (e.g., image processing unit 110) that generates an output image by repeating a noise removal step of removing noise components from an input image multiple times, and a user interface unit (e.g., user interface unit 120) that outputs a noise-removed image acquired during the multiple noise removal steps and acquires parameters related to the removal of noise components based on operation data. The image processing unit executes the noise removal step using the acquired parameters (e.g., step width, diversity, constraint conditions). According to this configuration, the user can set parameters to be used in executing the noise removal steps, using the noise-removed images obtained before the multiple noise removal steps are completed as a clue. The user can adjust the parameters and restart the noise removal steps while referring to the obtained noise-removed images, without waiting for the series of noise removal steps to be completed, thereby more efficiently obtaining a desired output image.

[0070] The user interface unit may acquire at least one of text, an image, and an image feature quantity indicating a constraint condition corresponding to a characteristic of the output image, and the image processing unit may execute the noise removal step by applying the acquired constraint condition. According to this configuration, an output image having corresponding characteristics can be obtained under constraint conditions indicated by at least one of text, image, and image feature quantities. By using information that can be recognized by the user in specifying the constraint conditions, the desired output image can be obtained more efficiently.

[0071] The user interface unit may output the noise-removed images acquired in the noise removing steps in a juxtaposed manner between different steps. With this configuration, the user can view the noise-removed images obtained in different steps, and can adjust parameters, etc. while observing the progress of the noise-removed images to more efficiently obtain the desired output image.

[0072] The user interface unit may obtain an application period of the constraint condition, and output, for each constraint condition, a denoised image obtained by the noise removal step during the application period in association with the application period. According to this configuration, the user can view the noise-removed image at each step within the application period for each constraint condition, and can grasp the progress of the noise-removed image by associating the constraint condition with the application period. By adjusting parameters and the like while observing the progress of the noise-removed image according to the constraint condition, the desired output image can be obtained more efficiently.

[0073] When updating the constraint conditions, the user interface unit replaces the noise-removed image acquired using the constraint conditions before the update with the noise-removed image acquired using the constraint conditions after the update, and outputs the replaced image. According to this configuration, the noise-removed image before and after updating the constraint conditions, as well as the change over time, are displayed, so that the user can easily grasp the effect and degree of the effect of the constraint conditions on the noise-removed image.

[0074] The user interface unit may store in the memory unit 130 for each trial a noise-removed image obtained by multiple noise removal steps, a constraint condition, and history data indicating the application period for each constraint condition, refer to the history data, and construct for each trial a history display that associates the noise-removed image obtained by the noise removal step during the application period for each constraint condition with the application period, and output the history display in juxtaposition between trials. According to this configuration, a history display for each trial including the noise-removed image at each step within the application period for each constraint condition is presented across multiple trials. Therefore, the user can view the transition of the noise-removed image between trials by associating the constraint condition with the application period. Therefore, the setting of parameters, etc. for obtaining a desired output image is supported while observing the transition of the noise-removed image due to the difference in the constraint condition or application period for each trial.

[0075] Although the present embodiment has been described above in detail with reference to the drawings, the specific configuration is not limited to the above-mentioned configurations, and includes designs within the scope of the present embodiment. The above-mentioned configurations can be arbitrarily combined, and some of them may be omitted. [Explanation of symbols]

[0076] S1...information processing system, 10...information processing device, 20...user terminal device, 110...image processing unit, 112...generation unit, 112a...encoder, 112b...spreading unit, 112c...despreading unit, 112d...decoder, 114...conversion unit, 116...feature extraction unit, 120...user interface unit, 122...display unit, 124...display control unit, 126...reception unit, 130...storage unit

Claims

1. an image processing unit that generates an output image by repeating a noise removal step multiple times to remove noise components from the input image; outputting a noise-removed image obtained during the multiple times of the noise removal step; a user interface unit for acquiring parameters relating to the removal of the noise components; The image processing unit includes: Performing the denoising step using the parameters Information processing system.

2. The user interface unit includes: obtaining at least one of a text, an image, and an image feature quantity indicating a constraint condition corresponding to a characteristic of the output image; The image processing unit includes: Applying the constraint and performing the noise removal step The information processing system according to claim 1 .

3. The user interface unit includes: The noise-removed images obtained by the noise removal steps are output in a juxtaposed manner between different steps. The information processing system according to claim 1 .

4. The user interface unit includes: Obtaining a period during which the constraint is applicable; The noise-removed image acquired by the noise removal step during the application period is output in association with the application period for each of the constraint conditions. The information processing system according to claim 2 .

5. The user interface unit includes: When updating the constraints, The noise-removed image acquired using the constraint condition before the update is replaced with the noise-removed image acquired using the constraint condition after the update, and the noise-removed image is output.

5. The information processing system according to claim 4.

6. The user interface unit includes: storing, in a storage unit for each trial, a noise-removed image obtained by performing the noise removal step a plurality of times, the constraint conditions, and history data indicating an application period for each of the constraint conditions; With reference to the history data, a history display is constructed for each trial, in which the noise-removed image acquired by the noise removal step during the application period is associated with the application period for each of the constraint conditions. The history display for each trial is output side by side.

5. The information processing system according to claim 4.

7. an image processing unit that generates an output image by repeating a noise removal step multiple times to remove noise components from the input image; outputting a noise-removed image obtained during the multiple times of the noise removal step; a user interface unit for acquiring parameters relating to the removal of the noise components; The image processing unit includes: Performing the denoising step using the parameters Information processing device.

8. Equipped with an image processing unit and a user interface unit An information processing method in an information processing system, comprising: the image processing unit executes an image generation procedure for generating an output image from input data by repeating a noise removal step a plurality of times to remove noise components from a noise removed image of a previous step to obtain a noise removed image of a current step; the user interface unit outputs a noise-removed image acquired during a plurality of times of the noise removal step; Executing a user interface procedure for acquiring parameters related to the removal of the noise components; In the image generating step, The noise removal step is performed using the parameters. Information processing methods.

Images