Information processing device, information processing method, and information processing program

The information processing device addresses the challenge of generating images without a pre- and post-color conversion set by learning a diffusion model to accurately apply color changes, enhancing image quality and applicability.

JP7580441B2Active Publication Date: 2024-11-11LY CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022202512
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-12-19
Publication Date
2024-11-11
Estimated Expiration
2042-12-19

AI Technical Summary

Technical Problem

Conventional techniques struggle with generating appropriate images when a set of an image before and after color conversion is unavailable, leading to difficulties in creating images that accurately reflect desired color changes.

Method used

An information processing device that learns a diffusion model by grayscaling images and using labels to approximate the original image colors, allowing it to generate images by gradually removing noise and applying desired color attributes.

Benefits of technology

Enables the generation of appropriate images by accurately converting colors and attributes, improving image quality and suitability for applications such as image search and product recommendation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580441000001
    Figure 0007580441000001
  • Figure 0007580441000002
    Figure 0007580441000002
  • Figure 0007580441000003
    Figure 0007580441000003
Patent Text Reader

Abstract

To enable appropriate image generation.SOLUTION: An information processing apparatus includes an acquisition unit and a learning unit. The acquisition unit acquires a learning grayscale image which is obtained by representing a learning image, which is a captured image of a predetermined target, by grayscale, and a label indicating characteristics of the predetermined target in the learning image. The learning unit learns a diffusion model which transforms one noise image, which is an image including noise, so as to approximate the learning image, on the basis of the learning grayscale image and the label acquired by the acquisition unit.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there have been provided techniques for converting various types of color attribute information related to the colors of an image. For example, there has been provided a technique for learning a color conversion model using a pair of an image before and after color conversion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-149825 [Non-patent literature]

[0004] [Non-Patent Document 1] “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”, Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros<Internet>https: / / junyanz.github.io / CycleGAN / (Retrieved December 5, 2022) [Non-Patent Document 2] “Denoising Diffusion Probabilistic Models”, Jonathan Ho, Ajay Jain, and Pieter Abbeel, <Internet>https: / / arxiv.org / pdf / 2006.11239.pdf (Retrieved December 5, 2022) [Non-Patent Document 3] “Palette: Image-to-Image Diffusion Models”, Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi, <Internet> https: / / arxiv.org / pdf / 2111.05826.pdf (retrieved December 5, 2020) Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is room for improvement in the above conventional techniques. For example, when a pair of an image before color conversion and an image after color conversion is required and a pair of an image before color conversion and an image after color conversion does not exist, it is difficult to generate an image that has undergone color conversion corresponding to the pair. Therefore, it is desired to enable appropriate image generation.

[0006] The present application has been made in view of the above, and has an object to provide an information processing device, an information processing method, and an information processing program that enable appropriate image generation. [Means for solving the problem]

[0007] The information processing device according to the present application is characterized in that it includes an acquisition unit that acquires a learning grayscale image in which a learning image, which is an image of a specific object, is grayscaled, and a label indicating features of the specific object in the learning image, and a learning unit that learns a diffusion model that converts a noise image, which is an image containing noise, so as to approach the learning image, based on the learning grayscale image and the label acquired by the acquisition unit. Effect of the Invention

[0008] According to one aspect of the embodiment, an effect is achieved in that appropriate image generation can be enabled. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating another example of information processing according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a model information storage unit according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of information processing according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of information processing according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments. In addition, the same parts in the following embodiments are denoted by the same reference numerals, and duplicated descriptions are omitted.

[0011] (Embodiment) [1. Information Processing] An example of information processing according to the embodiment will be described below with reference to FIG. 1. FIG. 1 is a diagram showing an example of information processing according to the embodiment. Specifically, FIG. 1 shows an example of information processing including a learning process of a model according to the embodiment and a generation process of an image using the model. In FIG. 1, a case will be described as an example in which an information processing device 100 (see FIG. 3) learns (generates) a diffusion model M1 (hereinafter also referred to as "model M1"), which is a diffusion model for generating an image from a noise image, and generates an image using the generated model M1. Note that, in the following, a case will be described as an example in which a grayscale image including an object whose characteristics such as color are converted is used, but the image including the object is not limited to a grayscale image and any image may be used, which will be described later.

[0012] [1-1. Learning process example] First, an example of the learning process of the model M1 will be described with reference to Fig. 1. First, data (learning data) used in the learning process of the model M1 will be described.

[0013] 1, the information processing device 100 acquires data (learning data) used in the learning process of the model M1. For example, the information processing device 100 acquires the learning data of the model M1 from a learning data storage unit 121 (see FIG. 4). Note that, when the learning data is not registered in the learning data storage unit 121, the information processing device 100 may request an external device such as the information providing device 50 (see FIG. 3) to provide the learning data, and register the data received from the external device in the learning data storage unit 121 as learning data.

[0014] 1, as an example of data used in the learning process of the model M1, an image group IG1 including a plurality of stage images in which noise is added to a predetermined image in stages is shown. The information processing device 100 learns the model M1 using learning data including the image group IG1 shown in FIG.

[0015] The image group IG1 includes image P10, which is an image (learning image) of a predetermined object. Image P10 is an image of a hoodie as the predetermined object. Image P10 is at stage #0 in terms of noise addition, that is, an image with no noise added. Note that in FIG. 1, colors are shown with hatching for ease of illustration, but the color of the hoodie included in image P10 is blue.

[0016] The image group IG1 also includes an image P11, which is an image in which noise has been added to the image P10. The image P11 is an image in which one level of noise has been added to the image P10. The level of noise addition for the image P11 is level #1, that is, an image in which one level of noise has been added. For example, the image P11 is generated by a noise addition process (process SN1 in FIG. 1) that adds noise to the image P10.

[0017] The image group IG1 also includes an image P12, which is an image obtained by adding noise to the image P11. The image P12 is an image obtained by adding one more level of noise to the image P11. The level of noise addition for the image P12 is level #2, that is, an image to which two levels of noise have been added. For example, the image P12 is generated by a noise addition process (process SN2 in FIG. 1) that adds noise to the image P11.

[0018] The image group IG1 also includes an image P13, which is an image obtained by adding noise to the image P12. For example, a noise adding process (process SN3 in FIG. 1) adds noise to the image P12 to generate an image (image P13) at stage #3, which is an image with three stages of noise added. Note that, since the noise is repeated, a description of the intermediate processes is omitted.

[0019] The image group IG1 also includes an image P1K-2 which is an image obtained by adding noise to an image P1K-3 to which K-3 levels of noise have been added, where K is an arbitrary number (value), for example, 30, 1000, etc. For example, a noise adding process (process SNK-2 in FIG. 1) that adds noise to the image P1K-3 generates an image (image P1K-2) at stage #K-2 to which K-2 levels of noise have been added.

[0020] Furthermore, the image group IG1 includes an image P1K-1, which is an image in which noise has been added to the image P1K-2. The image P1K-1 is an image in which one more level of noise has been added to the image P1K-2. The level of noise addition for the image P1K-1 is level #K-1, that is, the image in which K-1 levels of noise have been added. For example, the image P1K-1 is generated by a noise addition process (process SNK-1 in FIG. 1) that adds noise to the image P1K-2.

[0021] Moreover, the image group IG1 includes an image P1K which is a noise image. For example, the image P1K is an image in which one level of noise is further added to the image P1K-1. The level of noise addition of the image P1K is level #K, that is, the image in which K levels of noise are added. For example, the image P1K is generated by a noise adding process (process SNK in FIG. 1) that adds noise to the image P1K-1. Processes SN1 to SNK in FIG. 1 correspond to a forward process in a diffusion model.

[0022] For example, each image in the image group IG1 may be generated by a process (noise adding process) in which Gaussian noise is added to data and the data gradually transitions from real data to complete Gaussian noise. For example, when K is 1000, each image in the image group IG1 is generated by mixing data and noise while changing the ratio so that the data becomes complete noise after 1000 noise adding processes. Note that the case where K is 1000, that is, the number of times of noise adding processes is 1000, is merely an example, and the number of times of noise adding processes may be less than 1000 times, such as 32 times or 1 time, or may be more than 1000 times. In addition, each image in the image group IG1 may be generated by the information processing device 100 or an external device such as the information providing device 50.

[0023] In this way, the image group IG1 shown in FIG. 1 includes a plurality of images such as images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc. Hereinafter, a case where the information processing device 100 learns the model M1 using the image group IG1 described above will be described as an example. Note that the above is merely an example, and the images included in the image group IG1 may be any images. For example, the image P1K may be a noise image such as so-called white noise.

[0024] From here, details of the learning process example will be described. In FIG. 1, the information processing device 100 executes a learning process for learning a model M1 using an image group IG1 and the like (step S1). The information processing device 100 inputs an image containing noise (noise image) and learns the model M1 to output (predict) an image obtained by removing noise from the previous process, that is, the noise image by one step. For example, the information processing device 100 learns the model M1 using as input a noise image, a grayscale image obtained by grayscaling an original image (an image without noise) corresponding to the noise image, and a label indicating a feature such as a color of an object included in the original image.

[0025] In this way, the information processing device 100 trains the model M1 so as to generate an image in which noise has been removed by one step from a noise image of one step. In FIG. 1, the information processing device 100 trains the model M1 using a grayscale image (also referred to as a "training grayscale image") obtained by grayscaling the image P10 and a label (also referred to as a "training label") indicating that the color of the hoodie in the image P10 is blue. In this way, the information processing device 100 can train the model M1 as a conditional diffusion model by learning information (image) indicating an object to be included in the image to be generated and a label indicating the feature of the object as input.

[0026] For example, the information processing device 100 executes a learning process (process SR1 in FIG. 1) for learning a model M1 to output an image P1K-1 using an image P1K, a learning grayscale image (e.g., grayscale image GS1), and learning labels as inputs. In this manner, the information processing device 100 trains the model M1 to generate an image P1K-1, which is a noise image at the stage #K-1 of the previous process, from the image P1K, which is a noise image at the stage #K.

[0027] Furthermore, the information processing device 100 executes a learning process (process SR2 in FIG. 1) for training the model M1 to output an image P1K-2, which is a noise image of stage #K-2, using as input an image P1K-1, which is a noise image of stage #K-1, a learning grayscale image, and a learning label. Similarly, the information processing device 100 executes a learning process (process SR3 in FIG. 1) for training the model M1 to output an image P1K-3, which is a noise image of stage #K-3, using as input an image P1K-2, which is a noise image of stage #K-2, a learning grayscale image, and a learning label. Note that the processes between process SR3 and process SRK-2 in FIG. 1 are similar to the above-mentioned processes in that they train the model M1 to output an image one stage before the noise addition of the noise image using as input an image, and therefore detailed explanations thereof will be omitted.

[0028] The information processing device 100 executes a learning process (process SRK-2 in FIG. 1) for training a model M1 to output an image P12, which is a noise image of stage #2, using as input image P13, which is a noise image of stage #3, a learning grayscale image, and a learning label. The information processing device 100 executes a learning process (process SRK-1 in FIG. 1) for training a model M1 to output an image P11, which is a noise image of stage #1, using as input image P12, which is a noise image of stage #2, a learning grayscale image, and a learning label.

[0029] Then, the information processing device 100 executes a learning process (process SRK in FIG. 1) for learning a model M1 so as to output image P10, which is an image at stage #0, i.e., an image to which no noise has been added, using image P11, which is a noise image at stage #1, a learning grayscale image, and learning labels as input. Processes SR1 to SRK in FIG. 1 correspond to a generation process (reverse process) in a diffusion model. In FIG. 1, the information processing device 100 learns a model M1 using an image group IG1 including images P10, P11, P12, ... P1K-2, P1K-1, P1K, etc.

[0030] In this way, the information processing device 100 learns the model M1, which is a model that generates data by inputting a noise image and gradually removing noise. In Fig. 1, the information processing device 100 learns the model M1, which receives a noise image, a grayscale image, and a label as input, and gradually removes noise from the noise image to generate data (also called a "converted image") in which the features indicated by the label are reflected in the object included in the grayscale image.

[0031] 1, the information processing device 100 learns a model M1 that generates a converted image such that a hoodie, which is an object included in a grayscale image, is colored "blue" as indicated by a label by gradually removing noise from the noisy image. In this way, the information processing device 100 learns a model M1 that converts a noisy image so as to approach the learning image, based on the grayscale image and the label.

[0032] The learning process using the image group IG1 shown in FIG. 1 is a part of the learning of the model M1, and the information processing device 100 learns the model M1 using learning data including various image groups other than the image group IG1. For example, the information processing device 100 learns the model M1 using an image group including learning images of various transaction objects such as shoes, bags, and hats, not limited to hoodies. For example, the information processing device 100 learns the model M1 using an image group including learning images of various colors such as red, yellow, green, and purple, not limited to blue. The above-mentioned process is merely an example, and the information processing device 100 may learn the model M1 based on a learning method related to a diffusion model disclosed in Non-Patent Documents 2 and 3.

[0033] In this way, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the learning image based on a grayscale image of the learning image including a predetermined object and a label indicating the characteristics of the predetermined object. This allows the information processing device 100 to learn a model that converts the characteristics of the predetermined object in the image. Therefore, the information processing device 100 can generate an appropriate image.

[0034] In the above example, the information processing device 100 inputs the color domain (red, blue, etc.) of the object such as a product included in the original image as a label together with the grayscale image to the diffusion model, and learns the diffusion model. For example, if images are collected by color when they are collected, the labels are also known and learning data can be easily collected, so that the information processing device 100 can easily learn the model by self-supervised learning.

[0035] [1-2. Example of generation process] Next, an example of an image generation process using the model M1 will be described. In FIG. 1, for the sake of explanation, an image P10 is used as a target image for generating a converted image (also referred to as a "target image"). However, the target image is not limited to the image P10 and may be any image. For example, the information processing device 100 receives a designation from a user that the image P10 is the target image.

[0036] 1 shows a case where a label LB1 is used to indicate that the feature to be converted (also referred to as a "target feature") is the color "green", but the target feature is not limited to green and may be any color such as red, brown, etc. For example, the information processing device 100 accepts the label LB1 from a user.

[0037] First, the information processing device 100 generates a grayscale image of the target image by performing a grayscale process of the target image (step S11). The information processing device 100 performs grayscale conversion of the image by appropriately using various techniques related to grayscaling of the image. In FIG. 1, the information processing device 100 generates a grayscale image GS1 of the image P10 by performing a grayscale process of the image P10.

[0038] Then, the information processing device 100 executes a generation process to generate a converted image using the noise image, grayscale image, and label as input (step S12). For example, the noise image NZ1 may be the same noise image as the image P1K, or may be a different noise image. In FIG. 1, the information processing device 100 generates an image P20 by repeating image generation using the model M1 with the noise image NZ1, grayscale image GS1, and label LB1 as input information IN1 to the initial model M1. Note that in FIG. 1, colors are shown by hatching for ease of illustration, but the color of the hoodie included in the image P20 is green.

[0039] For example, the information processing device 100 generates an image P20 by repeating the input to the model M1 K times. For example, when K is 1000, the information processing device 100 repeats the input to the model M1 999 times, using as input the noise image NZ1, the grayscale image GS1, and the image output by the model M1 to which the label LB1 has been input.

[0040] For example, in the first (first) iterative process, the information processing device 100 inputs a noise image NZ1, a grayscale image GS1, and a label LB1 to the model M1 and causes the model M1 to output an image #1. In the second iterative process, the information processing device 100 inputs an image #1, a grayscale image GS1, and a label LB1 to the model M1 and causes the model M1 to output an image #2. In the third iterative process, the information processing device 100 inputs an image #2, a grayscale image GS1, and a label LB1 to the model M1 and causes the model M1 to output an image #3.

[0041] Similarly, in the 4th to K-1th iterations of the repetition process, the information processing device 100 inputs each of the images #3 to #K-2, the grayscale image GS1, and the label LB1 to the model M1, and causes the model M1 to output each of the images #4 to #K-1. Then, in the Kth iteration of the repetition process, the information processing device 100 inputs the image #K-1, the grayscale image GS1, and the label LB1 to the model M1, and causes the model M1 to output the image P20.

[0042] In this way, the information processing device 100 gradually removes noise from the noise image to generate an image P20 in which the feature indicated by the label LB1 is reflected in the object included in the grayscale image GS1. In Fig. 1, the information processing device 100 gradually removes noise from the noise image using the model M1 to generate an image P20, which is a converted image in which the hoodie, which is an object included in the grayscale image GS1, is colored "green" indicated by the label LB1.

[0043] [1-3. Flattening] Although the above-mentioned process can generate an image that reflects the desired features of a desired object, there is still room for improvement. For example, when learning a diffusion model using an input image (learning image) that has only been converted to grayscale (grayscale image), when features such as color are converted, the conversion of the features may not be performed appropriately.

[0044] For example, when there is a bias in the distribution of pixel values ​​of each pixel of a grayscale image obtained by grayscaling a learning image, it may be difficult to learn a diffusion model that appropriately generates a converted image. For example, when there is a bias in the distribution of pixel values ​​(e.g., values ​​from 0 to 255) corresponding to brightness (e.g., gradations from white to black) among multiple types of color attribute information of each pixel of a grayscale image, it may be difficult to learn a diffusion model that appropriately generates a converted image.

[0045] Therefore, the information processing device 100 generates an image in which the distribution of color attribute information of a predetermined type in the learning image is flattened (also called a "flattened image"), and uses the generated flattened image to learn the diffusion model. An example of this point will be described with reference to Fig. 2. Note that the same points as those in Fig. 1 will be appropriately omitted by assigning the same reference numerals, etc.

[0046] For example, the information processing device 100 generates a grayscale image in which the lightness distribution in the learning image is flattened as a flattened image, and learns a diffusion model using the grayscale image in which the lightness distribution is flattened (also called the "flattened grayscale image"). Note that lightness is merely an example of color attribute information to be flattened, and color attribute information to be flattened is not limited to lightness, but this point will be described later.

[0047] The histogram HG1 in FIG. 2 is a histogram showing the distribution of brightness of each pixel in the grayscale image GS1 when the flattening process is not performed. The histogram HG1 corresponds to the histogram before the flattening process. In FIG. 2, the shade of hatching in the horizontal bar from white to black indicates the high and low of the pixel value. For example, the white (right end) of the histogram HG1 corresponds to the minimum value (e.g., 0) of the pixel value (e.g., brightness) that a pixel can have in the grayscale image, and the black (left end) corresponds to the maximum value (e.g., 255) that a pixel can have in the grayscale image. The vertical direction of the histogram HG1 indicates the number (number) of pixels having each pixel value. For example, the low (bottom end) of the histogram HG1 indicates that the number of pixels having the corresponding pixel value is 0, and the high (top end) of the histogram HG1 indicates that the number of pixels having the corresponding pixel value is large. The waveform in the histogram HG1 indicates the number of pixels having each pixel value. In FIG. 2, the histogram HG1 corresponding to the grayscale image GS1 is a histogram in which the number (occurrence frequency) of pixels having certain pixel values ​​(values ​​near the middle between black and white) is large, and the distribution is biased.

[0048] Moreover, histogram HG2 in Fig. 2 is a histogram showing the distribution of brightness of each pixel in grayscale image GS2 when flattening processing is performed. Histogram HG2 corresponds to the histogram after flattening processing. Note that the explanation of the same points in histogram HG2 as in histogram HG1 will be omitted as appropriate. In Fig. 2, histogram HG2 corresponding to grayscale image GS2 is a histogram in which there is no bias in the distribution of the number of pixels having each pixel value from white to black, and the number of each pixel value is uniform overall.

[0049] The flattening process referred to here is a histogram flattening process, which converts the density so that the histogram of a predetermined pixel value becomes flat (uniform) overall. For example, the flattening process converts the density so that the histogram of a predetermined type of color attribute information (pixel values ​​corresponding to a predetermined type, etc.) among multiple types of color attribute information of each pixel in an image becomes flat overall. Various conventional techniques can be used for the histogram flattening process, and detailed explanations will be omitted.

[0050] For example, when generating a grayscale image from image P10 in Fig. 1, the information processing device 100 performs a flattening process in addition to a grayscale conversion process to generate a grayscale image GS2 that is a flattened grayscale image of image P10. Note that the information processing device 100 may also perform a flattening process on the grayscale image GS1 to generate the grayscale image GS2 that is a flattened grayscale image.

[0051] The information processing device 100 learns a diffusion model using a grayscale image GS2 that is a flattened grayscale image generated by a flattening process. That is, the information processing device 100 learns a diffusion model using the grayscale image GS2 that is a flattened grayscale image as an input to the model M1.

[0052] In this case, the information processing device 100 uses the flattened grayscale image as the learning grayscale image to train the model M1 in the learning process shown in Fig. 1. For example, the information processing device 100 trains the model M1 using the grayscale image GS2 as the learning grayscale image in the learning process shown in Fig. 1. Note that the learning process using the flattened grayscale image is similar to the learning process shown in Fig. 1 except that the grayscale image input to the model M1 is the flattened grayscale image, and therefore a detailed description thereof will be omitted.

[0053] In this way, the information processing device 100 generates a flattened image in which the distribution of brightness, which is a predetermined type of color attribute information, is flattened from a learning image in which multiple types of color attribute information are associated with each pixel. Then, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the learning image, based on the generated flattened image and the label. This allows the information processing device 100 to appropriately learn the model using a flattened image in which there is no bias in the predetermined type of color attribute information. Therefore, the information processing device 100 can enable appropriate image generation.

[0054] For example, if there is a bias in the combination of histograms of grayscale images input for each color label specified when training a diffusion model, and the diffusion model is trained including this bias, when a combination of a blue image turned gray and a white label is input during color conversion, there may be a large difference from the data distribution at the time of training, and the conversion may not be successful.

[0055] Therefore, when learning a model, the information processing device 100 can appropriately learn the model by using an image that has been subjected to not only grayscale conversion but also histogram equalization as an input image. For example, the diffusion model generated by the information processing device 100 not only colors a grayscale image but also learns to convert an equalized histogram into a histogram according to a specified color. This allows the information processing device 100 to generate a converted image that is well colored even when a label of an arbitrary color different from that of the original image is replaced during coloring.

[0056] In addition, when using a flattened grayscale image, in the generation process of an image using the model M1, the information processing device 100 may generate a converted image using the flattened grayscale image of the target image. In this case, in step S11 of the generation process shown in FIG. 1, the information processing device 100 generates a grayscale image GS2 that is a flattened grayscale image of the image P10 by performing a flattening process in addition to a grayscale conversion process (grayscale conversion process). Then, the information processing device 100 generates a converted image using the flattened grayscale image. Note that the generation process using the flattened grayscale image is similar to the generation process shown in FIG. 1 except that the grayscale image input to the model M1 becomes a flattened grayscale image, and therefore a detailed description will be omitted.

[0057] [1-4. Other (removal of restrictions, services, etc.)] The above-mentioned process is merely an example, and the information processing device 100 may perform the process using various information as appropriate. In this regard, an example will be described below. For example, in the above-mentioned example, a grayscale image is used as an image including a target to be input to the model M1 (also called a "target captured image"), but the target captured image is not limited to a grayscale image, and any image such as a color image may be used.

[0058] In the above example, the case where the color is the conversion target has been described, but the information processing device 100 may convert any object such as the shape, atmosphere, etc. of the object, not limited to the color of the object contained in the image. For example, the information processing device 100 converts at least one of the color, shape, and atmosphere of the object contained in the image.

[0059] For example, when the shape of an object included in an image is a conversion target, the information processing device 100 uses a label indicating the shape of the object (also called a "shape label") as a label to be input to the learning diffusion model. For example, during the learning process, when the shape of the object included in the image is rectangular, the information processing device 100 uses a shape label indicating a rectangle as a label during learning using the image.

[0060] Also, for example, during the generation process, when the information processing device 100 wants to make the shape of an object included in a target image circular, the information processing device 100 generates a transformed image whose shape has been transformed by inputting a shape label indicating a circular shape together with the target image to the diffusion model. Note that the above is merely an example, and the information processing device 100 learns the diffusion model using appropriate information according to the conversion object, and generates a transformed image using the learned diffusion model.

[0061] For example, in the above example, the case where the lightness is the target of flattening among multiple types of color attribute information has been described, but the information processing device 100 may flatten any target related to the image, not limited to the lightness. For example, the information processing device 100 may flatten hue, saturation, etc.

[0062] Furthermore, the information processing device 100 may flatten color attribute information according to the color expression format of the image. For example, when the color expression format of the image is RGB (red, green, blue), the information processing device 100 may flatten at least one of the values ​​related to R (red), G (green), and B (blue) associated with each pixel of the image. For example, when the values ​​related to G (green) are flattened, the information processing device 100 executes a process of density conversion so that the histogram of pixel values ​​related to G (green) in the image becomes uniform overall.

[0063] Furthermore, for example, when the color expression format of an image is CMYK (cyan, magenta, yellow, black (key plate)), the information processing device 100 may flatten at least one of the values ​​related to C (cyan), M (magenta), Y (yellow), and K (black) associated with each pixel of the image. For example, when the information processing device 100 flattens the values ​​related to K (black), it executes a process of density conversion so that the histogram of pixel values ​​related to K (black) in the image becomes uniform overall.

[0064] Furthermore, the information processing device 100 may provide various services using the generated post-conversion image. In Fig. 1, the information processing device 100 may provide a service using the generated post-conversion image, image P20, to a user (also referred to as a "designating user") who designated image P10, which is a target image. The information processing device 100 transmits search results of a similar image search similar to image P20 to the terminal device 10 used by the designating user.

[0065] The information processing device 100 may provide a search service to the designated user. In this case, the information processing device 100 accepts the image P20 as an image that is a search query (also referred to as a "query image"), executes a search process to search for images similar to the query image (image P20) from among various images to be searched, and provides the search results.

[0066] For example, the information processing device 100 executes a search process for images similar to the image P20 using an index of an image (trading target image) including a trading target such as a product put up for sale in an online shopping mall (EC site, etc.), and provides the search results for images similar to the image P20. The information processing device 100 provides information on the search results for images similar to the image P20 to the designated user. The information processing device 100 transmits the search results for images similar to the image P20 to the terminal device 10 used by the designated user.

[0067] As described above, the information processing device 100 learns a diffusion model that converts a noise image to approach a learning image according to a grayscale image of a learning image in which a specific object is captured and a label indicating the characteristics of the specific object. For example, the information processing device 100 learns a diffusion model to generate a plurality of stage images in which noise is added to the learning image in stages, and generate a stage image in which noise is removed from the stage image by one stage. Also, for example, the information processing device 100 acquires an image from a user, converts the acquired image into a grayscale image, generates an image in which a desired object is captured using input characteristics, and uses the generated image to perform an image search for products similar to the specific object in an EC (electronic commerce) service or the like.

[0068] Furthermore, the information processing device 100 generates a flattened image (suggested image) in which the distribution of the predetermined type of color information is flattened, which is a suggested image indicated by the predetermined type of color information, from a learning image in which a predetermined object is photographed and in which multiple types of color attribute information (color information) are associated with each pixel. For example, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the learning image, according to the suggested image and a label indicating the characteristics of the predetermined object. For example, the information processing device 100 may generate a channel estimation model that has learned flattening of a blown-out image and flattening of a normal image. In this case, the information processing device 100 may generate a suggested image using the generated channel estimation model.

[0069] As described above, the information processing device 100 learns to color objects included in a grayscale image as desired. For example, the information processing device 100 prepares an original image and a noise image obtained by adding noise to the original image, and learns to return the noise image to the original image. For example, the information processing device 100 creates and learns various noise images in stages, and the model removes noise from the noise many times, thereby reproducing the original image.

[0070] Furthermore, when learning a diffusion model, the information processing device 100 inputs information indicating an original image (various labels, the name of the subject, the type of scene, the color of the subject, etc.) to learn. The information processing device 100 inputs various images as learning images, noise images, grayscale images of the learning images, and labels, and performs noise removal many times, thereby learning a model that generates an image in which the features indicated by the labels are reflected in the grayscale image.

[0071] As described above, the information processing device 100 learns a model to restore a noise image to a learning image from a flattened image obtained by flattening one channel, which is a predetermined type among a plurality of types of color attribute information, and a label corresponding to the learning image. For example, the information processing device 100 may generate a blown-out highlight image from a normal image and perform learning using the generated blown-out highlight image.

[0072] Furthermore, the information processing device 100 may switch processing depending on the bias of a histogram of a predetermined type of color attribute information among multiple types of color attribute information. For example, when the histogram of the predetermined type of color attribute information has a bias equal to or greater than a predetermined standard, the information processing device 100 may generate a converted image using a GAN model such as StarGAN (Generative Adversarial Network). Furthermore, when the histogram of the predetermined type of color attribute information does not have a bias equal to or greater than a predetermined standard, the information processing device 100 may generate a converted image using a diffusion model.

[0073] Furthermore, the information processing device 100 may generate a converted image in which the features of only a part of the region (range) in the image are converted. In this case, the information processing device 100 may learn a model in which range designation information that designates a part of the region in the image is also input, and may use the learned model to generate a converted image in which only a part of the region is the target of conversion. For example, the information processing device 100 may learn a model in which mask information that masks the rest of the region in the image is input, and may use the learned model to generate a converted image in which only a part of the region is the target of conversion. Furthermore, the above-mentioned label may be any information, not limited to character information, as long as it is information that indicates the feature to be converted. For example, the label may be any image, such as an image indicating an atmosphere, a histogram, or another image.

[0074] 2. Configuration of the Information Processing System Next, a configuration example of the information processing system 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the information processing system according to the embodiment. As shown in Fig. 3, the information processing system 1 includes an information processing device 100, an information providing device 50, and a terminal device 10. The information processing device 100, the information providing device 50, and the terminal device 10 are connected to each other via a predetermined network N so as to be able to communicate with each other by wire or wirelessly. Note that the information processing system 1 shown in Fig. 3 may include a plurality of information processing devices 100, a plurality of information providing devices 50, and a plurality of terminal devices 10.

[0075] The information processing device 100 is a computer that learns a model such as a diffusion model. That is, the information processing device 100 is a learning device that performs learning processing. The information processing device 100 learns a diffusion model based on a learning grayscale image obtained by grayscaling a learning image in which a predetermined object is captured, and a label indicating a feature of the predetermined object in the learning image. For example, the information processing device 100 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approach the learning image.

[0076] Moreover, the information processing device 100 is a generating device that generates a flattened image in which a distribution of a predetermined type of color attribute information is flattened from a learning image in which multiple types of color attribute information are associated with each pixel. For example, the information processing device 100 learns a diffusion model that converts a noise image so as to approach the learning image, based on the flattened image and a label indicating a feature of a predetermined target in the learning image.

[0077] The information processing device 100 receives from the user a designation of a target image to be transformed, and generates a transformed image using a grayscale image of the target image designated by the user and a diffusion model. The information processing device 100 also receives designation information from the user designating features after transformation, and generates a transformed image reflecting the features designated by the user. The information processing device 100 provides the user with a service using the generated transformed image. The information processing device 100 transmits search results of a similar image search similar to the transformed image to the terminal device 10 used by the user.

[0078] The information processing device 100 is a computer that provides a search service to a user who uses the terminal device 10. In this case, the information processing device 100 has a function as a search server that provides the search service. For example, when the information processing device 100 receives a converted image as an image (query image) that is a search query, it executes a search process that searches for an image similar to the query image (converted image) from various images to be searched, and provides the search results.

[0079] The information processing device 100 receives, from the terminal device 10, designation information designating a target image to be transformed and its features, and generates a transformed image in which the features indicated by the designation information are reflected in the objects included in the target image, using a model M1 that is a diffusion model. For example, the information processing device 100 uses a noise image, a target image, and designation information indicating a color designated by a user as input to the model M1, and repeats a process of removing noise to generate a transformed image in which the objects included in the target image are colored in the color indicated by the designation information.

[0080] The information processing device 100 executes a search process based on the converted image and provides the search results. For example, the information processing device 100 may execute a search process using an index of an image (a trading object image) including a trading object such as a product to be searched for, and provide the search results. For example, the information processing device 100 executes a search process for each trading object put up for sale in an online shopping mall (an EC site, etc.) using a trading object image registered as an image showing the trading object.

[0081] For example, the information processing device 100 searches for trading objects whose images are similar to a query image (converted image) from among trading objects put up for sale in the online shopping mall, and extracts information on trading objects whose converted images are similar to the query image (converted image) as search results for similar images. Note that, as long as the information processing device 100 is capable of searching for similar images, it may perform image search processing using any image search technology as appropriate, and detailed description of image search will be omitted.

[0082] The information processing device 100 provides the user with information regarding the search results of similar images. The information processing device 100 transmits the search results of similar images to the terminal device 10 used by the user. For example, the information processing device 100 provides the converted image and price of the transaction object extracted as the search results, information on the store in the online shopping mall that sells the transaction object, and the like.

[0083] In this way, the information processing device 100 provides information on transaction objects whose transaction object images are similar to the converted image serving as the query image. The information processing device 100 may generate a ranking according to the similarity between the converted image and the transaction object image, and provide the generated ranking as a search result. In this case, the information processing device 100 may assign a higher ranking to the transaction object image with a higher similarity to the converted image, generate list information indicating a ranking of transaction objects arranged in descending order of the ranking, and transmit the generated list information to the terminal device 10 used by the user as a search result of similar images.

[0084] Note that the above is merely an example, and the device that provides a search service to the user is not limited to the information processing device 100, and may be another device included in the information processing system 1. For example, the information providing device 50 may provide a search service to the user, and the information processing system 1 may include a search server that provides a search service to the user.

[0085] The information providing device 50 is a server device that provides the information processing device 100 with information (data) that can be used for model learning. For example, the information providing device 50 transmits data that is permitted to be used for model learning to the information processing device 100 as learning data. For example, the information providing device 50 publishes public data on the Internet, and transmits the requested data to the external device that is the request destination, in response to a request from the external device such as the information processing device 100. For example, the information providing device 50 receives request information requesting data from the information processing device 100, and transmits data corresponding to the request information to the information processing device 100 in response to the reception of the request information.

[0086] For example, the information providing device 50 transmits information as learning data to the information processing device 100, in which a group of images in which noise has been gradually added to an image is associated with feature information indicating features of an object included in the image. For example, the information providing device 50 transmits data as learning data to the information processing device 100, in which a plurality of staged images in which noise has been gradually added to an image of a trading object traded on an EC site or the like is associated with feature information indicating features such as the color of the trading object.

[0087] The terminal device 10 is an information processing device used by a user. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. The terminal device 10 may be referred to as a user. In other words, the user may be read as the terminal device 10.

[0088] The terminal device 10 accepts operations by a user. In addition, the terminal device 10 transmits various information to the information processing device 100 and the information providing device 50 in response to the user's operations. For example, the terminal device 10 transmits request information requesting various information to the information processing device 100 and the information providing device 50. In addition, the terminal device 10 receives various information from the information processing device 100 and the information providing device 50.

[0089] The terminal device 10 transmits an image to be converted (target image) designated by the user to the information processing device 100. The terminal device 10 transmits, to the information processing device 100, information indicating the feature designated by the user (designated information).

[0090] The terminal device 10 receives the search results for similar images from the information processing device 100. For example, the terminal device 10 receives from the information processing device 100 the converted image and price of the transaction object extracted as the search results, information on the store in the online shopping mall that sells the transaction object, and the like.

[0091] The terminal device 10 receives information on transaction objects whose images are similar to the converted image serving as a query image from the information processing device 100. The terminal device 10 receives a ranking according to the similarity between the converted image and the transaction object image from the information processing device 100. The terminal device 10 assigns a higher rank to the transaction object images with a higher similarity to the converted image and the transaction object image, and receives list information indicating the ranking in descending order of the transaction objects with the higher rank from the information processing device 100 as a search result of similar images.

[0092] 3. Configuration of Information Processing Device Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 4, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may have an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0093] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to a network by wire or wirelessly, and transmits and receives information to and from the terminal device 10 and the information providing device 50.

[0094] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4, the storage unit 120 according to the embodiment has a learning data storage unit 121 and a model information storage unit 122.

[0095] (Learning data storage unit 121) The learning data storage unit 121 according to the embodiment stores various information (learning data) related to data used for learning. The learning data storage unit 121 stores teacher data used for generating a model. The learning data storage unit 121 stores learning data information including information such as data used for input and correct answer information (correct answer image) corresponding to the data.

[0096] The learning data storage unit 121 stores learning data in which a combination of a noise image, a grayscale image, and a label used for input is associated with image data (correct image) corresponding to the input.

[0097] The learning data storage unit 121 may store various information according to the purpose, not limited to the above. For example, the learning data storage unit 121 stores information indicating the type of input data in association with each input data. For example, the learning data storage unit 121 stores information indicating the type of color attribute to be flattened in association with each data.

[0098] (Model information storage unit 122) The model information storage unit 122 according to the embodiment stores information about a model. For example, the model information storage unit 122 stores information (model data) about a trained model (model) trained (generated) by a learning process. FIG. 5 is a diagram illustrating an example of a model information storage unit according to the embodiment. In the example shown in FIG. 5, the model information storage unit 122 includes items such as "model ID," "purpose," and "model data."

[0099] "Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model Data" indicates the data of the model. Figure 5 shows an example in which conceptual information such as "MDT1" is stored in "Model Data", but in reality, various information constituting the model is included, such as information on the model configuration (network configuration) and information on parameters. For example, "Model Data" includes information including the nodes in each layer of the network, the functions employed by each node, the connection relationships between the nodes, and the connection coefficients set for the connections between the nodes.

[0100] 5, the model (model M1) identified by the model ID "M1" has a use of "image generation", for example, model M1 is a model used to convert or color an object in an image of the object. Also, it is shown that the model data of model M1 is model data MDT1.

[0101] The model information storage unit 122 may store various information according to the purpose, not limited to the above. The model such as the model M1 may be expected to be used as a program module that is a part of artificial intelligence software. The model such as the model M1 may be a program.

[0102] (Control unit 130) Returning to the explanation of Fig. 4, the control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs (corresponding to examples of information processing programs such as learning programs and generation programs) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0103] 4, the control unit 130 has an acquisition unit 131, a generation unit 132, a learning unit 133, a processing unit 134, and a provision unit 135, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may be other configurations as long as they perform the information processing described below.

[0104] (Acquisition part 131) The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from an external information processing device such as the terminal device 10 or the information providing device 50. The acquisition unit 131 may receive various information indicated in the learning data storage unit 121, the model information storage unit 122, etc. from the external information processing device.

[0105] The acquisition unit 131 acquires various pieces of information from the storage unit 120. The acquisition unit 131 acquires various pieces of information from the learning data storage unit 121, the model information storage unit 122, etc. The acquisition unit 131 receives various pieces of information from an external device.

[0106] The acquisition unit 131 receives various information from the terminal device 10 used by the user. For example, the acquisition unit 131 receives a grayscale image including an object to be subjected to color conversion from the terminal device 10. For example, the acquisition unit 131 receives a grayscale image specified by the user from the terminal device 10. For example, the acquisition unit 131 receives feature information indicating features specified by the user using the terminal device 10 from the terminal device 10.

[0107] The acquisition unit 131 receives various information from the information providing device 50. For example, the acquisition unit 131 receives information used for learning from the information providing device 50. For example, the acquisition unit 131 receives learning data used for learning the model M1 from the information providing device 50.

[0108] The acquisition unit 131 acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a predetermined object. The acquisition unit 131 acquires a label indicating a feature of the predetermined object in the learning image. The acquisition unit 131 acquires a label indicating a feature including at least one of a color, a shape, and an atmosphere of the predetermined object in the learning image. The acquisition unit 131 acquires a label indicating a color of the predetermined object in the learning image.

[0109] The acquisition unit 131 acquires one noise image. The acquisition unit 131 acquires a plurality of stage images in which noise is added stepwise to a learning image. The acquisition unit 131 acquires a plurality of stage images including a first stage image and a second stage image in which one stage noise is added to the first stage image. The acquisition unit 131 acquires a plurality of stage images including a third stage image in which one stage noise is added to the second stage image.

[0110] The acquisition unit 131 acquires a target image to be converted from a user, and acquires designation information for designating features from the user.

[0111] (Generation unit 132) The generating unit 132 executes a generation process to estimate various pieces of information. For example, the generating unit 132 executes the generation process based on various pieces of information acquired by the acquiring unit 131. The generating unit 132 executes the generation process based on various pieces of information stored in the storage unit 120. The generating unit 132 executes the generation process based on various pieces of information received from an external information processing device. The generating unit 132 executes the generation process based on a model learned by the learning unit 133. The generating unit 132 executes the generation process based on information determined by the processing unit 134.

[0112] The generating unit 132 generates a flattened image in which the distribution of a predetermined type of color attribute information is flattened from a learning image that is an image of a predetermined object and in which multiple types of color attribute information are associated with each pixel. The generating unit 132 generates, as the flattened image, a grayscale image in which the distribution of brightness of the multiple types of color attribute information is flattened.

[0113] (Learning Section 133) The learning unit 133 learns various information. The learning unit 133 generates various models by a learning process. For example, the learning unit 133 performs a learning process based on information stored in the learning data storage unit 121, etc. The learning unit 133 generates various models based on information stored in the learning data storage unit 121, etc. For example, the learning unit 133 learns a model M1 using the learning data stored in the learning data storage unit 121.

[0114] The learning unit 133 learns a diffusion model that converts one noise image, which is an image containing noise, so as to approach the learning image, based on the learning grayscale image and label acquired by the acquisition unit 131. The learning unit 133 learns a diffusion model that converts one noise image, which is an image containing noise, so as to approach the learning image, based on the flattened image generated by the generation unit 132 and a label indicating a feature of a predetermined target in the learning image.

[0115] The learning unit 133 learns a diffusion model that converts one noise image so as to approach a learning image including a predetermined object of a color indicated by a label. The learning unit 133 learns a diffusion model that converts one noise image acquired by the acquisition unit 131 so as to approach the learning image.

[0116] The learning unit 133 learns a diffusion model using the multiple stage images acquired by the acquisition unit 131. The learning unit 133 learns the diffusion model so as to generate a stage image that is an image in which noise has been removed by one stage from one of the multiple stage images. The learning unit 133 learns the diffusion model so as to generate a first stage image in which noise has been removed by one stage from a second stage image. The learning unit 133 learns the diffusion model so as to generate a second stage image in which noise has been removed by one stage from a third stage image.

[0117] For example, the learning unit 133 executes a learning process based on various information acquired by the acquisition unit 131. For example, the learning unit 133 learns a model based on information generated by the generation unit 132. The learning unit 133 executes a learning process based on information from an external information processing device and information stored in the storage unit 120. The learning unit 133 executes a learning process based on information stored in the model information storage unit 122. The learning unit 133 stores the model generated by learning in the model information storage unit 122.

[0118] The learning unit 133 performs a learning process. The learning unit 133 performs various learning. The learning unit 133 learns various information based on the information acquired by the acquisition unit 131. The learning unit 133 learns (generates) a model. The learning unit 133 learns various information such as a model. The learning unit 133 generates a model by learning. The learning unit 133 learns the model using various machine learning techniques. For example, the learning unit 133 learns parameters of the model (network). The learning unit 133 learns the model using various machine learning techniques.

[0119] The learning unit 133 generates various learning models such as the model M1. The learning unit 133 learns network parameters. For example, the learning unit 133 learns network parameters of various learning models such as the model M1. The learning unit 133 performs a learning process using learning data stored in the model information storage unit 122, thereby generating various learning models such as the model M1. For example, the learning unit 133 generates a model used for generating an image. The learning unit 133 generates various learning models such as the model M1 by learning network parameters of various learning models such as the model M1.

[0120] The learning unit 133 performs a learning process based on the learning data (teacher data) stored in the model information storage unit 122. The learning unit 133 performs a learning process using the learning data stored in the model information storage unit 122, thereby generating various learning models such as model M1.

[0121] For example, the learning unit 133 performs a learning process using a method such as backpropagation so that an image output by the model M1 approaches correct answer information (information indicating a desired generation result) associated with input information input to the model M1. For example, the learning unit 133 performs a learning process so that an image output by the model M1 to which input information including a noise image is input approaches an image (correct answer information) in which noise has been removed from the input noise image by one step.

[0122] For example, the learning unit 133 adjusts the value of the weight (i.e., the connection coefficient) that is taken into consideration when a value is transmitted between nodes by the learning process. In this way, the learning unit 133 learns the model M1 by a process such as backpropagation that corrects the parameters (connection coefficients) so as to reduce the error between the output in the model M1 and the correct answer information corresponding to the input. For example, the learning unit 133 generates the model M1 by a process such as backpropagation that minimizes a predetermined loss function. This allows the learning unit 133 to perform a learning process that learns the parameters of the model M1.

[0123] The learning method of the model is not limited to the above-mentioned method, and any known technology can be applied. The generation of each model may be performed by appropriately using various conventional technologies related to machine learning. For example, the generation of the model may be performed by using a technology related to machine learning of supervised learning, such as SVM (Support Vector Machine). Also, for example, the generation of the model may be performed by using a technology related to machine learning of unsupervised learning. For example, the generation of the model may be performed by using a deep learning technology. For example, the generation of the model may be performed by appropriately using various deep learning technologies such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network). Note that the description regarding the generation of the above model is an example, and the generation of the model may be performed by a learning method appropriately selected according to the obtainable information, etc. That is, the learning unit 133 may generate the model M1 by any method as long as the model M1 can be learned so as to output information corresponding to the correct answer information when the input information included in the learning data is input.

[0124] As described above, the method of learning by the learning unit 133 is not particularly limited, but for example, learning data in which data (input information) and its correct answer information (output information) are linked may be prepared, and the learning data may be input to a computation model based on a multi-layer neural network for learning. In addition, a method based on DNN such as CNN or 3D-CNN may be used. Specifically, the learning method by the learning unit 133 may be a method based on U-net. When targeting time-series data such as voice, the learning unit 133 may use a method based on a recurrent neural network (RNN) or LSTM (Long Short-Term Memory units), which is an extension of RNN.

[0125] (Processing section 134) The processing unit 134 processes various types of information. The processing unit 134 performs information processing based on the information stored in the storage unit 120. The processing unit 134 functions as an inference unit that performs inference processing. The processing unit 134 performs inference processing by using a model stored in the model information storage unit 122. For example, the learning unit 133 performs inference processing based on the information generated by the generation unit 132.

[0126] The processing unit 134 performs inference using the model M1 learned by the learning unit 133. The processing unit 134 performs inference using the model M1, which is a diffusion model. The processing unit 134 performs inference processing to generate an image in which a color corresponding to the label is applied to an object in the grayscale image by inputting a noise image, a grayscale image, and a label as input information to the model M1. The processing unit 134 uses the diffusion model learned by the learning unit 133 to generate a converted image.

[0127] (Provider 135) The providing unit 135 provides various information. For example, the providing unit 135 provides various information to the terminal device 10 and the information providing device 50. The providing unit 135 transmits various information to the terminal device 10 and the information providing device 50. The providing unit 135 provides various information based on the various information acquired by the acquiring unit 131. The providing unit 135 provides various information based on the various information generated by the generating unit 132.

[0128] The providing unit 135 provides various information based on the various information learned by the learning unit 133. The providing unit 135 provides various information based on the model learned by the learning unit 133. The providing unit 135 provides various information based on the various information processed by the processing unit 134. For example, the providing unit 135 transmits information to the terminal device 10.

[0129] The provision unit 135 provides a service using the diffusion model learned by the learning unit 133. The provision unit 135 provides a service using the converted image generated by the diffusion model to a user. The provision unit 135 provides an image search service using the converted image to a user.

[0130] The providing unit 135 transmits a result of the image search using the converted image to the terminal device 10 used by the user. The providing unit 135 transmits a similar image similar to the converted image to the terminal device 10 used by the user. The providing unit 135 provides the user with a service using the converted image generated using a grayscale image of the target image and a diffusion model. The providing unit 135 provides the user with a service using the converted image generated using the designation information.

[0131] [4. Information processing flow] Next, a flow of information processing by the information processing device 100 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are flowcharts showing an example of information processing according to an embodiment.

[0132] First, a processing example shown in Fig. 6 will be described. As shown in Fig. 6, the information processing device 100 acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a predetermined object captured, and a label indicating a feature of the predetermined object in the learning image (step S101).

[0133] The information processing device 100 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approach the learning image, based on the learning grayscale image and the label (step S102).

[0134] Next, a processing example shown in Fig. 7 will be described. As shown in Fig. 7, the information processing device 100 generates a flattened image in which the distribution of a predetermined type of color attribute information is flattened from a learning image, which is an image of a predetermined target and in which a plurality of types of color attribute information are associated with each pixel (step S201).

[0135] The information processing device 100 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approximate the training image based on the flattened image and a label indicating a feature of a specific target in the training image (step S202).

[0136] 5. Effects As described above, the information processing device 100 according to the embodiment includes the acquisition unit 131 and the learning unit 133. The acquisition unit 131 acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a predetermined target, and a label indicating a feature of the predetermined target in the learning image. The learning unit 133 learns a diffusion model that converts a noise image, which is an image containing noise, so as to approximate the learning image, based on the learning grayscale image and the label acquired by the acquisition unit 131.

[0137] In this way, the information processing device 100 according to the embodiment can learn a model that converts the features of a predetermined object in an image by learning a diffusion model that converts a noise image, which is an image containing noise, to approach the learning image based on a learning grayscale image obtained by grayscaling a learning image in which a predetermined object is captured, and a label indicating the features of the predetermined object in the learning image. Therefore, the information processing device 100 can generate an appropriate image.

[0138] Moreover, in the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a label indicating a feature including at least one of the color, shape, and atmosphere of a predetermined object in a learning image.

[0139] In this way, the information processing device 100 according to the embodiment can learn a model that converts the color, shape, atmosphere, etc. of a predetermined object in an image by acquiring a label indicating a feature including at least one of the color, shape, and atmosphere of the predetermined object in the learning image. Therefore, the information processing device 100 can enable appropriate image generation.

[0140] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a label indicating a color of a predetermined object in a learning image. The learning unit 133 learns a diffusion model that converts a noise image so as to approximate the learning image including the predetermined object having the color indicated by the label.

[0141] In this way, the information processing device 100 according to the embodiment can learn a model for converting the color of a predetermined object in an image by acquiring a label indicating the color of the predetermined object in the learning image. Therefore, the information processing device 100 can generate an appropriate image.

[0142] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires one noise image. The learning unit 133 learns a diffusion model that converts the one noise image acquired by the acquisition unit 131 so as to approximate the learning image.

[0143] In this way, the information processing device 100 according to the embodiment can appropriately learn the model by acquiring one noise image and learning the diffusion model using the acquired one noise image. Therefore, the information processing device 100 can generate an appropriate image.

[0144] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of stage images in which noise is added stepwise to the learning image. The learning unit 133 uses the plurality of stage images acquired by the acquisition unit 131 to learn a diffusion model.

[0145] In this way, the information processing device 100 according to the embodiment can appropriately learn the diffusion model by using a plurality of step images in which noise is gradually added to the learning image. Therefore, the information processing device 100 can generate an appropriate image.

[0146] Moreover, in the information processing device 100 according to the embodiment, the learning unit 133 learns the diffusion model so as to generate a stage image that is an image in which noise is removed by one stage from one of the plurality of stage images.

[0147] In this way, the information processing device 100 according to the embodiment can appropriately learn the model by learning the diffusion model so as to generate a stage image in which noise is removed by one stage from one of a plurality of stage images. Therefore, the information processing device 100 can appropriately generate an image.

[0148] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of stage images including a first stage image and a second stage image in which one stage noise is added to the first stage image. The learning unit 133 learns a diffusion model so as to generate a first stage image in which one stage of noise is removed from the second stage image.

[0149] In this way, the information processing device 100 according to the embodiment can appropriately learn the model by learning the diffusion model so as to generate a first-stage image in which noise is removed by one step from a second-stage image in which one-step noise is added to the first-stage image. Therefore, the information processing device 100 can appropriately generate images.

[0150] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a plurality of stage images including a third stage image in which one stage noise is added to the second stage image. The learning unit 133 learns a diffusion model so as to generate a second stage image in which one stage of noise is removed from the third stage image.

[0151] In this way, the information processing device 100 according to the embodiment can appropriately learn the model by learning the diffusion model so as to generate a second stage image in which one stage of noise is removed from a third stage image in which one stage of noise is added to the second stage image. Therefore, the information processing device 100 can appropriately generate images.

[0152] Moreover, the information processing device 100 according to the embodiment includes a providing unit 135. The providing unit 135 provides a service using the diffusion model trained by the training unit 133.

[0153] In this way, the information processing device 100 according to the embodiment can provide an appropriate service by providing the service using the diffusion model.

[0154] Moreover, in the information processing device 100 according to the embodiment, the providing unit 135 provides a service to a user using a converted image generated by a diffusion model.

[0155] In this manner, the information processing device 100 according to the embodiment can provide the user with an appropriate service by providing the user with a service using a converted image generated by a diffusion model.

[0156] Moreover, in the information processing device 100 according to the embodiment, the providing unit 135 provides the user with an image search service using the converted image.

[0157] In this manner, the information processing device 100 according to the embodiment can provide the user with an appropriate search service by providing the user with an image search service using the converted image.

[0158] Moreover, in the information processing device 100 according to the embodiment, the providing unit 135 transmits the result of the image search using the converted image to the terminal device 10 used by the user.

[0159] In this manner, the information processing device 100 according to the embodiment can provide the user with appropriate search results by transmitting the results of the image search using the converted image to the terminal device 10 used by the user.

[0160] Moreover, in the information processing device 100 according to the embodiment, the providing unit 135 transmits a similar image that is similar to the converted image to the terminal device 10 used by the user.

[0161] In this manner, the information processing device 100 according to the embodiment can provide the user with appropriate search results for similar images by transmitting similar images similar to the converted image to the terminal device 10 used by the user.

[0162] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires a target image to be converted from a user. The provision unit 135 provides the user with a service using a converted image generated using a grayscale image of the target image and a diffusion model.

[0163] In this manner, the information processing device 100 according to the embodiment can provide the user with an appropriate service by acquiring a target image to be converted from the user and providing the user with a service using a converted image generated using a grayscale image of the acquired target image and a diffusion model.

[0164] In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires designation information for designating a feature from a user. The provision unit 135 provides the user with a service using a converted image generated using the designation information.

[0165] In this way, the information processing device 100 according to the embodiment can provide the user with an appropriate service by acquiring specification information that specifies characteristics from the user and providing the user with a service that uses a converted image generated using the acquired specification information.

[0166] Moreover, the information processing device 100 according to the embodiment includes a processing unit 134. The processing unit 134 uses the diffusion model trained by the training unit 133 to generate a converted image.

[0167] In this way, the information processing device 100 according to the embodiment can perform appropriate image generation by generating a transformed image using a diffusion model.

[0168] [6. Hardware Configuration] Moreover, the information processing device 100 and the terminal device 10 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in Fig. 8, for example. The information processing device 100 will be described below as an example. Fig. 8 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected by a bus 1090.

[0169] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0170] The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB memory or a Secure Digital (SD) memory card. The secondary storage device 1050 may be a cloud storage device (online storage device), a NAS (Network Attached Storage), a file server, or the like.

[0171] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0172] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0173] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device, such as an input / output I / F.

[0174] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0175] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0176] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0177] For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with the other device via the network I / F 1080 and use functions, data, etc. of a program by calling them from another program of the other device.

[0178] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0179] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0180] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0181] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, an acquisition section can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0182] 1. Information Processing Systems 100 Information processing device 121 Learning data storage unit 122 Model information storage unit 130 Control section 131 Acquisition Department 132 Generation part 133 Learning Department 134 Processing section 135 Provision Department 10 Terminal Equipment 50 Information provision device N Network

Claims

1. an acquisition unit that acquires a learning grayscale image obtained by grayscaling a learning image, which is an image of a predetermined object captured, and a label that indicates a feature of the predetermined object in the learning image; a learning unit that learns a diffusion model that converts a noise image, which is an image including noise, so as to approximate the learning image based on the learning grayscale image and the label acquired by the acquisition unit; a provision unit that provides a user with an image search service using a transformed image generated by the diffusion model trained by the learning unit; An information processing device comprising:

2. The acquisition unit is The label indicating the feature including at least one of the color, shape, and atmosphere of the predetermined object in the learning image is obtained.

2. The information processing apparatus according to claim 1,

3. The acquisition unit is obtaining the label indicating the color of the predetermined object in the learning image; The learning unit is The diffusion model is trained to convert the one noise image so as to approximate the training image including the predetermined object of the color indicated by the label.

2. The information processing apparatus according to claim 1,

4. The acquisition unit is acquiring the one noise image; The learning unit is The diffusion model is trained to convert the one noise image acquired by the acquisition unit so as to approach the learning image.

2. The information processing apparatus according to claim 1,

5. The acquisition unit is Obtaining a plurality of step images by adding noise to the learning image in a stepwise manner; The learning unit is The diffusion model is trained using the plurality of stage images acquired by the acquisition unit.

5. The information processing apparatus according to claim 4.

6. The learning unit is The diffusion model is trained to generate a step image, which is an image obtained by removing noise from one step image of the plurality of step images by one step.

6. The information processing apparatus according to claim 5,

7. The acquisition unit is acquiring the plurality of stage images including a first stage image and a second stage image obtained by adding one stage noise to the first stage image; The learning unit is The diffusion model is trained to generate the first stage image by removing noise from the second stage image by one step.

6. The information processing apparatus according to claim 5,

8. The acquisition unit is acquiring the plurality of stage images including a third stage image in which one stage noise is added to the second stage image; The learning unit is The diffusion model is trained to generate the second stage image by removing noise from the third stage image by one step.

8. The information processing apparatus according to claim 7,

9. The providing unit is A result of an image search using the converted image is transmitted to a terminal device used by the user.

2. The information processing apparatus according to claim 1,

10. The providing unit is A similar image similar to the converted image is transmitted to a terminal device used by the user.

2. The information processing apparatus according to claim 1,

11. The acquisition unit is Obtaining a target image to be converted from the user; The providing unit is A service using the converted image generated using the grayscale image of the target image and the diffusion model is provided to the user.

2. The information processing apparatus according to claim 1,

12. The acquisition unit is acquiring specification information specifying a characteristic from the user; The providing unit is A service using the converted image generated using the specified information is provided to the user.

2. The information processing apparatus according to claim 1,

13. a processing unit that generates a transformed image by using the diffusion model trained by the training unit; The information processing apparatus according to claim 1 , further comprising:

14. 1. A computer-implemented information processing method, comprising: an acquisition step of acquiring a learning grayscale image in which a learning image, which is an image of a predetermined object, is grayscaled, and acquiring a label indicating a feature of the predetermined object in the learning image; a learning process for learning a diffusion model that converts a noise image, which is an image including noise, so as to approximate the learning image based on the learning grayscale image acquired by the acquisition process and the label; a providing step of providing an image search service to a user using a transformed image generated by the diffusion model trained in the training step; 13. An information processing method comprising:

15. An acquisition step of acquiring a learning grayscale image in which a learning image, which is an image of a predetermined object, is grayscaled, and acquiring a label indicating a feature of the predetermined object in the learning image; a learning step of learning a diffusion model that converts a noise image, which is an image containing noise, so as to approximate the learning image based on the learning grayscale image acquired by the acquisition step and the label; a providing step of providing a user with an image search service using a transformed image generated by the diffusion model trained by the learning step; An information processing program characterized by causing a computer to execute the above.

Citation Information

Patent Citations

  • Railway wagon manual brake shaft chain falling fault image recognition method

    CN111080605A

  • Road protection intelligent early warning method and system combined with AI vision algorithm

    CN113239960A

  • Image generation system training method, image generation method, and image generation system

    CN115170825A

  • Image processing apparatus and program

    JP2021149825A