Image processing device, image processing method, and program

Abstract

The present disclosure relates to an image processing device, an image processing method, and a program that achieve a generation model suitable for camera signal processing. This image processing device comprises: a VAE encoder for converting an input image into a feature amount of a latent space; a UNet including a plurality of levels, each including a residual coupling block and an attention mechanism block; a Gaussian diffusion unit for generating a plurality of pieces of control information on the basis of an input value and outputting the control information to one or more levels; and a frequency decomposition unit for frequency-decomposing a plurality of feature amounts, which are obtained according to the plurality of pieces of control information input to the levels, into a low frequency component and a high frequency component. The low frequency component is output to the subsequent level, the high frequency component is output to the level that is skip-connected to the level, the Gaussian diffusion unit generates Gaussian diffusion noise corresponding to the plurality of pieces of control information, and the UNet performs machine learning for denoising the Gaussian diffusion noise during machine learning. The technology of the present disclosure can be applied to, for example, a camera image captured by a camera.

Description

Image processing device, image processing method, and program

[0001] The present disclosure relates to an image processing device, an image processing method, and a program, and more particularly to an image processing device, an image processing method, and a program that enable realization of a generative model suitable for camera signal processing.

[0002] Recently, "diffusion technology," which generates images by machine learning Gaussian dediffusion processing, a denoising process, from an image to which Gaussian diffusion noise has been added, and then repeatedly executing this Gaussian dediffusion processing, has rapidly become popular. This diffusion technology has extremely high image generation capabilities, even compared to other generative AI technologies such as GAN (generative adversarial network) technology and VAE (variational autoencoder) technology. A foundation model using diffusion technology in the form of multimodal generative AI that combines text and music has been developed, and various methods have been proposed around the world.

[0003] An example of technology related to generative AI is Patent Document 1, which proposes super-resolution technology using UNet. UNet is an effective method for segmentation analysis and depth analysis, and is used in image processing AI as a method for high-speed processing from high to low frequencies, along with variational autoencoders (VAE) and vector quantization variational autoencoders (VQ-VAE).

[0004] US Patent Application Publication No. 2023 / 0267652

[0005] Diffusion-based generative AI technology has higher image generation capabilities than VAE and GAN technologies, but when considering application to camera signal processing, it has been confirmed that there are problems such as significantly altering the information in the captured image and generating fake objects.

[0006] The present disclosure has been made in light of these circumstances, and makes it possible to realize a generative model suitable for camera signal processing.

[0007] An image processing device according to one aspect of the present disclosure includes: a variational autoencoder that converts an input image into features in a latent space; a layered network consisting of a plurality of layers, each layer including a residual connection block and an attention mechanism block; a control information generation unit that generates a plurality of control information based on an input value and outputs the control information to one or more of the layers; and a frequency decomposition unit that frequency-decomposes a plurality of features obtained according to the plurality of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer skip-connected to the layer; the control information generation unit generates Gaussian diffusion noise corresponding to the plurality of control information; and the layered network performs machine learning to denoise the Gaussian diffusion noise during machine learning.

[0008] An image processing method according to one aspect of the present disclosure includes: a variational autoencoder converting an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual combination block and an attention mechanism block, performing machine learning to denoise Gaussian diffusion noise during machine learning; a control information generation unit generating Gaussian diffusion noise corresponding to multiple pieces of control information based on an input value and outputting the Gaussian diffusion noise to one or more of the layers; and a frequency decomposition unit frequency-decomposing the multiple features obtained in accordance with the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer connected to the layer by a skip connection.

[0009] A program according to one aspect of the present disclosure causes a computer to function as: a variational autoencoder that converts an input image into features in a latent space; a layered network consisting of a plurality of layers, each layer including a residual connection block and an attention mechanism block; a control information generation unit that generates a plurality of control information based on an input value and outputs the control information to one or more of the layers; and a frequency decomposition unit that frequency-decomposes a plurality of features obtained in accordance with the plurality of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer skip-connected to the layer; the control information generation unit generates Gaussian diffusion noise corresponding to the plurality of control information; and the layered network performs machine learning to denoise the Gaussian diffusion noise during machine learning.

[0010] In one aspect of the present disclosure, an input image is converted into features in a latent space, each layer includes a residual combination block and an attention mechanism block, and machine learning is performed in a layered network consisting of multiple layers to denoise Gaussian diffusion noise during machine learning, Gaussian diffusion noise corresponding to multiple pieces of control information is generated based on input values ​​and output to one or more of the layers, multiple features obtained in accordance with the multiple pieces of control information input to the layers are frequency-decomposed into low-frequency components and high-frequency components, the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer connected to the layer by a skip connection.

[0011] The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0012] The image processing device may be an independent device or an internal block constituting a single device.

[0013] 5 is a diagram showing an example of an image generated by a generative model using a diffusion-type image generation AI technology. FIG. 6 is a diagram showing an example of an image generated by a generative model using the image generation AI technology of the present disclosure. FIG. 7 is a block diagram showing an example of a configuration of a surface noise generation model. FIG. 8 is a block diagram showing an example of a configuration of a phase noise generation model of the present disclosure. FIG. 9 is a diagram showing an overview of control information association learning. FIG. 10 is a block diagram showing a detailed configuration example of the hierarchy of the encoder unit of the UNet of FIG. 4. FIG. 11 is a flowchart showing the learning process of the phase noise generation model of FIG. 4. FIG. 12 is a flowchart showing the inference process of the phase noise generation model of FIG. 13. FIG. 14 is a block diagram showing another configuration of the residual combination block and the attention mechanism block. FIG. 15 is a block diagram showing another example of a configuration of the phase noise generation model of the present disclosure. FIG. 16 is a diagram showing a first example of control information association learning. FIG. 17 is a diagram showing a second example of control information association learning. FIG. 18 is a diagram showing a third example of control information association learning. FIG. 19 is a diagram showing the learning process of the surface noise generation model. FIG. 20 is a diagram showing the learning process of the phase noise generation model. FIG. 21 is a diagram showing a simplified representation of the phase noise generation model of FIG. 4. FIG. 22 is a block diagram showing a first application example to which the phase noise generation model is applied. FIG. 23 is a block diagram showing a second application example to which the phase noise generation model is applied. FIG. 24 is a block diagram showing a third application example to which the phase noise generation model is applied. FIG. 25 is a diagram showing a template matching unit. It is a block diagram showing a fourth application example to which a phase noise generation model is applied.It is a block diagram showing a fifth application example to which a phase noise generation model is applied.It is a block diagram showing a hardware configuration example of an information processing device of the present disclosure.

[0014] Hereinafter, with reference to the accompanying drawings, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment). Note that in this specification and the drawings, components having substantially the same functional configuration are assigned the same reference numerals, and redundant description will be omitted. The description will be given in the following order: 1. Overview of the image generation AI technology of the present disclosure 2. Configuration example of a surface noise generation model 3. Configuration example of a phase noise generation model 4. Learning process and inference process of a phase noise generation model 5. Frequency decomposition after attention mechanism block processing 6. Example of control information association learning 7. Comparison of learning progress 8. Simplified representation of a phase noise generation model 9. Application example to a metadata generation model 10. Application example as a control net 11. Application example to template matching 12. Application example to a multimodal net 13. Application example to an image generation net using both surface and phase 14. Summary 15. Computer configuration example

[0015] 1. Overview of the Image Generation AI Technology of the Present Disclosure First, the image generation AI technology of the present disclosure will be described while comparing it with a diffusion-type image generation AI technology.

[0016] Figure 1 shows an example of an image generated by a generative model (machine learning model) using diffusion-based image generation AI technology.

[0017] The diffusion method generates images by machine learning a Gaussian de-diffusion process, a de-noising step, from an image to which Gaussian diffusion noise has been added, and then repeatedly executing this Gaussian de-diffusion process. Figure 1 shows an example of six images generated using the DDPM (denoising diffusion probabilistic model), a representative diffusion model. As shown in Figure 1, the diffusion generative model acquires the ability to generate images by adding noise to the image surface, i.e., the image region, and then performing machine learning to de-noise it, thereby enabling it to randomly generate images from the noise. By providing the generative model with a text prompt input interface, images related to the input text are generated.

[0018] FIG. 2 shows an example of an image generated by a generative model (machine learning model) using the image generation AI technology of the present disclosure.

[0019] The generative model disclosed herein (hereinafter referred to as the present generative model) generates images as inter-image information by adding noise to phase information connecting multiple images and then performing machine learning to denoise the noise. For example, as shown in FIG. 2, a generative model that has acquired the ability to generate images based on phase information connecting an image at time T and an image at time T+1 can generate an image at 8.3 msec between the image at time T=0 and the image at time T+1=16.6 msec in a 60 fps video. Phase information is information that continuously captures the correlation between two deterministically existing images and shifts the position of the correlation between the two images. For example, if the two images are an image with a front focus position (front focus) and an image with a back focus position (back focus), the phase information is intermediate focus information, and the generative model can generate an intermediate focus image. For example, if the two images are a multi-camera image captured from a first viewpoint among multi-camera images captured from multiple viewpoints, and a multi-camera image captured from a second viewpoint different from the first viewpoint, the phase information is viewpoint information representing an arbitrary viewpoint connecting the multiple viewpoints, and the generative model can generate a multi-camera image of an arbitrary viewpoint connecting the multiple viewpoints.

[0020] When a diffusion-based generative model is trained to correct image blur, blur is dramatically reduced compared to VAE and GAN, but objects such as grass or window frames may be replaced with other objects. Furthermore, white balance, gamma characteristics, and noise characteristics are also significantly altered, making it extremely difficult to apply this technology to camera signal processing, especially as a video technology for film production and professional broadcasting.

[0021] In contrast, this generative model, which acquires the ability to generate images from phase information, takes the existence of two images, one before the other, for granted, and generates information to connect them. Since the existence of captured images is self-evident in camera signal processing, image generation in the phase direction using this generative model tends to faithfully generate intermediate information based on the image captured by the camera, making it a highly compatible approach for image generation in camera signal processing. Image generation from noise on a diffusion-based surface is a concept that contradicts the approach of generating images based on self-evident images, and is therefore not compatible with camera signal processing.

[0022] The diffusion-based generative model assumes image generation from noise and performs image generation processing using an image-to-image method in which a noise removal rate is set and an image is generated. In contrast, this generative model uses multiple obvious camera images, does not add noise to the camera images, and generates phase information from the noise between them. In the following description, the image generation method for phase information used by this generative model is referred to as Phase Shift Denoising Diffusion, and a generative model that performs Phase Shift Denoising Diffusion is referred to as a phase noise generative model. Furthermore, the image generation method for surfaces is referred to as Denoising Diffusion, and a generative model that performs Denoising Diffusion is referred to as a surface noise generative model.

[0023] 2. Example of the Configuration of a Surface Noise Generation Model FIG. 3 is a block diagram showing an example of the configuration of a surface noise generation model that executes denoising diffusion.

[0024] The surface noise generation model 10 shown in FIG. 3 includes a Gaussian diffusion unit 21, a VAE encoder (VAE ENC) 22, a UNet 23, a VAE decoder (VAE DEC) 24, and encoders 25 and 26.

[0025] The Gaussian diffusion unit 21 generates an image (hereinafter referred to as a noise image) by adding noise according to a Gaussian distribution (Gaussian diffusion noise) to the input image, and supplies the image to the VAE encoder 22 .

[0026] The VAE encoder 22 is a variational autoencoder that converts noise images in pixel space into features in latent space. The latent space is a feature space that learns the features of the data (e.g., images) to be trained and simplifies the data representation. The VAE encoder 22 may be a vector quantization variational autoencoder (VQ-VAE) equipped with a vector quantization unit that vector quantizes the features in the latent space converted by the variational autoencoder. The vector quantization variational autoencoder vector quantizes the features in the obtained latent space, thereby improving the image generation capability.

[0027] The UNet 23 has a multi-layered network structure including a residual connection block (ResBlock) 31 and an attention block (AttnBlock) 32. The first half of the UNet 23, the residual connection block 31E and the attention block 32E, are encoder layers that convert from high resolution to low resolution, while the second half of the UNet 23, the residual connection block 31D and the attention block 32D, are decoder layers that convert from low resolution to high resolution. The attention block 32E has a skip connection that connects it to the attention block 32E of the same resolution on the decoder side.

[0028] The residual combination block 31 includes a convolutional layer and is configured to add the input to the residual combination block 31 to the output from the convolutional layer. Time information converted into a vector by the encoder 25 is supplied to the residual combination block 31 and embedded in the features within the residual combination block 31. The attention mechanism block 32 includes an attention layer and adds weights according to the importance of the information. Metaprompt information converted into a vector by the encoder 26 is converted into a query (Q), key (K), and value (V) and supplied to the attention mechanism block 32, which are embedded in the features within the attention mechanism block 32.

[0029] The VAE decoder 24 is a decoder corresponding to the VAE encoder 22, converts the feature quantity in the latent space output from the UNet 23 into the pixel space, and outputs a generated image after the conversion. When the VAE encoder 22 is configured as a vector quantization variational autoencoder (VQ-VAE), the VAE decoder 24 is also a decoder corresponding to the vector quantization variational autoencoder (VQ-VAE).

[0030] The encoder 25 converts the input time information into a vector and outputs it to the residual combination block 31 of each layer of the UNet 23. The encoder 26 converts the input metaprompt information into a vector of query, key, and value and outputs it to the attention mechanism block 32 of each layer of the UNet 23. The metaprompt information is composed of, for example, text information or music information.

[0031] The surface noise generation model 10 configured as described above acquires image generation capabilities by performing machine learning to denoise noise images to which noise has been added while associating them with text information and music information input as meta-prompt information.

[0032] 3. Example of the Configuration of a Phase Noise Generation Model FIG. 4 is a block diagram showing an example of the configuration of a phase noise generation model that executes Phase Shift Denoising Diffusion.

[0033] The phase noise generation model 50 shown in Fig. 4 has a VAE encoder (VAE ENC) 22, a VAE decoder (VAE DEC) 24, encoders 25 and 26, a Gaussian diffusion unit 61, and a UNet 62. In the phase noise generation model 50, the VAE encoder (VAE ENC) 22, the VAE decoder (VAE DEC) 24, and the encoders 25 and 26 have the same configurations as those in the surface noise generation model 10 shown in Fig. 3, and therefore their description will be omitted where appropriate. In the phase noise generation model 50, the Gaussian diffusion unit 21 of the surface noise generation model 10 shown in Fig. 3 is changed to a Gaussian diffusion unit 61, and the UNet 23 is changed to a UNet 62.

[0034] The Gaussian diffusion unit 61 generates multiple pieces of control information based on input values ​​received from an input unit (not shown), generates noise corresponding to the multiple pieces of control information, and outputs the generated noise to each residual combination block 71E in the first half of the encoder unit of the UNet 62. The multiple pieces of control information are generated from the input values ​​and control the phase of the image. The input value is a reference value for generating the control information, and in this embodiment, is set to, for example, 0.5. The input value is a hyperparameter in machine learning, and the optimal value is found by adjustment. The Gaussian diffusion unit 61 generates a random value Δ that follows a Gaussian distribution, and generates two pieces of control information (0.5 + Δ) and (0.5 - Δ) using the input value and the random value Δ. Here, the random value Δ follows a Gaussian distribution with a mean of 0.5, and the control information (0.5 + Δ) and (0.5 - Δ) are random values ​​centered around the control information 1 and 0. The Gaussian diffusion unit 61 generates Gaussian diffusion noise corresponding to a plurality of pieces of control information (0.5±Δ) and outputs it to each residual combination block 71E.

[0035] The UNet 62 is configured with a multi-layered network structure including a residual connection block (ResBlock) 71 and an attention mechanism block (AttnBlock) 72. The first half of the UNet 62, the residual connection block 71E and the attention mechanism block 72E, are encoder layers that convert from high resolution to low resolution, while the second half of the UNet 62, the residual connection block 71D and the attention mechanism block 72D, are decoder layers that convert from low resolution to high resolution. The residual connection block 71E has a skip connection that connects it to the residual connection block 71D of the same resolution on the decoder side.

[0036] A value (input value) that is the basis for controlling the phase is input to the phase noise generation model 50, and is supplied to a Gaussian diffusion unit 61. The Gaussian diffusion unit 61 generates multiple pieces of control information from the input value, and outputs noise corresponding to the multiple pieces of control information to each residual combination block 71E of the first half encoder unit of the UNet 62.

[0037] In addition, the image input to the phase noise generation model 50 is converted into features in latent space by the VAE encoder 22. The time information is converted into a vector by the encoder 25 and then supplied to each residual combination block 71 for use as time embedding. The meta-prompt information is composed of, for example, camera metadata (gyro data, lens data, etc.), text, numerical values, and other input data, and is appropriately converted into latent space by the encoder 26. The information is then converted into queries, keys, and values ​​and supplied to each attention mechanism block 72.

[0038] The residual combination block 71 includes a convolutional layer and is configured to add the input to the residual combination block 71 to the output from the convolutional layer. Time information converted into a vector by the encoder 25 is embedded in the features in the residual combination block 71. The attention mechanism block 72 includes an attention layer and adds weights according to the importance of the information. Metaprompt information converted into a vector by the encoder 26 is converted into a query (Q), key (K), and value (V), which are embedded in the features in the attention mechanism block 72. Various attention layers have been proposed for the attention mechanism block 72, such as Cross-Attention, Self-Attention, Temporary-Attention, and Spartial (Space)-Attention, but this embodiment is not particularly limited and any configuration may be adopted.

[0039] The residual combination block 71E and attention mechanism block 72E of the encoder part of the UNet 62 convert the input latent space features into more abstract latent space features while downsizing to reduce computational costs. This residual combination block 71E is composed of a residual combination block (ResBlock PhaseShift) that uses a phase diffusion method different from the residual combination block 31 of the surface noise generation model 10. The residual combination block 71E frequency-decomposes the latent space features into high-frequency components H and low-frequency components L, thereby creating highly abstract latent space information related to the phase information of the image.

[0040] The residual connection block 71D and attention mechanism block 72D in the decoder section of the UNet 62 decode the features in the latent space decomposed by the encoder section and restore the image from the phase-diffused information. The residual connection block 71D does not perform frequency decomposition by phase diffusion. The residual connection block 71D and attention mechanism block 72D, which are the decoder layers, are similar to the residual connection block 31D and attention mechanism block 32D in the surface noise generation model 10.

[0041] The UNet 62 configured as described above performs machine learning by associating the feature quantities of the latent space corresponding to the two input images with a plurality of pieces of control information (0.5±Δ) as phase information.

[0042] An overview of control information association learning in which the phase noise generation model 50 associates and learns control information will be described with reference to FIG.

[0043] Two consecutive frames of images are input to the phase noise generation model 50 and converted into features in a latent space. Of the two consecutive frames, the image of the first frame is designated the previous frame, and the image of the second frame is designated the next frame. The UNet 62 performs machine learning while associating the control information with the input images, such that when the control information (phase information) is (0.5 - Δ) = 0.0, it outputs the previous frame, and when (0.5 + Δ) = 1.0, it outputs the next frame. By intentionally adding an appropriate level of noise corresponding to the control information during training, the model acquires image generation capabilities through the machine learning process of de-noising (de-noising) the control information, despite being disturbed by the noise. This control information association learning results in a phase shift in the output image in response to changes in the control information. When an intermediate value such as 0.5 is assigned as control information, an intermediate frame image can be output. If the two images input corresponding to control information (0.5 ± Δ) are multi-camera images from two different viewpoints, the output image generated in response to changes in the control information will be a multi-camera image from any viewpoint between the two viewpoints. For example, if the two images input corresponding to the control information (0.5±Δ) are an image at a front focus position (front focus) and an image at a rear focus position (rear focus), the output image generated in conjunction with the change in the control information will be an image with an intermediate focus.

[0044] FIG. 6 is a block diagram showing an example of a detailed configuration of a layer including a residual combination block 71E and an attention mechanism block 72E of the encoder section of the UNet 62.

[0045] In FIG. 6 , a frequency decomposition unit 73E is provided downstream of the residual combination block 71E of the encoder section of the UNet 62. The frequency decomposition unit 73E may be provided as part of the residual combination block 71E. The frequency decomposition unit 73E performs frequency decomposition, for example, into a low-frequency component L = (A + B) / 2 and a high-frequency component H = (AB) / 2 using the output values ​​A and B of the residual combination block 71E obtained in response to input control information (0.5 ± Δ). Here, the output value of the control information (0.5 + Δ) is A, and the output value of the control information (0.5 - Δ) is B. This frequency decomposition can be interpreted as performing a Haar Wavelet transform on the features generated within the UNet 62. Here, the value 0.5 corresponding to the input value is a hyperparameter in machine learning, and the optimal value is found by adjustment. The low-frequency component L = (A + B) / 2 is output to the immediately following attention mechanism block 72E. On the other hand, the high-frequency component H=(AB) / 2 is output to the Skip-connected residual combination block 71D of the same resolution on the decoder side. This simulates the concept of the frequency hierarchy of Wavelet transform on the UNet 62, and performs Latent-Wavelet transform on features at multiple resolutions / multiple levels of abstraction, which can be considered an AI extension of the Wavelet transform method.

[0046] The low-frequency component L = (A + B) / 2 and the high-frequency component H = (AB) / 2 generated by the frequency decomposition unit 73E of the encoder section of the UNet 62 can be output externally as intermediate products and used externally. Image generation using the phase noise generation model 50 involves machine learning of correlations between multiple images based on observational information, making it possible to generate equations based on physical characteristics (like a physics engine). The surface noise generation model 10 targets the generation of a variety of images, such as CG, illustrations, and animation, and therefore does not generally identify physical laws for the observational information. In contrast, the phase noise generation model 50 captures physical changes from camera observational information and learns their trends through machine learning. Therefore, the features created at each layer within the UNet 62 each have a physical meaning. Therefore, the frequency decomposition information of the features subjected to Latent-Wavelet transformation by the frequency decomposition unit 73E of the encoder section of the UNet 62 can be used as intermediate products for metadata generation, control networks, hybrid networks using both surface and phase, as described below.

[0047] Returning to FIG. 4 , the UNet 62 performs control information association learning, which associates and learns control information. It has a structure that repeatedly performs frequency decomposition (L=(A+B) / 2, H=(AB) / 2) of output values ​​A and B corresponding to multiple pieces of control information before being connected to the downstream stage. The frequency decomposition (L=(A+B) / 2, H=(AB) / 2) is a decomposition of latent space features related to image phase control using a method similar to Wavelet transform. By performing frequency decomposition in multiple stages in the UNet 62, the frequency decomposition is layered based on the same concept as the multi-layered Wavelet transform. The decomposed information is not simple frequency information, but rather the frequency decomposition of latent space features, with the level of abstraction increasing with each layer. The frequency decomposition method shown here is the simplest example of frequency decomposition, but this is intended to reduce computational costs. The filter coefficients are not limited to this method. Higher-level frequency decomposition is also possible as needed.

[0048] 4. Learning Process and Inference Process of Phase Noise Generation Model Next, the learning process of the phase noise generation model 50 will be described with reference to the flowchart of FIG.

[0049] First, in step S1, the phase noise generation model 50 initializes a machine learning environment, such as a neural network such as the UNet 62, dictionary data, a training data set, and optimization options. The optimization options include convergence conditions, error propagation settings, LOSS calculation settings, and the like.

[0050] The phase noise generation model 50 repeatedly executes the acquisition (update) of input data and target data in step S2, forward processing in step S3, loss calculation processing for calculating the error with the target data in step S4, and weight update by error backpropagation in step S5 until it is determined in step S6 that the convergence condition is satisfied. If it is determined in step S6 that the convergence condition is satisfied, the phase noise generation model 50 ends the repeated processing of steps S2 to S6, and saves the dictionary data in step S7.

[0051] Next, the inference process of the phase noise generation model 50 will be described with reference to the flowchart of FIG.

[0052] First, in step S11, the phase noise generation model 50 loads dictionary data obtained by the learning process and initializes networks such as the UNet 62. In the next step S12, the phase noise generation model 50 acquires two input images and calculates two pieces of control information using a random value Δ for an input intermediate value of 0.5. In step S13, the VAE encoder 22 converts the input images into features in a latent space.

[0053] Next, the process of the residual combination block 71E in step S14, the process of the frequency decomposition unit 73E in step S15, and the process of the attention mechanism block 72E in step S16 are executed in order at each layer of the encoder unit of the UNet 62. In the process of the frequency decomposition unit 73E, the low-frequency component L = (A + B) / 2 and the high-frequency component H = (AB) / 2 are generated using the output values ​​A and B of the residual combination block 71E obtained using two pieces of control information (0.5 ± Δ). The low-frequency component L = (A + B) / 2 is output to the immediately succeeding attention mechanism block 72E, and the high-frequency component H = (AB) / 2 is output to the residual combination block 71D of the same resolution, which is connected via Skip.

[0054] Next, the processing of the residual combination block 71D in step S17 and the processing of the attention mechanism block 72D in step S18 are executed in order at each layer of the decoder section of UNet 62, and then in step S19, the feature quantities in the latent space output from UNet 62 are converted into an image in pixel space and output, and the inference processing ends.

[0055] 5. Frequency Decomposition After Attention Mechanism Block Processing In the above example, the frequency decomposition unit 73E is provided downstream of the residual combination block 71E. However, as shown in FIG. 9, the frequency decomposition unit 73E may also be provided downstream of the attention mechanism block 72E. The frequency decomposition unit 73E may also be provided as part of the attention mechanism block 72E. The attention mechanism block 72E' in FIG. 9 represents the attention mechanism block 72E when the frequency decomposition unit 73E is provided downstream of it. The frequency decomposition unit 73E performs frequency decomposition into a low-frequency component L=(A+B) / 2 and a high-frequency component H=(AB) / 2 using the output values ​​A and B of the attention mechanism block 72E' obtained in response to input of control information (0.5±Δ). FIG. 10 is a block diagram showing the overall configuration of the phase noise generation model 50 when frequency decomposition is performed including the results of the attention mechanism block 72E'. The low-frequency component L=(A+B) / 2 is output to the immediately following residual combination block 71E. The high frequency component H=(AB) / 2 is output to a Skip-connected residual merging block 71D of the same resolution on the decoder side.

[0056] 6. Example of Control Information Association Learning FIG. 11 shows a first example of control information association learning.

[0057] When the phase noise generation model 50 is trained using two images input corresponding to control information (0.5±Δ), one with a focus position (in focus) on the foreground and the other with a focus position on the background, the phase noise generation model 50 generates and outputs an image with an intermediate focus that is linked to control information with an intermediate value between 0 and 1. The control information may be set to three by adding an intermediate value of 0.5 to (0.5±Δ), and training may be performed using three images with three focus positions. In this case as well, the phase noise generation model 50 is trained to perform arbitrary focus movement based on three images.

[0058] In this way, control information association learning enables control using phase information associated with optical information such as noise and flicker. Furthermore, not only optical information such as noise and flicker can be extracted, but also more abstract information, as shown in the example of FIG. 12, can be extracted using a wavelet transform-like technique. FIG. 12 shows a second example of control information association learning, in which the phase noise generation model 50 is trained using control information (0.5-Δ) as a winter landscape image and control information (0.5+Δ) as a spring landscape image. The trained phase noise generation model 50 can generate images of intermediate seasons depending on the control information values ​​between 0 and 1, making cherry blossoms bloom on dead trees. In this way, control information association learning enables control in the form of phase control for the abstract level of "seasonal scenery."

[0059] FIG. 13 shows a third example of control information association learning.

[0060] Figure 13 shows an example of control information association learning performed on two completely uncorrelated images. By inputting two images, an image of a cat and an image of a cactus, and performing learning by associating them with control information (0.5±Δ), it is possible to create a good intermediate contour, as shown in the output image in Figure 13. In this way, control information association learning has the ability to generate contours that connect frames. This effect can be used to realize a Control-Net, as described below.

[0061] As described above, the phase noise generation model 50 that performs control information association learning has the ability to control images ranging from relatively low-level contour information to highly abstract concepts such as seasonal scenery. The phase noise generation model 50 also has the ability to control contours even between uncorrelated images, enabling diverse control information association learning.

[0062] 7. Comparison of Learning Processes The learning processes of the surface noise generation model 10 that performs denoising diffusion and the phase noise generation model 50 that performs phase shift denoising diffusion will be compared and described with reference to FIGS. 13 and 14 .

[0063] FIG. 14 shows an example of an image showing the learning process of the surface noise generation model 10.

[0064] The surface noise generation model 10 requires a considerable computational cost before it can generate a satisfactory image. A typical DDPM method requires repeating the despreading process 1,000 times, resulting in a very heavy computational cost. In the inventor's development environment, even after running the despreading process for a week on a data set of 200,000 sets, the resolution shown in FIG. 14 was achieved after 11 epochs, which confirmed that the resulting image was not faithful to the captured image.

[0065] FIG. 15 shows an example of an image showing the learning process of the phase noise generation model 50.

[0066] In contrast, the phase noise generation model 50 can basically generate an image in just one go, and can generate high frame rate images such as those in which two consecutive frames are divided 1000 times in the phase direction. In the example of FIG. 15 , an image similar to the captured image is already generated after one epoch has elapsed. In this respect, the phase noise generation model 50 can perform processing at speeds 100 times or more faster than the surface noise generation model 10. Since the two models have different use cases and purposes, a direct comparison is difficult, but the method of the phase noise generation model 50 is also reasonable from the perspective of machine learning.

[0067] The surface noise generation model 10 adds noise to the surface of an image and generates an image in a denoising process (despreading process). As a result, it is possible to generate a wide variety of interesting images. On the other hand, the phase noise generation model 50 performs a diffusion process on the phase of the image and generates an image in phase space by despreading, thereby generating an image suitable for camera signal processing.

[0068] The surface noise generation model 10 creates many patterns that do not exist in the camera image captured by the camera, making it extremely incompatible with camera signal processing. In contrast, the phase noise generation model 50 generates images by performing diffusion processing on the inter-frame, i.e., phase space, where camera image information is inherently present and does not exist there. This enables the formation of physically plausible image contours for camera images, demonstrating its compatibility with camera signal processing. Furthermore, based on the premise that previous and following frames exist naturally, the machine learning speed is approximately 30 to 100 times faster than that of the surface noise generation model 10, and the machine specifications (computational cost / VRAM cost) required for machine learning can be significantly reduced. However, the phase noise generation model 50 cannot generate images from completely noise images and always requires two or more image input sources.

[0069] While machine learning generative models that do not use Gaussian diffusion noise can achieve a certain level of phase control capability, this capability is comparable to that achieved with conventional variational autoencoders, and they face particular issues with blurring and contrast fluctuations, requiring post-processing adjustments by the designer. Because of this adjustment process, only a limited amount of phase control for image processing, such as focus control, which requires relatively low levels of abstraction, can be achieved. In contrast, the phase noise generative model 50 employs control information association learning with Gaussian diffusion, eliminating the post-processing required for phase control using conventional variational autoencoders and achieving highly stable generation capabilities that are closer to the intended target than conventional methods.

[0070] 8. Simplified Representation of Phase Noise Generation Model Below, an example of a generation model in which the phase noise generation model 50 is combined with another neural network will be described, rather than an example in which the phase noise generation model 50 is used alone.

[0071] In the drawings of the generative model described below, the phase noise generative model 50 shown in FIG. 4 is represented in a simplified representation as shown in FIG. 16. While the frequency decomposition unit 73E is omitted in FIG. 16, the frequency decomposition unit 73E at each layer of the encoder unit of the UNet 62 of the phase noise generative model 50 can output the low-frequency component L = (A + B) / 2 and the high-frequency component H = (AB) / 2 as intermediate products to the outside of the device. The simplified representation in FIG. 16 also includes the case where the frequency decomposition unit 73E is provided after the attention mechanism block 72E, as shown in FIG. 9, and frequency decomposition is performed including the results of the attention mechanism block 72E'. While the Gaussian diffusion unit 61, encoders 25 and 26, etc. are omitted in the simplified representation in FIG. 16, time information and metaprompt information are appropriately embedded in the residual combination block 71 or the attention mechanism block 72.

[0072] 9. Example of Application to Metadata Generation Model FIG. 17 is a block diagram showing a first example of application of the phase noise generation model 50, in which the phase noise generation model 50 is applied to a metadata generation model.

[0073] In the example of FIG. 17 , a neural network is configured by combining a phase noise generation model 50 and a metadata generation model 100. The metadata generation model 100 may be configured as either an image generation model for a plane for camera metadata generation or an image generation model for a phase for camera metadata generation. The metadata generation model 100 includes a UNet 101 and a VAE decoder (VAE DEC) 102. The UNet 101 has a multi-level hierarchy including a residual connection block 111 and an attention mechanism block 112, and has a network structure including an encoder unit and a decoder unit. The UNet 101 is either a Metadata Denoising UNet or a Metadata Phase Shift Denoising UNet.

[0074] In the neural network of FIG. 17 , the phase noise generation model 50 receives an image captured by a camera as input and generates and outputs an image based on control information indicating a value between 0 and 1 (e.g., an intermediate value of 0.5). Furthermore, the low-frequency component L=(A+B) / 2 and the high-frequency component H=(AB) / 2, which are intermediate products output by the frequency decomposition unit 73E of each layer of the encoder unit of the UNet 62 of the phase noise generation model 50, are supplied to a residual combination block 111D of the decoder unit of the metadata generation model 100 as frequency decomposition information (feature information) obtained by frequency-resolving the features of the latent space. The metadata generation model 100 generates metadata using the frequency decomposition information, which is an intermediate product of the phase noise generation model 50. Examples of the generated metadata include contour extraction data used during CG compositing in content production such as filmmaking, depth data of image depth, segment data of image composition, lens data, and gyro data.

[0075] 10. Example of Application as a Control Net FIG. 18 is a block diagram showing a second example of application of the phase noise generation model 50, in which the phase noise generation model 50 is applied as a control net.

[0076] A control net is a generative model that uses contour and skeleton information to control the composition of people and landscapes during image generation. Details are disclosed, for example, in "Adding Conditional Control to Text-to-Image Diffusion Models," by Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, Stanford University, https: / / arxiv.org / pdf / 2302.05543.pdf. A control net is configured by connecting a Denosing Diffusion net (hereinafter referred to as the "main net") and a control net via a zero-convolution block. The control net is identical to the main net. The zero-convolution block is a multiplication of zeros, and adds only zero data to the main net, so it has no effect on the main net in its initial state. In a generative model using a control net, only the main net is machine-learned in the initial state. Next, the main net is marked as completed, the dictionary data is locked, the same learning parameters are copied to the control net, and the zero-convolution block is unlocked. Contour information and skeleton information are input to the control net, and the information converted into latent space features within the control net is supplied to the main net via the zero convolution block. As a result, the latent space features of the control net act on the main net, generating an image with a similar composition to the input contour information and skeleton information. The control net uses a contour extraction filter, such as a Canny filter.

[0077] In the example of FIG. 18 , a neural network is configured by combining a phase noise generation model 50 and a surface noise generation model 140. The surface noise generation model 140 includes a VAE encoder (VAE ENC) 151, a UNet 152, a VAE decoder (VAE DEC) 153, and a zero-convolution block 154. The UNet 152 has a network structure with an encoder section and a decoder section, with multiple hierarchical layers including a residual connection block 161 and an attention mechanism block 162. The phase noise generation model 50 is used as a control net instead of a network using a contour extraction filter such as a Canny filter. Frequency decomposition information (feature information) obtained by frequency-resolving the features of the latent space obtained by the phase noise generation model 50 is supplied to each residual connection block 161D of the UNet 152 of the surface noise generation model 140 via the zero-convolution block 154.

[0078] In this way, according to a generation model that applies the phase noise generation model 50 as a control net, by using the frequency decomposition information obtained by the phase noise generation model 50, it is possible to control the composition of people and landscapes when generating images with more expressive contour information.

[0079] 11. Application Example to Template Matching FIG. 19 is a block diagram showing a third application example to which the phase noise generation model 50 is applied, in which an intermediate product of the phase noise generation model 50 is applied to template matching.

[0080] 19 includes a phase noise generation model 50, a buffering unit 171, and a template matching unit 172. The template matching model 170 performs template matching using intermediate products of the camera image input to the phase noise generation model 50, and, based on the obtained matching information, calculates and outputs vector information such as local motion vector information representing the movement of the subject in the camera image and global motion vector information representing the movement of the camera itself.

[0081] The frequency decomposition unit 73E of each layer of the encoder unit of UNet 62 outputs the output values ​​A and B of the input image obtained according to the control information (0.5±Δ) to the buffering unit 171. The buffering unit 171 temporarily stores the intermediate products supplied from the frequency decomposition unit 73E of each layer of the encoder unit of UNet 62.

[0082] As shown in Fig. 20, the template matching unit 172 performs template matching using the Sum of Absolute Difference (SAD) method between output values ​​A and B of multiple channels at each layer of the encoder unit of the UNet 62. Here, as a result of comparing output values ​​A and B of the latent space at the first layer of the encoder unit of the UNet 62, vector information (X V1 ,Y V1 ) is obtained, and the vector information (X V2 ,Y V2 ) is obtained, and the vector information (X V3 ,Y V3 ) is obtained. The template matching unit 172 uses, for example, the vector information (X V3 ,Y V3 ) as the final vector information of the template matching. Alternatively, the template matching unit 172 outputs the vector information (X V1 ,Y V1 ) ~ Vector Information (X V3 ,Y V3 The vector information of each layer may be merged by a predetermined calculation, such as averaging the vectors of the respective layers, and the result may be output as the final vector information. The two images to be matched are in a latent space, and feature points are captured from various viewpoints, improving the matching accuracy.

[0083] By performing template matching using the feature values ​​of each layer output by control information association, it is possible to improve the accuracy of vector analysis to a higher level than methods such as SAD and ZNCC. It is also possible to improve the accuracy of optical flow analysis and motion analysis.

[0084] 12. Example of Application to a Multimodal Network FIG. 21 is a block diagram showing a fourth example of application of the phase noise generation model 50, in which the phase noise generation model 50 is applied to a multimodal network.

[0085] Different types of modal data, namely, camera images and IMU data, are input to the multimodal network 200. The multimodal network 200 outputs a stabilized image after applying electronic image stabilization (EIS) to the input camera images, and also outputs attitude control information for controlling the attitude of the camera. The IMU data is data output from an IMU sensor including an acceleration sensor and an angular velocity sensor, and the IMU sensor is fixed to the camera. The IMU data includes acceleration data in each of three orthogonal axes and angular velocity data about each of the three orthogonal axes.

[0086] The multimodal network 200 includes a phase noise generation model 50, a noise generation model 210, a buffering unit 241, a template matching unit 242, a cache control unit 243, and a memory reference unit 244. The noise generation model 210 includes a VAE encoder (VAE ENC) 211, a UNet 212, and a VAE decoder (VAE DEC) 213. The UNet 212 has a network structure with multiple hierarchical layers including a residual connection block 221 and an attention mechanism block 222, and an encoder unit and a decoder unit.

[0087] Each layer of the UNet 62 receives metaprompt information, which is an image obtained by downconverting a camera image to N*M pixels. This metaprompt information controls the image composition. The frequency decomposition unit 73E in each layer of the encoder unit of the UNet 62 outputs the output values ​​A and B of the input image obtained according to the control information (0.5±Δ) to the buffering unit 241. The buffering unit 241 temporarily stores the intermediate products provided by the frequency decomposition unit 73E in each layer of the encoder unit of the UNet 62. The template matching unit 242 performs template matching between the output values ​​A and B of multiple channels in each layer of the encoder unit of the UNet 62 using the Sum of Absolute Difference (SAD) method. The processing of the buffering unit 241 and the template matching unit 242 is similar to that of the buffering unit 171 and the template matching unit 172 in the third application example shown in FIG. 20 . The template matching unit 242 outputs vector information, which is the result of template matching for each layer, to the same layer of the UNet 212 of the noise generation model 210 as a feature.

[0088] The cache control unit 243 acquires posture control features from the residual combination block 221D or the attention mechanism block 222D of each layer of the decoder unit of the UNet 212 of the noise generation model 210. The cache control unit 243 calculates a warp amount for rotating or deforming the pixel position of each pixel of the camera image based on the posture control features supplied from each layer of the decoder unit of the UNet 212 and the vector information from the template matching unit 242. Based on the warp amount calculation result, the cache control unit 243 calculates the pixel positions of multiple reference pixels (e.g., 4x4 pixels) to be referenced in the buffering unit 241 for each pixel of the stabilized image, and outputs the pixel positions to the memory reference unit 244. For each pixel of the stabilized image, the memory reference unit 244 acquires latent space features corresponding to the pixel values ​​of the multiple reference pixels from the buffering unit 241, and outputs the latent space features to each layer of the decoder unit of the UNet 62. Each layer of the decoder section of the UNet 62 converts from low resolution to high resolution while embedding the feature amount supplied from the memory reference section 244 .

[0089] In the multimodal net 200, the features of the latent space, rather than the camera image itself, are buffered in the buffering unit 241. In the multimodal net 200, the encoder and decoder units of the UNet 62 of the phase noise generation model 50 are not connected, and the features of the latent space referenced by the memory reference unit 244 are embedded in each layer of the decoder unit of the UNet 62, thereby providing the effect of pixel interpolation.

[0090] Numerous techniques have been proposed to analyze one-dimensional signals, such as speech recognition, using autoencoders to analyze scenes such as walking and running. Methods for controlling electronic stabilization based on scene understanding have also been proposed. However, autoencoders have limited information generation capabilities, and in particular, they have only been able to implement limited processing, such as applying effects that add a sense of realism to electronic stabilization. The multimodal network 200 in Figure 21 employs a multimodal network configuration that connects both camera images and IMU data, enabling it to generate effect information based on image information and rotation information, thereby providing the ability to apply unique effects to electronic stabilization.

[0091] 13. Example of Application to an Image Generation Network Using Both Planes and Phases> FIG. 22 is a block diagram showing a fifth example of application to which the phase noise generation model 50 is applied, and is an example of a network using both an image generation model for phases and an image generation model for planes.

[0092] The image generation network 250 includes a phase noise generation model 50, which is an image generation model for phase, and a surface noise generation model 260, which is an image generation model for surfaces. A first image is input to the phase noise generation model 50, which generates and outputs a first generated image. A second image is input to the surface noise generation model 260, which generates and outputs a second generated image. The input first and second images may be the same or different. A possible application for using different first and second images is, for example, the image synthesis shown in FIG. 13 , which aims to form completely different contours. This is expected to be used, for example, in CG synthesis in movie production. A possible application for using the same image is, for example, performing phase control and surface control as signal processing of camera images.

[0093] The surface noise generation model 260 includes a VAE encoder (VAE ENC) 261, a UNet 262, a VAE decoder (VAE DEC) 263, and a pixel prompt generator 264. The UNet 262 has a network structure with multiple layers, each including a residual connection block 271 and an attention mechanism block 272, and an encoder and decoder. The pixel prompt generator 264 generates a down-converted image by down-converting the second image to N*M pixels. The generated down-converted image is supplied to each layer of the UNet 262 as meta-prompt information.

[0094] Each layer of the UNet 62 of the phase noise generation model 50 outputs the generated intermediate products to the residual combination block 271 or the attention mechanism block 272 of the same layer of the UNet 262 of the surface noise generation model 260. Each layer of the UNet 262 of the surface noise generation model 260 outputs the generated surface features to the residual combination block 71 or the attention mechanism block 72 of the same layer of the UNet 62 of the phase noise generation model 50.

[0095] The phase noise generation model 50 is capable of generating only phase information from camera images as faithfully as possible, but some use cases may also involve generating some objects. In the surface noise generation model 260, to minimize the generation of false objects on surfaces, the input second image is downconverted to N*M, and the resulting downconverted image is input as metaprompt information to each layer of the UNet 262, controlling the second generated image. For example, if the input second image is 512x512 pixels, the downconverted image is reduced to, for example, 16x16 to 32x32 pixels. In the pre-training stage, the surface noise generation model 260 performs machine learning to restore the metaprompt information as closely as possible to super-resolution, and then in the fine-tuning stage, fine-tuning is performed specifically for videos of specific scenes. As a result, images for surfaces are generated to the extent possible using objects present in the scenery of the second image, enabling image generation suitable for camera signal processing. Combining both surface and phase image generation enables image generation that takes into account frames distant from the input image, which is a weakness of the phase noise generation model 50. This is because landscape information is accumulated in dictionary data during the fine-tuning stage on the surface noise generation model 260 side, meta-prompt information becomes a reference address for the dictionary data, and the necessary features in the latent space are referenced and transferred to the phase noise generation model 50.

[0096] <14. Summary> The phase noise generation model 50 described above includes a VAE encoder 22 that converts an input image into features in a latent space, a UNet 62 consisting of multiple layers, each layer including a residual combination block 71 and an attention mechanism block 72, a Gaussian diffusion unit 61 (control information generation unit) that generates multiple pieces of control information based on input values ​​and outputs the generated pieces of control information to one or more layers, and a frequency decomposition unit 73E that frequency-decomposes the multiple latent space features obtained in accordance with the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, the low-frequency components being output to a layer subsequent to the layer, and the high-frequency components being output to a layer skip-connected to the layer, the Gaussian diffusion unit 61 generating Gaussian diffusion noise corresponding to the multiple pieces of control information, and the UNet 62 performing machine learning to denoise the Gaussian diffusion noise during machine learning.

[0097] The phase noise generation model 50 performs a diffusion process in the phase direction of the image and generates an image in the phase direction by de-diffusion. Because the input camera image obviously exists, it is possible to generate an image that is faithful to the input camera image and to generate an image that compensates for only the missing information. Furthermore, because the camera image obviously exists, the speed of machine learning is very fast, making it possible to generate images much faster than the surface noise generation model 10.

[0098] The phase noise generation model 50 performs Wavelet transform on the features in the latent space at each frequency / abstraction level of the UNet 62. Using the components obtained by Wavelet transform, it is possible to use the latent space from high resolution / low abstraction to low resolution / high abstraction. Mathematically, this technology is an AI extension of Wavelet transform.

[0099] While general image generation AI aims to generate multimodal images including illustrations, animation, and CG, the phase noise generation model 50 is a physics-based generation technology that restores the behavior of frames between camera observation data in a physically plausible manner, and can be used for observation cameras that require higher fidelity. The target generation domain is not limited to camera images, but can also include inertial sensors, depth sensors, and spectral sensors, and it is expected that it can be used to generate plausible images by combining multiple sensors.

[0100] In conventional CG-integrated VFX video production, it was common to use a filming style requiring specialized equipment, such as chromakey technology for creating a green screen background, rotoscope technology for extracting contours, and motion tracking using markers. In contrast, the Phase Noise Generation Model 50 simplifies the process using AI technology, proposing a solution that can be used for online video distribution, which does not require specialized equipment.

[0101] 15. Example Computer Configuration The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes microcomputers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0102] FIG. 23 is a block diagram showing an example of the hardware configuration of an information processing device that executes the above-described series of processes by a program.

[0103] The information processing device 400 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, and a RAM (Random Access Memory) 403. The CPU 401, the ROM 402, and the RAM 403 are connected to one another by a bus 404.

[0104] The information processing device 400 further has an input / output interface 405 connected to the bus 404 , and the input / output interface 405 is connected to an input unit 406 , an output unit 407 , a storage unit 408 , a communication unit 409 , and a drive 410 .

[0105] The input unit 406 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 407 includes a display, speaker, output terminal, etc. The storage unit 408 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 409 includes a network interface, etc. The drive 410 drives removable media 411 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0106] In the information processing device 400 configured as above, the CPU 401 loads a program stored in the storage unit 408 into the RAM 403 via the input / output interface 405 and the bus 404 and executes the program, thereby performing the above-described series of processes. The RAM 403 also stores data and the like necessary for the CPU 401 to execute various processes as appropriate. The information processing device 400 can appropriately select and execute the above-described phase noise generation model 50 alone or the generation models of the first to fifth application examples to which the phase noise generation model 50 is applied, for example, by switching setting information such as parameters.

[0107] The program executed by the CPU 401 of the information processing device 400 can be provided by being recorded on a removable medium 411 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0108] By inserting the removable medium 411 into the drive 410, the program can be installed in the storage unit 408 via the input / output interface 405. The program can also be received by the communication unit 409 via a wired or wireless transmission medium and installed in the storage unit 408. Alternatively, the program can be installed in the ROM 402 or the storage unit 408 in advance.

[0109] The program executed by CPU 401 may be a program that is processed chronologically in the order described in this specification, or may be a program that is processed in parallel or at the required timing, such as when called.

[0110] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0111] For example, it is possible to adopt a form in which all or part of the above-mentioned multiple generative models and neural networks are appropriately combined.

[0112] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0113] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0114] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0115] The technology disclosed herein may employ the following configuration: (1) An image processing device comprising: a variational autoencoder that converts an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual connection block and an attention mechanism block; a control information generation unit that generates multiple pieces of control information based on input values ​​and outputs the control information to one or more of the layers; and a frequency decomposition unit that frequency-decomposes the multiple features obtained according to the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer skip-connected to the layer; the control information generation unit generates Gaussian diffusion noise corresponding to the multiple pieces of control information; and the layered network performs machine learning to denoise the Gaussian diffusion noise during machine learning. (2) The image processing device described in (1), wherein the variational autoencoder converts two images into features in a latent space during machine learning, and the layered network performs machine learning by associating the features in the latent space corresponding to the two images with the multiple pieces of control information during machine learning. (3) The image processing device according to (1) or (2), wherein the frequency decomposition unit outputs the low-frequency components and high-frequency components to an external device. (4) The image processing device according to any of (1) to (3), wherein the frequency decomposition unit frequency-decomposes output values ​​of the residual combined block corresponding to the plurality of pieces of control information into low-frequency components and high-frequency components. (5) The image processing device according to any of (1) to (4), wherein the frequency decomposition unit frequency-decomposes output values ​​of the attention mechanism block corresponding to the plurality of pieces of control information into low-frequency components and high-frequency components. (6) The image processing device according to any of (1) to (5), wherein the low-frequency components and high-frequency components are wavelet transforms of the feature quantities of the latent space. (7) The image processing device according to any of (1) to (6), further comprising a vector quantization unit that vector-quantizes the feature quantities of the latent space transformed by the variational autoencoder.(8) The image processing device according to any one of (1) to (7), further comprising: a metadata generation model that generates metadata using, as input, frequency decomposition information obtained by frequency-resolving the plurality of feature quantities obtained from the plurality of pieces of control information. (9) The image processing device according to any one of (1) to (8), further comprising: a surface noise generation model that generates an image using, as input, frequency decomposition information obtained by frequency decomposing the plurality of feature quantities obtained from the plurality of pieces of control information, the surface noise generation model performing machine learning to denoise Gaussian diffusion noise on the surface of the image during machine learning. (10) The image processing device according to any one of (1) to (9), further comprising: a template matching unit that performs template matching using, as input, the plurality of feature quantities obtained from the plurality of pieces of control information. (11) The image processing device according to any one of (1) to (10), further comprising: a generative model that inputs modal data different from the image, the generative model inputs the plurality of feature quantities obtained from the plurality of pieces of control information, and the hierarchical network inputs the feature quantities of the latent space obtained by the generative model. (12) The image processing device according to any one of (1) to (11), further comprising a surface noise generation model that performs machine learning to denoise Gaussian diffusion noise on image surfaces during machine learning, wherein the surface noise generation model inputs the plurality of feature quantities obtained from the plurality of control information in the hierarchical network, and the hierarchical network inputs the feature quantities of the latent space obtained by the surface noise generation model.(13) An image processing method including: a variational autoencoder converting an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual connection block and an attention mechanism block, performing machine learning to denoise Gaussian diffusion noise during machine learning; a control information generation unit generating Gaussian diffusion noise corresponding to multiple pieces of control information based on an input value and outputting the Gaussian diffusion noise to one or more of the layers; and a frequency decomposition unit frequency-decomposing multiple features obtained according to the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer connected to the layer by a skip connection. (14) A program for causing a computer to function as: a variational autoencoder that converts an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual connection block and an attention mechanism block; a control information generation unit that generates multiple pieces of control information based on input values ​​and outputs the control information to one or more of the layers; and a frequency decomposition unit that frequency-decomposes multiple features obtained according to the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer skip-connected to the layer; the control information generation unit generates Gaussian diffusion noise corresponding to the multiple pieces of control information; and the layered network denoises the Gaussian diffusion noise during machine learning.

[0116] 50 Phase noise generation model, 61 Gaussian diffusion unit, 71 Residual combination block, 72 Attention mechanism block, 73E Frequency decomposition unit, 100 Metadata generation model, 111 Residual combination block, 112 Attention mechanism block, 140 Surface noise generation model, 154 Zero convolution block, 161 Residual combination block, 162 Attention mechanism block, 170 Template matching model, 171 Buffering unit, 172 Template matching unit, 200 Multimodal net, 210 Noise generation model, 221 Residual combination block, 222 Attention mechanism block, 241 Buffering unit, 242 Template matching unit, 243 Cache control unit, 244 Memory reference unit, 250 Image generation net, 260 Surface noise generation model, 264 Pixel prompt generation unit, 271 Residual combination block, 272 Attention mechanism block, 400 Information processing device, 401 CPU, 402 ROM, 403 RAM, 406 Input unit, 407 Output unit, 408 Storage unit, 409 Communication unit, 410 Drive, 411 Removable media

Claims

1. An image processing device comprising: a variational autoencoder that converts an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual connection block and an attention mechanism block; a control information generation unit that generates multiple pieces of control information based on input values ​​and outputs the control information to one or more of the layers; and a frequency decomposition unit that frequency-decomposes multiple features obtained in accordance with the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer skip-connected to the layer; the control information generation unit generates Gaussian diffusion noise corresponding to the multiple pieces of control information; and the layered network performs machine learning to denoise the Gaussian diffusion noise during machine learning.

2. The image processing device described in claim 1, wherein the variational autoencoder converts the two images into features in a latent space during machine learning, and the hierarchical network performs machine learning by associating the features in the latent space corresponding to the two images with the multiple pieces of control information during machine learning.

3. The image processing device according to claim 1, wherein the frequency decomposition unit outputs the low-frequency components and the high-frequency components to an external device.

4. The image processing device according to claim 1, wherein the frequency decomposition unit performs frequency decomposition of the output values ​​of the residual combined block corresponding to the plurality of pieces of control information into low frequency components and high frequency components.

5. The image processing device according to claim 1, wherein the frequency decomposition unit frequency-decomposes the output values ​​of the attention mechanism block corresponding to the plurality of pieces of control information into low-frequency components and high-frequency components.

6. The image processing device according to claim 1, wherein the low-frequency components and high-frequency components are wavelet transforms of the features in the latent space.

7. The image processing device according to claim 1, further comprising a vector quantization unit that vector-quantizes the feature quantities in the latent space transformed by the variational autoencoder.

8. The image processing device according to claim 1, further comprising: a metadata generation model that generates metadata using, as an input, frequency decomposition information obtained by frequency-decomposing the plurality of feature quantities obtained from the plurality of pieces of control information.

9. The image processing device according to claim 1, further comprising a surface noise generation model that generates an image using frequency decomposition information obtained by frequency-decomposing the plurality of feature quantities obtained from the plurality of control information as input, wherein the surface noise generation model performs machine learning to denoise Gaussian diffusion noise on the surface of the image during machine learning.

10. The image processing device according to claim 1, further comprising a template matching unit that performs template matching using the plurality of feature amounts obtained from the plurality of pieces of control information as input.

11. The image processing device according to claim 1, further comprising a generative model that inputs modal data different from the image, wherein the generative model inputs the plurality of features obtained from the plurality of control information, and the hierarchical network inputs the features of the latent space obtained by the generative model.

12. The image processing device according to claim 1, further comprising a surface noise generation model that performs machine learning to denoise Gaussian diffusion noise on image surfaces during machine learning, wherein the surface noise generation model inputs the multiple feature quantities obtained from the multiple pieces of control information in the hierarchical network, and the hierarchical network inputs the feature quantities of the latent space obtained by the surface noise generation model.

13. An image processing method comprising: a variational autoencoder converting an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual connection block and an attention mechanism block, performing machine learning to denoise Gaussian diffusion noise during machine learning; a control information generation unit generating Gaussian diffusion noise corresponding to multiple pieces of control information based on input values ​​and outputting the generated Gaussian diffusion noise to one or more of the layers; and a frequency decomposition unit frequency-decomposing the multiple features obtained in accordance with the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer connected to the layer by a skip connection.

14. A machine learning program that causes a computer to function as: a variational autoencoder that converts an input image into features in a latent space; a layered network consisting of multiple layers, each layer including a residual connection block and an attention mechanism block; a control information generation unit that generates multiple pieces of control information based on input values ​​and outputs the control information to one or more of the layers; and a frequency decomposition unit that frequency-decomposes multiple features obtained in accordance with the multiple pieces of control information input to the layers into low-frequency components and high-frequency components, wherein the low-frequency components are output to a layer subsequent to the layer, and the high-frequency components are output to a layer skip-connected to the layer, wherein the control information generation unit generates Gaussian diffusion noise corresponding to the multiple pieces of control information, and the layered network denoises the Gaussian diffusion noise during machine learning.

Images