Image processing apparatus, image processing method, and image processing program

Abstract

To improve the quality of super-resolution of an image including a blurred part.SOLUTION: Inference using the machine learning model (generator) 43 is performed with target image data to be subjected to image processing and reference image data based on the target image data as input, output image data based on the target image data is generated, feature information (ChannelAttention map) representing a feature of the target image data is calculated using the reference image data based on the target image data in the inference, and the feature information is multiplied by the target image data for each channel and applied, thereby generating output image data reflecting the feature information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] Image generation AI (generative AI) technologies such as generative adversarial networks (GANs) are known (Patent Document 1). Additionally, in the field of image generation, there is a well-known technology called super-resolution, which generates high-resolution images from low-resolution images. Super-resolution is not simply a technique for enlarging a low-resolution image, but rather a technique for predicting and restoring the high-frequency components of the low-resolution image to increase the resolution and generate a high-resolution image. Simply enlarging a low-resolution image results in a blurred image, but super-resolution can increase the resolution while removing the resulting blur. In relation to super-resolution, real-ESRGAN (Non-Patent Document 1) has been proposed. In real-ESRGAN, blurred images are obtained by performing degradation processing on the output teacher image data, and this is set as the input teacher image data to train a generator. Here, the input-side teacher image data and the output-side teacher image data are the teacher image data that is input to the machine learning model and the teacher image data that should be output from the machine learning model, respectively, when the machine learning model is trained. In the image generation AI, the input-side teacher image data is input to the learning model and image processing is performed, thereby causing the learning model to learn so that an image similar to the output-side teacher image data is obtained. An image is generated by inputting image data into a machine image model as a trained generator and performing inference. In the case of the super-resolution method in Non-Patent Document 1, a super-resolution image is generated in which the resolution is increased and blur is removed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-205030 [Non-patent literature]

[0004] [Non-Patent Document 1] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data, Xintao Wang, Liangbin Xie, Chao Dong, Ying Shan, 2021 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the super-resolution technique of Non-Patent Document 1, image generation is performed without distinguishing whether the input image data is an image blurred by degradation processing or an image in which the background, etc. is blurred based on the depth of field. As a result, the characteristics of the input image data may not be reflected, such as all blurs below a certain level, such as the background, being sharpened, resulting in an image of poor image quality. One aspect of the present invention is to improve the image quality of super-resolution images. [Means for solving the problem]

[0006] In one aspect, the present invention includes an inference unit that performs inference using a machine learning model using processing target data based on target image data and reference image data based on the target image data as input, and generates generated image data based on the processing target data, wherein the inference unit calculates first feature information representing characteristics of the target image data using the reference image data in the inference, and generates the generated image data that reflects the first feature information by applying the first feature information to the processing target data input to the machine learning model. [Effects of the Invention]

[0007] According to one aspect of the present invention, the image quality of a super-resolution image can be improved. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram of the image processing device. [Figure 3] FIG. 2 is a diagram illustrating an image generation process executed by the image processing device. [Figure 4] 1A to 1C are diagrams illustrating images generated by the image processing apparatus of this embodiment. [Figure 5] FIG. 2 is a diagram illustrating the configuration of a generator in a first example of this embodiment. [Figure 6] FIG. 1 is a diagram illustrating the configuration of a first example of an attention mechanism. [Figure 7] FIG. 10 is a diagram illustrating the configuration of a second example of an attention mechanism. [Figure 8] FIG. 10 is a diagram illustrating the configuration of a third example of an attention mechanism. [Figure 9] FIG. 10 is a diagram illustrating the configuration of a generator in a second example of this embodiment. [Figure 10] FIG. 10 is a diagram illustrating a process performed by a processing unit of a generator according to a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating a process performed by a processing unit of a generator according to a second embodiment. [Figure 12] 10A and 10B are diagrams illustrating problems that arise when super-resolution processing is performed for each block. [Figure 13] 10A and 10B are diagrams illustrating the relationship between block division of target image data and reference image data. [Figure 14] 10A and 10B are diagrams illustrating the relationship between block division of target image data and reference image data. [Figure 15] FIG. 10 is a diagram illustrating a method for identifying the most intricate part in an image. [Figure 16] 10 is a flowchart illustrating a learning process of a generator according to the present embodiment. [Figure 17] 10 is a flowchart illustrating an image generation process according to the present embodiment. [Figure 18] FIG. 1 is a block diagram illustrating an embodiment of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. First, the processing performed by the image processing apparatus according to this embodiment will be outlined. In machine learning models for upsampling (increasing resolution) that perform denoising (removing noise), convolution alone only references a narrow range. Therefore, the generator cannot distinguish whether a specific location in the upsampled image is blurred or not. As a result, the output super-resolution image may have blurred areas that are due to the depth of field inherent in the target image data, resulting in a loss of image quality due to excessive sharpening. The image processing device of this embodiment uses an attention mechanism in a machine learning model that performs super-resolution processing to improve the image quality of a super-resolution image. In this embodiment, the machine learning model that performs super-resolution processing is called a generator. It is well known that attention plays an important role in human perception. The human visual system does not process all visual information at once, but instead selectively focuses on salient parts. This is attention, and in recent years, there have been many attempts to incorporate attention mechanisms to improve the performance of CNNs. Known types of attention include Channel Attention and Spatial Attention. The Channel Attention Map (first feature information), a feature calculated by Channel Attention, focuses on "what" in the input image is meaningful. The Spatial Attention Map (second feature information), a feature calculated by Spatial Attention, complements Channel Attention by focusing on "where" in the input image is informative. The attention mechanism creates a channel attention map by utilizing the relationship between the channels of features in the feature map, and also generates a spatial attention map by utilizing the spatial relationship between features.

[0010] The generator of this embodiment calculates a Channel Attention map as a feature of the target image data by inputting image data based on the target image data into an improved attention mechanism as reference image data. By performing self-attention, which applies the Channel Attention map of the target data itself and a complementary Spatial Attention map to the target data, a high-quality super-resolution image that is closer to the target image can be obtained. Furthermore, the reference image data and the Channel Attention map based on it contain information such as the degree of blurring of the target image data as a whole. Therefore, the self-attention of Channel Attention has the effect that as a result of super-resolution processing, only the blur caused by upsampling is sharpened, and background blur caused by depth of field is not sharpened more than necessary.

[0011] In this embodiment, the target image data for super-resolution processing is divided into blocks, and the blocks are then combined to obtain a super-resolution image of the entire target image data. In this case, as will be described in detail with reference to FIG. 12, there is a possibility that the target block does not contain the characteristics of the entire target image data, including the degree of blur. In this case, even if self-attention is performed on the target block using a channel attention map extracted from the target block itself, the characteristics of the entire target image data will not be reflected, and a super-resolution image with the desired image quality cannot be obtained. This is because it is impossible to distinguish whether the target block should be blurred after upsampling. 13 and 14, the present embodiment performs processing in which blocks surrounding the target block, the entire target image data, the most complex part of the target image data, etc. are used as reference image data, and feature amounts extracted from the reference image data are applied to the target block. This embodiment can also be said to perform improved self-attention.

[0012] The image processing apparatus of this embodiment will be described in more detail below. The machine learning model used in this embodiment may be configured, for example, by a neural network or a convolutional neural network (CNN) including a convolution layer. In the following description, the machine learning model will be simply referred to as a learning model. <Image processing device> FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to this embodiment, and FIG. 2 is a block diagram illustrating the functional blocks of the image processing apparatus. As shown in FIG. 1, the image processing device 1 includes a control unit 10, an image processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, and a reading unit 16. Although the image processing device 1 and the operation of the image processing device 1 will be described below as being one computer, it may be configured so that processing is distributed among a plurality of computers.

[0013] The control unit 10 uses a processor such as a CPU (Central Processing Unit), a memory, and the like to control each component of the device and realize various functions. The image processing unit 11 uses a processor and memory such as a GPU (Graphics Processing Unit) or a dedicated circuit, and executes image processing in response to control instructions from the control unit 10. The control unit 10 and the image processing unit 11 may be integrated into one piece of hardware. Alternatively, the control unit 10 and the image processing unit 11 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor such as a CPU or GPU, a memory, and further a storage unit 12 and a communication unit 13.

[0014] A hard disk, flash memory, or RAM (Random Access Memory) is used as the storage unit 12. An image processing program 1P and a machine learning library 121L are stored in the storage unit 12. The storage unit 12 also stores definition data that defines a generator 43, which is a learning model created for each learning, parameter information of the learned generator 43, and the like. The definition data is information indicating, for example, the network configuration (layer configuration) of a neural network, and the parameter information is information indicating the weights of nodes in the neural network.

[0015] In the following description, parameters such as node weights in a neural network will be referred to as "NN parameters." The storage unit 12 can store target image data that is the target of super-resolution processing, output image data that is the result of the super-resolution processing, teacher image data for learning by the generator 43, and the like. A learning model defined by definition data and NN parameters is incorporated into the machine learning library 121L, and the machine learning library 121L is executed by the control unit 10, whereby the image processing unit 11 realizes a generator 43 that performs the learning and inference of this embodiment.

[0016] The generator 43 can be configured by appropriately combining a network such as a transposed convolution layer, a convolution layer, or an upsampling layer with an attention mechanism, which will be described later. The trained generator 43 generates and outputs upsampled image data from input seed data and reference image data, which will be described later. The machine learning library 121L has an inference engine (program) that performs learning and inference, and is executed by providing definition data and NN parameters, thereby providing the definition data and NN parameters with the functionality of a machine learning model. Typical examples of the machine learning library 121L include TensorFlow and Cafe, but any machine learning library may be used without being limited to these.

[0017] The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet, etc. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module. The display unit 14 uses a liquid crystal panel, an organic EL (Electro Luminescence) display, etc. The display unit 14 is capable of displaying an image through processing by an image processing unit 11 in accordance with an instruction from the control unit 10. The operation unit 15 includes a user interface such as a keyboard or a mouse. Physical buttons provided on the housing may be used, and software buttons displayed on the display unit 14 may also be used. The operation unit 15 notifies the control unit 10 of operation information by the user. The reading unit 16 can use, for example, a disk drive to read the image processing program 2P and the machine learning library 21L stored on a recording medium 2 such as an optical disk. The image processing program 1P and the machine learning library 121L stored in the storage unit 12 may be the image processing program 2P and the machine learning library 21L read by the reading unit 16 from the recording medium 2 and copied into the storage unit 12 by the control unit 10.

[0018] The control unit 10 of the image processing device 1 functions as a learning process execution unit 31 and an image processing execution unit (inference process control unit) 32 based on the image processing program 1P stored in the storage unit 12. The image processing unit 11 functions as a generator 43 using a memory based on the machine learning library 121L, definition data, and parameter information stored in the storage unit 12. The image processing unit 11 also functions as an input unit 41 for inputting an image to the generator 43, and an output unit 45 for outputting an image to the storage unit 12, etc. The functions of the input unit 41 and the output unit 45 may be possessed by the generator 43, or may be possessed by the learning process execution unit 31 and the image processing execution unit 32 executed by the control unit 10. The image processing unit 11 also functions as a division unit 42 that divides teacher image data and target image data into blocks, and a combination unit 44 that combines output data from the generator 43. The functions of the division unit 42 and the combination unit 44 may be possessed by the generator 43, or may be possessed by the learning process execution unit 31 and the image processing execution unit 32 executed by the control unit 10. The image processing unit 11 may function as an extraction unit that extracts a specific part or block with the most complex pixels from the teacher image data or the target image data. In FIG. 2, the function of the generator 43 is realized by the machine learning library 121L, and therefore the machine learning library 121L is not shown in the storage unit 12. The learning process execution unit 31 also performs processing to set the machine learning model as the generator 43 by learning the NN parameters of the generator 43 based on the machine learning library 121L and the teacher image data.

[0019] The image processing execution unit 32 inputs seed image data and reference image data to the learned generator 43 and executes a process of acquiring image data generated by the generator 43 . The image processing execution unit 32 may render the image data output from the generator 43 as an image and output it to the display unit 14.

[0020] FIG. 3 is a diagram illustrating the image generation process executed by the image processing device. The image generation process is performed under the control of the image processing execution unit 32. The input unit 41 acquires the target image data stored in the storage unit 12 . The dividing unit 42 inputs block image data (data to be processed) obtained by dividing the target image data to the generator 43 together with the reference image data. The generator 43 uses the input block image data, the reference image data, the definition data stored in the storage unit 12, and the NN parameters to generate an image using the input block image data as a seed, and generates output block image data (generated image data). The combining unit 44 combines the generated output block image data to obtain output image data. The output unit 45 outputs the output image data to the storage unit 12 or the like.

[0021] Figure 4 is a diagram illustrating an image generated by the image processing device of this embodiment, where (a) is a diagram of input image data, (b) is a diagram of the entire image being out of focus, (c) is a diagram of the entire image being sharpened, and (d) is a diagram of the foreground being sharpened and the background being blurred according to the depth of field. As shown in Figure 4(a), the input image is a low-resolution image in which the person in the foreground is in focus and the background scenery is blurred due to the depth of field. If the low-resolution image is enlarged by simply dividing the input image data into blocks and combining the blocks, the entire image will be blurred as shown in Figure 4(b). In the conventional technology described in Non-Patent Document 1, the resolution of the input image data is increased and sharpened, so that both the person in the foreground and the background are sharpened, resulting in an image that differs from the original image, as shown in Figure 4(c). On the other hand, according to the image processing device of this embodiment, it is possible to sharpen only the person in the foreground while keeping the background blurred in the original image as shown in Fig. 4(d). The image processing device 1 of this embodiment is able to realize high-quality super-resolution, as shown in Fig. 4(d), which captures the overall image characteristics of the low-resolution image of Fig. 4(a).

[0022] As described above, in super-resolution models such as Real-ESRGAN in Non-Patent Document 1, when a high-resolution image is generated based on seed image data, there is a problem in that all blur below a certain level is sharpened, resulting in an image that differs from the original. In contrast, as will be described in detail below, when generating a high-resolution image using block image data as a seed, the image processing device 1 processes the block image data to be processed (processing target block) using a channel attention map that indicates the characteristics of the entire image of the target image data. This allows the image processing device 1 to increase the resolution of the target image data with high image quality while maintaining the original image. For example, blurring of the background, etc., of the target image data is not sharpened, and the target image data is increased in resolution while leaving the unblurred parts as they are.

[0023] FIG. 5 is a diagram illustrating the configuration of a generator in a first example of this embodiment. 5 includes neural networks (hereinafter simply referred to as networks) 50 (50A, 50B, 50C, 50D) and attention mechanisms 51 (51A, 51B, 51C) provided between the networks 50. The attention mechanisms 51 can also be configured as neural networks including convolutional layers, fully connected layers, etc., but are distinguished from the networks 50 here. The number of networks 50 and attention mechanisms 51 is not limited to the example of FIG. 5, but the generator 43A includes two or more networks 50 and attention mechanisms 51 provided between the networks 50. The generator 43A receives input block image data obtained by dividing the target image data into blocks and reference image data, performs super-resolution processing on the input block image data, and outputs output block image data. The generator 43A combines the output block image data to obtain final output image data in which the target image data has been super-resolution processed. The reference image data is image data that represents the structure and characteristics of the target image data, including the degree of blur, based on the target image data. For example, the entire target image data or a specific part of the target image data can be used as the reference image data. In the generator 43A, the network 50A serves as an input stage for inputting input block image data, and the network 50D serves as an output stage for outputting output block image data. The network 50 and the attention mechanism 51 may be Residual Neural Networks (ResNet), Recurrent Neural Networks (RNN), the above-mentioned CNN, or the like. The network 50A receives input block image data, performs convolution (Conv), linear processing (ReLU), pooling, and other processes, and outputs the processed image data (feature map) to the next-stage attention mechanism 51A. The networks 50B, 50C, and 50D may have the same configuration as the network 50A, or each network may have a different configuration.

[0024] The attention mechanism 51A receives the output data from the network 50A and the RGB data of the reference image data. The attention mechanism 51A adds reference image data, which is a feature of the target image data, to the data input from the network 50A, performs convolution, linear processing, pooling, sigmoid function processing, etc., and outputs the result to the next-stage network 50B. The attention mechanisms 51B, 51C, and 51D may employ the same configuration as the attention mechanism 51A, or may employ different configurations. Details of the attention mechanisms will be described again using Figures 6 to 8, but all of the attention mechanisms have in common that they input RGB data of the reference image data and perform attention function processing.

[0025] The network 50 and the attention mechanism 51 repeatedly process each other's output data, and the output data of the output-stage network 50 (here, network 50D) is finally output as output block image data. By using the attention mechanism 51, features including the degree of blur of the target image data can be reflected in the processing in the network and attention mechanism, so that the target image data can be made high-resolution with high image quality while retaining the original image. For example, blur in the background, etc., of the target image data is not sharpened, and the target image data is made high-resolution while leaving the unblurred parts as they are. In addition, since the attention mechanism 51 in the subsequent stage performs processing to reflect reference image data on the output data from each network 50 except for the output-stage network 50D, super-resolution processing can be performed effectively.

[0026] Hereinafter, the configuration of the generator using the attention mechanism according to this embodiment will be described with reference to FIGS. In the following explanation, "C" in the format C×W×H represents the number of channels of the image data handled by the attention mechanism, "W" represents the width (number of pixels) of the image data handled by the attention mechanism, and "H" represents the height (number of pixels) of the same image data. The attention mechanism shown in Figures 6 and 7 can output new image data (feature map of C x W x H) that reflects the overall features of the target image data by multiplying the image data (C x W x H) input from the network 50 by information extracted from reference image data that represents the features of the target image data for each channel. Also, the attention mechanism shown in Figure 8 can output new image data (feature map of C x W x H) that reflects the overall features of the target image data by processing the information input from the network 50 with Scaled Dot Product Attention. In this way, by alternately performing network processing and processing in the attention mechanism, the target image data can be made high quality and high resolution while retaining the original image as a result of processing by the generator 43A. In Figures 6 to 8, Channel Attention is channel-direction attention, using global average pooling, global max pooling, etc., to weight each channel by focusing on the importance of the information contained in each channel of the image. Spatial Attention is spatial-direction attention, focusing on where objects are located within the image. Scaled Dot Product Attention uses scaling and matrix multiplication to extract information from V that is similar to K in terms of Q. Since the input information to Q includes information about the reference image, the output reflects information from the reference image data.

[0027] FIG. 6 is a diagram illustrating the configuration of a first example of an attention mechanism (Channel Attention). In the example of FIG. 6, the attention mechanism 51 executes the following processes in the learning and inference processes of the generator 43A. The attention mechanism 51 inputs RGB data (3×W'×H') of the reference image data shown in (A). Because pooling is included during processing, the data size can be selected arbitrarily, and W' and H' can have any value. This is because the reference image data can be various regions in the target image data, as explained in Figures 13 and 14. The attention mechanism 51 performs convolution on the RGB data of the reference image data in (B) to obtain the convolution result data (n × W' × H'). By performing convolution, the RGB 3ch data is combined into n channels, making it possible to refer to the overall characteristics of the reference image data. The number of channels n of the convolution result can be any number. The attention mechanism 51 performs average pooling and max pooling in parallel on the convolution results in (C), and obtains n×1×1 pooling results respectively. The attention mechanism 51 adds both pooling results in (D) to obtain an addition result of (n×2)×1×1. The attention mechanism 51 processes the sum of both pooling results with Dense (fully connected layer or linear regression) in (E), and processes it with a sigmoid function in (F), to obtain a C × 1 × 1 channel attention map shown in (G). The number of channels C can be the number of channels of the image data to be multiplied later in (I). By using a channel attention map that includes convolution with reference image data for the input block image data, position dependency can be eliminated. The same applies to the attention mechanisms in the second and third examples described below.

[0028] The attention mechanism 51 inputs image data (feature map) from the previous network 50 at (H) and performs convolution, linear processing, and pooling to generate image data (feature map of C×W×H). Attention mechanism 51 multiplies the channel attention map of (G) by the image data of (H) for each channel at (I), and obtains new image data (feature map of C×W×H) at (J). When no convolution or the like is performed on the image data from the preceding network 50, the attention mechanism 51 multiplies the image data (C×W×H) input from the network 50 by the Channel Attention map (G) for each channel in (I), and obtains new image data (feature map of C×W×H) in (J). The image data (J) is image data in which the features of the reference image data by Channel Attention are applied to weighted meaningful data in the image data (feature map) input from the network 50. The attention mechanism 51 outputs the new image data to the network 50 at the next stage.

[0029] 7 is a diagram illustrating the configuration of the second example of the attention mechanism (Spatial Attention). When Spatial Attention is used, as will be described later, processing that is almost the same as the Channel Attention described above is performed in the stage preceding Spatial Attention, where Spatial Attention processing is performed using feature signals for each channel obtained from the RGB data of the reference image data, and the result is multiplied by the data supplied from the network for each channel to obtain the processing result. In the example of FIG. 7, the attention mechanism 51 executes the following processes in the learning and inference processes of the generator 43A. The attention mechanism 51 obtains a Channel Attention map of the reference image data, similar to the first example in Fig. 6. That is, it performs the same processing as in Fig. 6 (B) to (F) on the RGB data (3 × W' × H') of the reference image data shown in (A) to obtain a Channel Attention map (C × 1 × 1) shown in (G). The number of channels C in the Channel Attention map can be the number of channels of the image data to be multiplied by (K) later.

[0030] The attention mechanism 51 inputs the RGB data of the original image (target image data) at (H), performs convolution on the RGB data of the original image at (I), and obtains new image data (C×W×H feature map) at (J). In (K), the attention mechanism 51 multiplies the channel attention map in (G) by the image data in (J) for each channel. The attention mechanism 51 performs pooling on the results of the multiplication at (L), performs convolution at (M), processes with a sigmoid function at (N), and obtains a spatial attention map (1 × W × H) for the original image at (O). The spatial attention map (second feature information) is a feature that indicates where an object is located in the image. The attention mechanism 51 inputs image data from the previous network 50 at (P) and performs convolution, linear processing, and pooling to generate image data (a feature map of C×W×H). Attention mechanism 51 multiplies the spatial attention map of (O) and the image data of (P) for each channel at (Q), and obtains new image data (a feature map of C×W×H) at (R).

[0031] When no convolution or the like is performed on the image data from the previous network 50, the attention mechanism 51 multiplies the image data (C×W×H) input from the network 50 by the Spatial Attention map (O) for each channel in (Q), and obtains new image data (feature map of C×W×H) in (R). The image data obtained in (R) is image data to which Channel Attention has been applied to areas where objects are present in the image data input from the network 50. The attention mechanism 51 outputs the new image data to the next network 50.

[0032] 8 is a diagram illustrating the configuration of a third example of an attention mechanism (Scaled Dot Product Attention). When using Scaled Dot Product Attention, the process before the Scaled Dot Product Attention is almost the same as that of the Channel Attention described above, and a feature signal for each channel is obtained from the RGB data of the reference image data. In the example of FIG. 8, the attention mechanism 51 executes the following processes in the learning and inference processes of the generator 43A. First, the attention mechanism 51 obtains a Channel Attention map of the reference image data, similar to the first and second examples in Figures 6 and 7. That is, the same processing as in Figures 6B to 6F is performed on the RGB data (3 x W' x H') of the reference image data shown in (A), and a Channel Attention map (C x 1 x 1) shown in (G) is obtained. The number of channels C in the Channel Attention map can be the number of channels of the image data to be multiplied by (K) later. The attention mechanism 51 inputs the RGB data of the original image (target image data) at (H), performs convolution on the RGB data of the original image at (I), and obtains new image data (C×W×H feature map) at (J). In (K), the attention mechanism 51 multiplies the channel attention map in (G) by the image data in (J) for each channel. The attention mechanism 51 performs convolution on the multiplication result at (L), processes it with a sigmoid function at (M), and obtains a C×W×H feature map at (N).

[0033] Attention mechanism 51 inputs image data from the previous network 50 at (O) and performs convolution, linear processing, and pooling to generate image data (a feature map of C×W×H). Attention mechanism 51 generates Query, Key, and Value from the image data of (O) at (P), inputs the C×W×H feature map obtained at (N) into Query, further multiplies the multiplication result of Query and Key by Value, and obtains new image data (a feature map of C×W×H) at (Q). In the attention mechanism, a query is a value that represents the data to be searched from the input data. A key is a value used to measure the proximity between the query and the data to be searched. A value is the data of the search result based on the key. By performing the processing in (P), new image data (feature map of C×W×H) can be obtained in (Q) by taking into account the properties of each location in the original image (target image data) and the weighting in the image data input from the network 50.

[0034] When convolution or the like is not performed on the image data from the network 50, the attention mechanism 51 performs processing (P) using the image data input from the network 50 and the C×W×H feature map data of (N), and obtains new image data (C×W×H feature map) in (Q). The image data of (Q) is image data that has been weighted (meaningful) in the image data input from the network 50, and has the Channel Attention map applied to areas where objects are present. The attention mechanism 51 outputs the new image data to the next-stage network 50.

[0035] Similarly, in the second embodiment described below, when the entire target image data or a specific region of the target image data (the region or block with the most complex pixels) is used as reference image data, the Channel Attention map based on the reference image data will always have the same value. Therefore, when performing super-resolution processing, after extracting a Channel Attention map based on reference image data for the first input block image data, the same Channel Attention map can be used for subsequent block image data. When the entire target image data is used as reference image data, a relatively large load and long processing time are required to calculate a Channel Attention map based on the entire target image data for processing the first block. However, since the Channel Attention map calculated once can be reused, no load or processing time is required for the remaining blocks. Furthermore, high image quality can be expected by referring to the characteristics of the entire target image data.

[0036] When a specific block of target image data is used as reference image data, a certain load and processing time is incurred to extract that specific block. However, a Channel Attention map that references the features of the entire target image data can be calculated for one block with low load and in a short time. Once calculated, the Channel Attention map can be reused, so there is no load or processing time for the remaining blocks. Furthermore, it is possible to train a generator that better reflects the user's intentions. Furthermore, using neighboring blocks including the target block as reference image data makes it easier to reference the characteristics of the target image data than when only the target block is referenced. However, because a different region is used as reference image data for each target block, a Channel Attention map must be calculated each time a block is processed. However, the Channel Attention map can be calculated each time with low load and short processing time based on relatively small reference image data. It is desirable to appropriately select which part or all of the target image data to use as reference image data based on the structure and size of the target image data, the desired image quality and learning intent, and the performance of the image processing device. In either case, it is possible to achieve significantly higher image quality than conventional self-attention, which uses the target block itself as reference image data. Note that any of the attention mechanisms shown in Figures 6 to 8 may be used for each attention mechanism 51 in Figure 5. That is, different attention mechanisms may be used for attention mechanisms 51A to 51C, or the same attention mechanism may be used for all of them, and the combinations of the attention mechanisms shown in Figures 6 to 8 are arbitrary. Furthermore, during learning by the generator 43A shown in Figure 5, the output block image data is compared with the corresponding training data (not shown), and parameters for each network 50 and each attention mechanism 51 are set.

[0037] FIG. 9 is a diagram illustrating the configuration of a generator in a second example of this embodiment. The generator 43B shown in Figure 9 has a configuration in which processing units 55 (55A, 55B, 55C) each having a residual block processing unit 60 that processes residual blocks (Resblock), a channel attention processing unit 61 that processes channel attention, and a spatial attention processing unit 62 that processes spatial attention are connected in series in multiple stages. The final-stage processing unit 55C further includes a convolutional layer (Conv) 64 for generating RGB data based on Spatial Attention. Image data generated based on reference image data is output from the convolutional layer 64A, and image data generated based on input block image data is output from the convolutional layer 64B. The image data generated based on the reference image data is compared with training data for the reference image, while the output block image data generated based on the input block image data is compared with training data for the block image, and various parameters of each processing unit 55 are set. Although the number of processing units 55 is not limited to the example of Figure 9, the generator 43B includes one or more processing units that process residual blocks, channel attention, and spatial attention, as well as a final-stage processing unit that processes residual blocks, channel attention, spatial attention, and includes a convolutional layer. The processing unit 55 shown in FIG. 9 applies a mechanism called a Convolutional Block Attention Module (CBAM) that serially connects the processing of Channel Attention and the processing of Spatial Attention.

[0038] The generator 43B receives input block image data obtained by dividing the target image data and reference image data, performs super-resolution processing on the input block image data, and outputs output block image data. The reference image data is image data that represents the structure and characteristics of the target image data based on the target image data. Details will be described later using Figures 13 and 14. The input block image data and the reference image data may be different sizes. For example, the entire target image data can be used as reference image data. Processing unit 55A, which serves as the input stage and receives input block image data and reference image data, processes residual blocks, channel attention, and spatial attention on these data, and outputs the processed data (spatial attention) to the next-stage processing unit 55B. The processing unit 55B has the same configuration as the processing unit 55A, performs the same processing on the input data from the processing unit 55A, and outputs the data to the processing unit at the subsequent stage. The generator 43B finally outputs the output of the processing unit 55 (here, processing unit 55C) which is the output stage as output block image data. The output block image data are combined to obtain the final output image data.

[0039] As with the generator 43A, the generator 43B applies a channel attention map calculated from reference image data based on the target image data to the input block image data, thereby reflecting the characteristics of the target image data in the input block image data. Therefore, the image processing device 1 can increase the resolution of the target image data while maintaining high image quality as the original image. For example, the target image data is increased in resolution without sharpening blurred backgrounds or the like that existed in the target image data, and the non-blurred portions are left as they are.

[0040] FIG. 10 is a diagram illustrating the processing (residual block and channel attention) performed by the processing unit of the generator according to the second embodiment. To calculate the Channel Attention Map, the processing unit 55 aggregates spatial information from the image data (feature map) output by the residual block, and by simultaneously using both average pooling and max pooling, the expressive power of the network can be significantly improved. The attention mechanism also transfers the aggregated results from average pooling and max pooling to a multilayer perceptron (MLP) with one hidden layer, and merges the aggregated results from the MLP using element-wise summation to generate a one-dimensional Channel Attention Map. The processing unit 55 receives the image data (feature map) of the reference image data output from the residual block in (A). The processing unit 55 performs average pooling and max pooling in parallel on this feature map at (B), inputs both pooling results to the MLP at (C), and obtains the processing result at (D), which is an element of both pooling results. The processing unit 55 adds the processing results of (D) in (E) and processes them with a sigmoid function in (F) to obtain the Channel Attention map shown in (G). The processing unit 55 multiplies the feature map of (A) and the channel attention map obtained in (G) for each channel in (H). The processing unit 55 outputs image data (multiplication result) obtained by applying the Channel Attention map (G) to the feature map (A) to the processing of FIG. 11 for calculating the Spatial Attention map.

[0041] Next, the processing unit 55 inputs the image data (feature map) of the input block image data output from the residual block in (I). The processing unit 55 multiplies the feature map of (I) and the Channel Attention map obtained in (G) for each channel in (J). The processing unit 55 outputs image data (multiplication result) obtained by applying the Channel Attention map (G) to the feature map (I) to the processing of FIG. 11 for calculating the Spatial Attention map.

[0042] FIG. 11 is a diagram illustrating the processing (Spatial Attention) performed by the processing unit of the generator of the second embodiment. To calculate the spatial attention map, processing unit 55 performs average pooling and max pooling along the channel axis and concatenates the results to generate a descriptor. Pooling can be applied to emphasize regions with high information content. The attention mechanism applies a convolutional layer to the concatenated descriptor to generate a two-dimensional spatial attention map that encodes the locations to be emphasized or suppressed. At (A), the processing unit 55 inputs the reference image data of the residual block output obtained at (J) in FIG. The processing unit 55 performs average pooling and max pooling serially on the input image data at (B), performs convolution at (C), and processes it with a sigmoid function at (D) to obtain a Spatial Attention map of the input block image data shown at (E). 10. At (A), the processing unit 55 receives the input block image data of the residual block output obtained at (H) in FIG. The processing unit 55 performs the processes (B) to (D) on the input image data, and obtains the Spatial Attention map of the reference image data shown in (E). As mentioned above, only the final stage processing unit (55C) performs convolution on the Spatial Attention map of the input block image data and the Spatial Attention map of the reference image data, respectively, generated in Figure 11 (E), to generate RGB data of super-resolution processed input block image data and reference image data (output block image data).

[0043] The reference image data used in this embodiment will be described below. As described above, in the image processing device 1 of this embodiment, the target image data is divided into blocks, and super-resolution processing is performed for each block of image data. In this case, the following problems arise. FIG. 12 is a diagram illustrating the problems that arise when super-resolution processing is performed for each block. In the target image data conceptually shown in Figure 12, when block image data B1 is the block to be processed, block image data B1 contains only blurred portions due to depth of field. Block image data B1 does not contain information to be referenced, such as clear (unblurred) portions. Therefore, if only block image data B1 is used as reference image data to calculate a Channel Attention map and apply the Channel Attention map to the block image (i.e., self-attention), a super-resolution image that reflects the characteristics of the target image data for this block cannot be obtained. In contrast, adjacent block image data B2 contains clear (unblurred) portions that serve as reference information.

[0044] In order to be able to calculate a Channel Attention Map including information about block image data B2 and apply the Channel Attention Map to the block image, the image processing device 1 of this embodiment inputs the surroundings of the block image data to be processed, the entire target image data, and specific locations of the target image data (locations where the image is thought to be the most complex and least blurred) as reference image data. For example, if the block to be processed is block image data B1, the area including the surrounding block image data B2, B3, and B4, or the block image data B2 with the most complex image, or the entire target image data in Figure 12, is input to the attention mechanism as reference image data. By incorporating the characteristics of the surrounding area of ​​the target block and the overall target image data into the processing of the target image data, blocks that reflect the characteristics of the target image data can be included in the reference image data, increasing the possibility of obtaining a more appropriate channel attention map. This makes it easier to approach the target image than conventional self-attention results, and can improve the quality of super-resolution of images that include blurred areas.

[0045] It should be noted that, since the purpose of the blocks surrounding the target block is to include in the reference image data blocks that better reflect the characteristics of the target image data, it is desirable that the blocks surrounding the target block be areas within the target image data that include the area of ​​the target image data. Areas on the front, back, left, and right of the target block may be predetermined, but in that case, areas outside the range of the target image data will be included in the reference image data, reducing the area within the range, which is a problem in that it reduces the possibility of including blocks that reflect the characteristics of the target image data in the reference image data.

[0046] 13 and 14 are diagrams showing the relationship between block division of target image data and reference image data. Figure 13 shows the target image data. Although the image shown is a monochrome image, it is actually a color image containing RGB data. The target image data is divided into blocks of, for example, 256 x 256 pixels, and super-resolution processing is performed on each divided block. FIG. 14(a) shows block image data B5 obtained by dividing the upper left block of the target image data in FIG. The reference image data input to the generator 43 (43A, 43B) together with the block image data B5 can be the entire target image data (a-1), the image data B7 (Figure 13) surrounding the block image data B5 (a-2), or the image data B9 (Figure 13) of the most complicated part (a-3). Figure 14(b) shows block image data B6 obtained by dividing the central block of the target image data in Figure 13. In this case, the reference image data input to the generator 43 (43A, 43B) together with block image data B5 can be image data B8 (Figure 13) surrounding block image data B5 (b-1). As in the case of Figure 13, the entire target image data (a-1) or image data B9 (Figure 13) of the most complicated part (a-3) can also be used.

[0047] By using the entire target image data as reference image data, it is possible to perform super-resolution processing that reflects the characteristics of the entire target image data. By using image data B7 and B8 surrounding the block image data as reference image data, super-resolution processing can be performed while reflecting the characteristics of the surrounding area of ​​the block to be processed. For example, image data with sides approximately twice the size of the block image data (four times the area) can be used as reference image data. The reference image data can be extracted from anywhere around the block image data, and the size of the reference image data can be determined appropriately.

[0048] Figure 15 illustrates a method for identifying the most complex part of an image. This can be achieved using known image processing techniques. For example, a gradient filter or edge detection filter can be used to roughly identify the complex part of the image. When the image processing device 1 applies Laplacian filtering to the target image data shown in Fig. 15(a), it obtains a filtered image as shown in Fig. 15(b). The image processing device 1 cuts out the target image data in Fig. 15(a) as reference image data at a position corresponding to an area with a lot of white (edge ​​areas) (areas where large values ​​are detected) in the filtered image in Fig. 15(b). In the image after filtering, areas with a lot of white are areas with a large change in pixels. For example, the cutting position should be around the pixel with the most white (the pixel with the largest change, the value obtained by the Laplacian filter is the largest). As a method for identifying the most complicated part of the image after filtering, only the vicinity of pixels where large values ​​are detected can be identified by a specific size, as shown in Figure 15(c). Alternatively, as shown in Figure 15(d), the image in Figure 15(b) can be divided into blocks, and the block containing the largest value can be identified as the most complicated part of the image (reference image data). Alternatively, the block with the highest total value in the block can be identified as the specific location (reference image data).

[0049] An example of a procedure for training the generator of this embodiment will be described. This is merely an example, and the generator may be trained using other procedures. The generator 43 (43A, 43B) performs learning using teacher image data and reference image data. The teacher image data is, for example, high-resolution image data. In this embodiment, when the generator learns, for example, an image obtained by reducing the teacher image data is used as target image data, and image generation processing is performed using the reference image data described below, and the obtained super-resolution image is compared with the teacher image data. The image processing device repeatedly performs image generation while adjusting the parameters constituting the machine learning model until there is no loss (difference) between the super-resolution image and the teacher image data. The parameters learned by the generator 43A are all parameters of all convolution layers included in the multiple neural networks 50 (50A, 50B, ...), all parameters of all convolution layers included in the multiple attention mechanisms 51 (51A, 51B, ...), and all parameters of Dense. The parameters learned by the generator 43B are the MLP in one Channel Attention processing unit 61 included in the multiple processing units 55 (55A, 55B, ...) and all parameters of the convolutional layers included in the two Spatial Attention processing units 62. The generator 43B also learns parameters of the convolutional layer 64 provided in the processing unit 55C at the output stage. In this embodiment, image generation (inference processing) is performed in units of blocks obtained by dividing target image data, and therefore, comparison of super-resolution images with teacher image data is also performed in units of blocks when training a machine learning model. The image processing device 1 performs super-resolution processing using the target image data divided into blocks and reference image data, and compares the result with teacher image data that has also been divided in the same way. The image processing device 1 learns parameters so that loss is zero for all blocks.

[0050] FIG. 16 is a flowchart illustrating the learning process of the generator in this embodiment. In step S101, the image processing device 1 (learning process execution unit 31) reduces the teacher image data to generate reduced teacher image data. In step S102, the image processing device 1 (learning process execution unit 31) divides the reduced teacher image data into blocks to generate block teacher image data. In step S103, the image processing device 1 (learning process execution unit 31) inputs to the generator 43 the block teacher image data and the reference image data based on the reduced teacher image data. In step S104, the image processing device 1 (learning process execution unit 31) generates image data based on the input image data. In step S105, the image processing device 1 (learning process executing unit 31) determines whether image data has been generated for all block teacher image data.

[0051] If unprocessed block teacher image data remains (No in step S105), the image processing device 1 (learning process execution unit 31) returns to step S103, inputs the next block teacher image data and reference image data to the generator 43, and generates image data in step S104. If unprocessed block teacher image data does not remain (Yes in step S105), the image processing device 1 (learning process execution unit 31) determines in step S106 whether the generated image data matches all the teacher image data. This may be determined based on whether the classifier described above determines that the image data is true. If it is determined that there is no match (No in step S106), the image processing device 1 (learning process execution unit 31) changes the NN parameters of the generator 43 in step S108, repeats the processes of steps S103 to S105 to generate image data, and again determines whether the image data match in step S106. If it is determined that they match (Yes in step S106), the image processing device 1 (learning process execution unit 31) ends the learning process of the generator 43 in step S107. In the case of the generator 43B, learning is also performed so that the super-resolution image (output from the convolution layer 64A) generated based on the reference image data matches the portion of the teacher image data that corresponds to the reference image data. The training of the generator using the block teacher image data as described above can be performed using the entire teacher image data, the most complicated part, and the blocks surrounding the divided blocks as reference image data. When the entire teacher image data is trained as reference image data, the entire target image data is used as reference image data during image generation (inference), and when the most complicated part is trained as reference image data, the most complicated part of the target image data is used as reference image data during image generation (inference).

[0052] FIG. 17 is a flowchart illustrating the image generation process in this embodiment. In step S201, the image processing device 1 (image processing execution unit 32) causes the input unit 41 to acquire target image data from the storage unit 12 or the like. In step S202, the image processing device 1 (image processing execution unit 32) causes the dividing unit 42 to divide the acquired target image data into block image data. In step S203, the image processing device 1 (image processing execution unit 32) inputs the input block image data and the reference image data to the generator 43. In step S204, the image processing device 1 (image processing execution unit 32) causes the generator 43 to generate output block image data based on the input block image data and the reference image data. In step S205, the image processing device 1 (image processing execution unit 32) determines whether output block image data has been generated for all input block image data. If unprocessed input block image data remains (No in step S205), the image processing device 1 (image processing execution unit 32) returns to step S203, inputs the next input block image data and reference image data to the generator 43, and generates output block image data in step S204. If there is no unprocessed input block image data remaining (Yes in step S205), the image processing device 1 (image processing execution unit 32) in step S206 causes the combining unit 44 to combine the generated output block image data to obtain output image data. The image processing device 1 (image processing execution unit 32) causes the output unit 45 to output the output image data to the storage unit 12 or the like, and ends the current image generation process.

[0053] As described above, when the image processing device 1 of this embodiment performs super-resolution processing on each block into which target image data is divided and then combines the blocks to obtain a super-resolution image, it calculates a Channel Attention map using blocks surrounding the target block to be processed, the entire target image data, a specific part of the target image data (the part with the most complex pixels), etc. as reference image data, and generates an image by applying the Channel Attention map to the target block to be processed. As a result, an image is generated that reflects information on how blurred the target image data is as a whole, so the image processing device 1 can generate an image that reflects information on how blurred the target image data is as a whole, solving the problem of generating an image that differs from the target image data and improving the quality of the super-resolution image.

[0054] FIG. 18 is a block diagram illustrating an embodiment of a computer system. The configuration of the computer device 100 will be described with reference to FIG. The computer device 100 is, for example, an image processing device that processes various types of information, and includes a control circuit 101, a storage device 102, a reading / writing device 103, a recording medium 104, a communication interface 105, an input / output interface 106, an input device 107, and a display device 109. The communication interface 105 is connected to a network 200. The components are connected to each other via a bus 110. The image processing device 1 can be configured by appropriately selecting some or all of the components described in the computer device 100.

[0055] The control circuit 101 controls the entire computer device 100 . The control circuit 101 is a processor such as a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The control circuit 101 functions as, for example, the control unit 10 in FIG. The image control circuit 108 controls image processing in the computer device 100 . The image control circuit 108 is a graphic processing unit (GPU) etc. The image control circuit 108 functions as the image processing unit 11 in FIG.

[0056] The storage device 102 stores various data. The storage device 102 is, for example, a memory such as a read-only memory (ROM) or a random access memory (RAM), a hard disk (HD), or a solid state drive (SSD). The storage device 102 may store an image processing program that causes the control circuit 101 to function as the control unit 10 in FIG. 1. The storage device 102 functions as, for example, the storage unit 12 in FIG. 1.

[0057] When performing image processing, the image processing device 1 reads out the program stored in the storage device 102 into the RAM. The image processing device 1 executes the program read into the RAM by the control circuit 101, thereby executing processing including one or more of a learning processing execution unit and an image processing execution unit. The program may be stored in a storage device of a server on the network 200 as long as the control circuit 101 can access it via the communication interface 105 .

[0058] The reading / writing device 103 is controlled by the control circuit 101 and reads / writes data from / to a removable recording medium 104 . The recording medium 104 stores various data. For example, the recording medium 104 stores an image processing program. The recording medium 104 is, for example, a non-volatile memory (non-transitory recording medium) such as a Secure Digital (SD) memory card, a Floppy Disk (FD), a Compact Disc (CD), a Digital Versatile Disk (DVD), a Blu-ray (registered trademark) Disk (BD), or a flash memory.

[0059] The communication interface 105 communicably connects the computer device 100 to other devices via the network 200. The communication interface 105 functions as, for example, the communication unit 13 in FIG. The input / output interface 106 is an interface that is detachably connected to, for example, various input devices. The input device 107 connected to the input / output interface 106 includes, for example, a keyboard and a mouse. The input / output interface 106 communicatively connects the various connected input devices to the computer apparatus 100. The input / output interface 106 outputs signals input from the various connected input devices to the control circuit 101 via the bus 110. The input / output interface 106 also outputs signals output from the control circuit 101 to the input / output devices via the bus 110. The input / output interface 106 and the input device 107 function as, for example, the operation unit 15 in FIG. 1 .

[0060] Display device 109 displays various types of information. Examples of display device 109 include a CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), PDP (Plasma Display Panel), and OELD (Organic Electroluminescence Display). Network 200 is, for example, a LAN, wireless communication, a P2P network, or the Internet, and communicatively connects computer device 100 to other devices. This embodiment is not limited to the embodiment described above, and various configurations or embodiments can be adopted within the scope of the gist of this embodiment. [Explanation of symbols]

[0061] 1 Image processing device, 2 Recording medium, 10 Control unit, 11 Image processing unit, 12 Memory unit, 13 Communication unit, 14 Display unit, 15 Operation unit, 16 Reading unit, 31 Learning processing execution unit, 32 Image processing execution unit, 41 Segmentation unit, 42 Input unit, 43 (43A, 43B) Generator, 44 Output unit, 45 Combination unit, 50 (50A, 50B, 50C) Network, 51 (51A, 51B) Attention mechanism, 55 (55A, 55B, 55C, 55D) Processing unit, 21L, 121L Machine learning library, 1P, 2P Image processing program

Claims

1. an inference unit that performs inference using a machine learning model using processing target data based on target image data and reference image data based on the target image data as input, and generates generated image data based on the processing target data; The image processing device is characterized in that, in the inference, the inference unit calculates first feature information representing the characteristics of the target image data using the reference image data, and generates the generated image data that reflects the first feature information by applying the first feature information to the processing target data input to the machine learning model.

2. 2. The image processing device according to claim 1, a dividing unit that divides the target image data into blocks as the processing target data; a combining unit that combines the generated image data generated by the inference unit based on the blocks to generate output image data.

3. 3. The image processing device according to claim 2, 10. An image processing apparatus, wherein the reference image data is image data of a peripheral block including the block to be processed.

4. 3. The image processing device according to claim 2, 10. An image processing apparatus according to claim 9, wherein the reference image data is the entire target image data.

5. 3. The image processing device according to claim 2, 10. An image processing apparatus according to claim 9, wherein the reference image data is image data of a specific block in the target image data that satisfies a predetermined condition.

6. 6. The image processing device according to claim 5, The image processing device according to claim 1, wherein the specific block includes an area in which a numerical value is high in the target image data after filtering.

7. 6. The image processing device according to claim 5, The image processing device according to claim 1, wherein the specific block includes an area in which the numerical value of the target image data after filtering is low.

8. 2. The image processing device according to claim 1, The image processing device is characterized in that the inference unit generates the generated image data by applying the first feature information to an intermediate output of inference on data based on processing target data input to the machine learning model.

9. 9. The image processing device according to claim 8, The image processing device is characterized in that the inference unit generates the generated image data by applying second feature information indicating the weighting of areas in the original processing target data, obtained by applying the first feature information to the intermediate output of inference for the original processing target data, to the intermediate output of inference for data based on the processing target data input to the machine learning model.

10. 9. The image processing device according to claim 8, The image processing device is characterized in that the inference unit generates the generated image data by applying third feature information indicating the position of an object in the original processing target data based on second feature information indicating the weighting of areas in the original processing target data obtained by applying the first feature information to an intermediate output of inference for data based on the processing target data input to the machine learning model.

11. 1. A processor-implemented image processing method, comprising: The processor: performing inference using a machine learning model with processing target data based on the target image data and reference image data based on the target image data as input, and generating generated image data based on the processing target data; In the inference, first feature information representing features of the target image data is calculated using the reference image data, and the generated image data reflecting the first feature information is generated by applying the first feature information to the processing target data input to the machine learning model. An image processing method comprising:

12. An image processing program that causes a processor to execute an image processing method, The processor: performing inference using a machine learning model with processing target data based on the target image data and reference image data based on the target image data as input, and generating generated image data based on the processing target data; In the inference, first feature information representing features of the target image data is calculated using the reference image data, and the generated image data reflecting the first feature information is generated by applying the first feature information to the processing target data input to the machine learning model.

1. An image processing program comprising:

Images