Image processing device, image processing method, and image processing program

Abstract

An image processing device according to an embodiment of the present invention includes a first processing unit, a decomposition unit, a second processing unit, and a synthesis unit. The first processing unit executes super-resolution processing on a first image, and generates a second image by increasing the resolution of the first image. The decomposition unit extracts a component image including a part of frequency components and shape information included in the first image. The second processing unit inputs the component image to a trained model that executes super-resolution processing, and generates a synthesis component image by increasing the resolution of the component image. The synthesis unit synthesizes the second image and the synthesis component image to generate a third image.

Description

Image processing device, image processing method, and image processing program

[0001] The embodiments relate to an image processing device, an image processing method, and an image processing program.

[0002] Super-resolution technology generates high-resolution images with enhanced spatial resolution using models trained by mathematical processing or deep learning on low-resolution images. However, images generated by super-resolution suffer from degradation in image quality compared to the target image, such as blurred image contours and noise in areas where brightness or color changes smoothly. To reduce these degradations, deep learning-based super-resolution processing employs a configuration that extracts image features contained in the low-resolution image using a differential operator or incorporates frequency domain components obtained by fast Fourier transform into a loss function (see, for example, Non-Patent Document 1).

[0003] Shang, Shuyao et al., "ResDiff: Combining CNN and Diffusion Model for Image Super-Resolution." [online], February 2, 2024, arXiv, [Retrieved May 1, 2024], Internet <URL: https: / / doi.org / 10.48550 / arXiv.2303.08714>

[0004] However, in the method using the differential operator or the fast Fourier transform, it is necessary to perform calculations using neighboring pixels or the entire image for the pixel of interest, which results in a problem of increased memory access and calculation volume.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing device, an image processing method, and an image processing program that can reduce the number of memory accesses and the amount of calculations.

[0006] An image processing device according to an embodiment includes a first processing unit, a decomposition unit, a second processing unit, and a synthesis unit. The first processing unit performs super-resolution processing on a first image to generate a second image by increasing the resolution of the first image. The decomposition unit extracts component images that include some of the shape information and frequency components contained in the first image. The second processing unit inputs the component images to a trained model that performs super-resolution processing, and generates component images for synthesis by increasing the resolution of the component images. The synthesis unit synthesizes the second image and the component images for synthesis to generate a third image.

[0007] According to the embodiments, it is possible to provide an image processing device, an image processing method, and an image processing program that can reduce the number of memory accesses and the amount of calculations.

[0008] FIG. 1 is a block diagram showing an image processing device according to a first embodiment. FIG. 2 is a flowchart showing an example of inference processing of the image processing device according to the first embodiment. FIG. 3 is a block diagram showing an image processing device according to a second embodiment. FIG. 4 is a flowchart showing an example of learning processing of the image processing device according to the second embodiment. FIG. 5 is a flowchart showing another example of learning processing of the image processing device according to the second embodiment. FIG. 6 is a block diagram showing the hardware configuration of the image processing device according to this embodiment.

[0009] Each embodiment will be described below with reference to the drawings. Each embodiment illustrates an apparatus or method for embodying the technical idea of ​​the invention. The drawings are schematic or conceptual. Hereinafter, the same reference numerals are used to designate components having substantially the same functions and configurations. The numbers following the letters that make up the reference numerals are used to distinguish between elements that are referred to by the reference numerals containing the same letters and that have similar configurations. When there is no need to distinguish between elements indicated by reference numerals containing the same letters or numbers, these elements will be referred to by reference numerals containing only letters or numbers.

[0010] First Embodiment In the first embodiment, an inference process of an image processing device 1 will be described. Fig. 1 is a block diagram of the image processing device 1 according to the first embodiment. The image processing device 1 according to the first embodiment includes an acquisition unit 11, a first processing unit 12, a decomposition unit 13, a second processing unit 14, a synthesis unit 15, and a storage unit 16.

[0011] The acquisition unit 11 acquires a low-resolution image (also referred to as a first image) to be processed. The first processing unit 12 performs super-resolution processing on the low-resolution image to generate an enlarged image (also referred to as a second image) in which the low-resolution image has been increased in resolution. The decomposition unit 13 extracts component images containing some of the shape information and frequency components contained in the low-resolution image. Specifically, the component images are assumed to be bit planes (images), and a masking process is performed to decompose the low-resolution image into bit planes, and lower-order bit planes related to some of the bits in the low-resolution image are extracted. A bit plane indicates, for example, an image decomposed into binary images each containing one bit of information for each pixel. In other words, in the case of a multilevel image consisting of n bits, n binary images corresponding to each bit can be generated as bit planes. The following describes the case where the component images are lower-order bit planes, but any image may be used as the component image as long as some components (frequency components or edge information) of the low-resolution image are extracted.

[0012] The second processing unit 14 inputs lower bit planes to the trained model that performs super-resolution processing, and generates a synthesis bit plane by increasing the resolution of the lower bit planes. The synthesis unit 15 synthesizes the enlarged image and the synthesis bit plane to generate a high-resolution image (also referred to as a third image). The storage unit 16 stores low-resolution images, enlarged images, trained models, high-resolution images, etc. The trained models may not be stored in the storage unit 16, but may be saved on an external server, etc. The second processing unit 14 may access the external server when it needs to use the trained model.

[0013] Next, an example of the inference processing of the image processing device 1 according to the first embodiment will be described with reference to the data processing flow of FIG.

[0014] In step SA1, the acquisition unit 11 acquires a low-resolution image to be processed. In step SA2, the first processing unit 12 performs super-resolution processing on the low-resolution image to generate an enlarged image. The super-resolution processing is assumed to be a general super-resolution processing, and may be super-resolution processing using mathematical operations such as bilinear interpolation, bicubic interpolation, or Lanczos interpolation. Alternatively, dictionary-based super-resolution processing may be used, in which a dictionary of pairs of high-resolution patch images and low-resolution patch images is referenced, and the high-resolution patch image paired with the low-resolution patch image most similar to a subregion of the low-resolution image is combined to generate a high-resolution image. Alternatively, super-resolution processing may be performed using a trained model based on a neural network, such as a convolutional neural network (CNN), that is trained to estimate a high-resolution image from a low-resolution image. As such, the basic super-resolution processing shown in step SA2 may be performed using mathematical operations or existing deep learning processes, and the image processing device 1 has the practical advantage of being highly versatile.

[0015] In step SA3, the decomposition unit 13 performs a mask process on the low-resolution image to decompose the low-resolution image into bit planes and extract the lower bit planes (lower bit plane images). The decomposition of the low-resolution image into bit planes can be performed by fixing a mask value according to the extraction conditions. For example, when extracting one bit plane from an 8-bit low-resolution image, the low-resolution image is decomposed into bit planes by fixing a mask value of 1 (00000001 （２） ), 2(00000010 （２） ), 4 (00000100 （２） ), 8 (00001000 （２） ) to extract a bit plane for the desired lower bits. Note that the values ​​in parentheses are binary notations. The masking process can be performed by performing a bit operation to AND each pixel of the low-resolution image with the mask value.

[0016] Alternatively, multiple lower bit planes may be extracted from the low-resolution image, for example, 1, 3 (00000011 （２） ), 7 (00000111 （２） For example, bit planes such as 3 (00000011) may be extracted. （２） ), a combination of the least significant bit plane and the second least significant bit plane can be used. （２） ), the first to third least significant bit planes can be used.

[0017] In step SA4, the second processing unit 14 generates a bit plane for synthesis by inputting the lower bit plane into a trained model that performs super-resolution processing and that has been trained to estimate a high-resolution image from a low-resolution image. The trained model that performs super-resolution processing in step SA4 processes the super-resolution image so that the enlargement ratio is the same as that of the enlarged image. In other words, the enlarged image and the bit plane for synthesis have the same image size.

[0018] In step SA5, the synthesis unit 15 synthesizes the enlarged image with the synthesis bit plane to generate a high-resolution image in which the low-resolution image is increased in resolution. The generated high-resolution image may be stored in the storage unit 16. By using the characteristics contained in the lower bit planes that tend to contain high-frequency components, the synthesis bit planes, which are obtained by performing super-resolution processing on the lower bit planes, are synthesized with the enlarged image, thereby achieving high resolution.

[0019] According to the first embodiment described above, the first processing unit performs super-resolution processing on the low-resolution image to generate an enlarged image having a higher resolution than the first resolution. The decomposition unit decomposes the low-resolution image into bit planes by masking the image and extracts the lower-order bit planes. The second processing unit inputs the lower-order bit planes to the trained model and generates synthesis bit planes by increasing the resolution of the lower-order bit planes. The synthesis unit generates a high-resolution image by synthesizing the enlarged image and the synthesis bit planes.

[0020] As a result, when extracting feature data from a low-resolution image using masking, super-resolution processing can be performed while significantly reducing the amount of calculation and the number of memory accesses. For example, for an image having a pixel count of width H and height V, the order of calculation required for a conventional fast Fourier transform is O(W×H×log(W×H)). On the other hand, with the image processing device according to the first embodiment, the amount of calculation can be significantly reduced to O(W×H) through masking. Furthermore, with regard to memory access, a two-dimensional differential operator requires W×H×9 memory accesses (the target pixel and its eight neighboring pixels). On the other hand, with the image processing device according to the first embodiment, masking reduces the number of memory accesses to W×H, i.e., only the target pixel, and furthermore, parallel processing on a pixel-by-pixel basis can be performed in a GPU (Graphics Processing Unit), significantly reducing the number of memory accesses.

[0021] Second Embodiment In the second embodiment, a learning process of the image processing device 1 will be described. The image processing device 1 according to the second embodiment will be described with reference to the block diagram of Fig. 3. The image processing device 1 according to the second embodiment includes an acquisition unit 21, a first processing unit 22, a decomposition unit 23, a second processing unit 24, a synthesis unit 25, a training unit 26, and a storage unit 27.

[0022] The acquisition unit 21 acquires learning data. The learning data is image data that pairs a low-resolution image with a high-resolution image (also referred to as a correct image). A method for generating learning data is, for example, to generate a low-resolution image by performing a resolution reduction process on the image, such as downsampling the high-resolution image, and then set the low-resolution image and the high-resolution image as a pair. Alternatively, a pair of a low-resolution image and a high-resolution image may be generated by capturing images of the same subject using cameras with different resolutions. The first processing unit 22 performs super-resolution processing on the low-resolution image to generate an enlarged image in which the low-resolution image has been increased in resolution.

[0023] The decomposition unit 23 performs masking to decompose the low-resolution image into bit planes and extracts the lower-order bit planes. The second processing unit 24 inputs the lower-order bit planes to a machine learning model and generates synthesis bit planes by performing super-resolution processing on the lower-order bit planes. The synthesis unit 25 synthesizes the enlarged image and the synthesis bit planes to generate an estimated image (also referred to as a third image).

[0024] The training unit 26 calculates the difference between the estimated image generated by the synthesis unit 25 and the correct image included in the training data. The training unit 26 trains a machine learning model based on the difference and generates a trained model. The storage unit 27 stores the training data and the generated trained model.

[0025] Next, an example of a learning process performed by the image processing device 1 according to the second embodiment will be described with reference to the flowchart of FIG.

[0026] In step SB1, the acquisition unit 21 acquires pairs of low-resolution and high-resolution images as learning data.

[0027] In step SB2, similarly to step SA2, the first processing unit 22 performs super-resolution processing on the low-resolution image to generate an enlarged image.

[0028] In step SB3, similarly to step SA3, the decomposing unit 23 performs mask processing on the low-resolution image to decompose the low-resolution image into bit planes and extract the lower bit planes.

[0029] In step SB4, the second processing unit 24 estimates a super-resolution processed bit plane for synthesis by inputting the lower bit plane into a machine learning model. The type of machine learning model may be, for example, a deep learning model based on a DCNN (Deep Convolutional Neural Network) or a GAN (Generative Adversarial Network). Note that the machine learning model is not limited to these, and any other machine learning model capable of performing super-resolution processing may be used.

[0030] In step SB5, the synthesis unit 25 synthesizes the enlarged image with the synthesis bit plane to generate an estimated image, which is a high-resolution image estimated for the low-resolution image.

[0031] In step SB6, the training unit 26 calculates the difference between the estimated image and the ground truth image, which is the training data, and calculates a loss value using a loss function that uses the difference. As the loss function, for example, mean square error (MSE) or mean absolute error (MAE) is assumed, but any loss function generally used in machine learning may be used.

[0032] In step SB7, the training unit 26 determines whether the learning of the machine learning model has finished. For example, it may be determined that the learning of the machine learning model has finished when the loss value calculated by the loss function is equal to or less than a threshold value. Alternatively, it may be determined that the learning of the machine learning model has finished when learning has been repeated a predetermined number of epochs. When the learning is finished, a trained model is generated. Note that the learning method of the machine learning model is not limited to the above method, and a general supervised learning method may be used. If the learning of the machine learning model has finished, the training unit 26 proceeds to step SB8, and if the learning of the machine learning model has not finished, the training unit 26 proceeds to step SB9.

[0033] In step SB8, the storage unit 27 stores the trained model. The trained model may be output to an external device.

[0034] In step SB9, the training unit 26 updates parameters such as the weights and biases of the machine learning model so as to minimize the loss value calculated in step SB6. Then, the process returns to step SB1, and the same processing is repeated until the learning is completed.

[0035] In the learning process shown in Figure 4, an example is shown of learning the machine learning model used in step SB4, but it is also possible to simultaneously learn a machine learning model that performs the super-resolution processing of the first processing unit 22 shown in step SB2.

[0036] In this case, for example, in step SB6, the training unit 26 sets the sum of the loss functions of the machine learning model used by the first processing unit 22 and the machine learning model used by the second processing unit 24 as the overall loss function. Note that the overall loss function may be a weighted sum of the loss functions of the machine learning model used by the first processing unit 22 and the machine learning model used by the second processing unit 24. In other words, the overall loss function may include not only information about the ground truth image and the estimated image, but also an equation corresponding to the internal configuration of the machine learning model used by the first processing unit 22. Thereafter, in step SB9, the training unit 26 may proceed with learning so as to update parameters of both the machine learning model used by the first processing unit 22 and the machine learning model used by the second processing unit 24.

[0037] Next, another example of the learning process of the machine learning model by the image processing device 1 will be described with reference to the flowchart of Fig. 5. Steps SB1 to SB4 are the same processes as those in the flowchart shown in Fig. 4.

[0038] In step SC1, the training unit 26 calculates the difference between the enlarged image and the correct image to generate a difference image. In step SC2, the training unit 26 calculates the difference between the difference image and the synthesis bit plane, and calculates a loss value using a loss function based on the difference.

[0039] 4, the machine learning model used in the second processing unit 24 may be trained so as to minimize the loss value. In this modification, when a machine learning model is used in the first processing unit 22, both the machine learning model used in the first processing unit 22 and the machine learning model used in the second processing unit 24 may be trained.

[0040] That is, the learning process shown in Fig. 4 is a process for learning to reduce the difference between the correct image and a high-resolution estimated image obtained by combining an enlarged image and a synthesis bit plane, i.e., to make the estimated image closer to the correct image. On the other hand, the learning process shown in Fig. 5 calculates a first difference between the enlarged image and the correct image, and learns to reduce a second difference between the first difference and the synthesis bit plane, i.e., to make the synthesis bit plane closer to the first difference. This allows the machine learning model to be trained so that there is no difference between the enlarged image and the correct image, i.e., so that an estimated image with a resolution comparable to that of the correct image can be obtained.

[0041] According to the second embodiment described above, the first processing unit performs super-resolution processing on a low-resolution image to generate a high-resolution enlarged image. The decomposition unit decomposes the low-resolution image into bit planes using mask processing and extracts the lower bit planes. The second processing unit inputs the lower bit planes to a machine learning model that performs super-resolution processing and generates synthesis bit planes by increasing the resolution of the lower bit planes. The synthesis unit synthesizes the enlarged image and the synthesis bit planes to generate an estimated image. The training unit trains the machine learning model to minimize the difference between the estimated image and the correct image, which is a high-resolution image. This makes it possible to generate a trained model that can reduce memory access and the amount of calculation during inference.

[0042] Next, an example of the hardware configuration of the image processing apparatus 1 according to the above-described embodiment will be described with reference to the block diagram shown in FIG.

[0043] As shown in FIG. 6, the image processing device 1 includes, for example, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a display 104, a communication interface 105, an input / output interface 106, and storage 107.

[0044] The CPU 101 is an integrated circuit mounted on a computer and capable of executing various programs, and controls the overall operation of the image processing device 1. The ROM 102 is, for example, a non-volatile semiconductor memory and stores programs and control data for controlling the image processing device 1. The RAM 103 is, for example, a volatile semiconductor memory and is used as a work area for the CPU 101. The display 104 is, for example, a liquid crystal display or a touch panel display and displays information. The communication interface 105 is, for example, an input / output component such as a USB, HDMI (registered trademark), or network interface for exchanging information with the outside. The input / output interface 106 is an interface for receiving user input from input devices such as a mouse, keyboard, or microphone, and an interface for outputting data or audio to output devices such as a display, projector, or speaker. The storage 107 is a non-volatile storage device. The storage 107 stores system software and the like for the image processing device 1. The CPU 101 may also be referred to as a "processor."

[0045] In the above embodiments, the CPU 101 of the image processing device 1 may be another circuit (or processor). For example, the image processing device 1 may include a GPU (Graphics Processing Unit), an NPU (Neural Network Processing Unit), an MPU (Micro Processing Unit), etc. instead of a CPU. Each of the processes described in each embodiment may be realized by dedicated hardware. The processes of the image processing device 1 may be a mixture of processes executed by software and processes executed by hardware, or may be only one of them.

[0046] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0047] DESCRIPTION OF SYMBOLS 1... Image processing device 11, 21... Acquisition unit 12, 22... First processing unit 13, 23... Decomposition unit 14, 24... Second processing unit 15, 25... Synthesis unit 16, 27... Storage unit 26... Training unit 101... CPU 102... ROM 103... RAM 104... Display 105... Communication interface 106... Input / output interface 107... Storage

Claims

1. An image processing device comprising: a first processing unit that performs super-resolution processing on a first image and generates a second image by increasing the resolution of the first image; a decomposition unit that extracts component images that include some of the shape information and frequency components contained in the first image; a second processing unit that inputs the component images to a trained model that performs super-resolution processing and generates component images for synthesis by increasing the resolution of the component images; and a synthesis unit that synthesizes the second image and the component images for synthesis to generate a third image.

2. An image processing device comprising: a first processing unit that performs super-resolution processing on a first image and generates a second image by increasing the resolution of the first image; a decomposition unit that extracts component images that include some of the shape information and frequency components contained in the first image; a second processing unit that inputs the component images to a machine learning model and outputs component images for synthesis that have been super-resolution processed on the component images; a synthesis unit that synthesizes the second image and the component images for synthesis to generate a third image; and a training unit that trains the machine learning model so as to minimize a loss value obtained from a loss function that uses the difference between the third image and a correct image that is an image with a higher resolution than the first image.

3. An image processing method in which a first processing unit performs super-resolution processing on a first image to generate a second image by increasing the resolution of the first image; a decomposition unit extracts component images including some of the shape information and frequency components contained in the first image; a second processing unit inputs the component images to a trained model that performs super-resolution processing to generate component images for synthesis by increasing the resolution of the component images; and a synthesis unit synthesizes the second image and the component images for synthesis to generate a third image.

4. An image processing program for causing a computer to function as each part of the image processing device according to claim 1.

Images