Image processing device, image processing method, image processing program, and endoscopic system

Abstract

To provide an image processing device, an image processing method, an image processing program, and an endoscope system that generate super-resolution images with improved image quality such as resolution and resolution senses.SOLUTION: An image processing device acquires a color image containing a plurality of primary color signals, generates a luminance signal image and a color difference signal image by performing color space transformation processing on the color image, generates a predicted luminance signal image by performing super-resolution processing on the luminance signal image and color difference signal image, and uses the predicted luminance signal image to generate a super-resolution color image with increased resolution of the color image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, an image processing program, and an endoscope system.

Background Art

[0002] With the technology of super-resolution using deep learning, it has become possible to generate a super-resolution image, which is an image with a higher resolution, by estimating from a low-resolution image. As a deep learning network for generating a super-resolution image, a generative adversarial network (GAN), a U-Net, etc. are used.

[0003] Also, performing super-resolution on medical images in the medical field has been studied. With the super-resolution image of a medical image, it has been developed to provide information for clarifying a CT image and assisting a doctor's diagnosis, to provide information for classifying lesions, cells, etc. and assisting the interpretation of examination results, and so on.

[0004] For example, there is known a method for generating a super-resolution image using a generative adversarial network including a generator and a discriminator, and implementing and learning a self-attention mechanism limited to the network of the discriminator (Patent Document 1). Also, there is known a method for generating a super-resolution image using a U-Net to generate a learned model learned with learning data including learning image data and an identifier for identifying the classification of the learning image data, and based on the first image data and a classification score which is the probability of the identifier (Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

[0006] Super-resolution images generated using super-resolution technology sometimes exhibited color shifts, color distortions, and edge misalignment. These issues could result in a decrease in perceived resolution in the generated super-resolution images.

[0007] The present invention aims to provide an image processing apparatus, an image processing method, an image processing program, and an endoscope system that generate super-resolution images with improved image quality, such as resolution and perceived resolution. [Means for solving the problem]

[0008] The image processing apparatus of the present invention comprises a processor, which acquires a color image containing multiple primary color signals, generates a luminance signal image and a chrominance signal image by performing a color space conversion process on the color image, generates a predicted luminance signal image by performing a super-resolution process on the luminance signal image and the chrominance signal image, and generates a super-resolution color image with increased resolution of the color image using the predicted luminance signal image.

[0009] Preferably, the processor generates an enlarged chrominance signal image by performing an enlargement process different from super-resolution processing on the chrominance signal image, and then generates a super-resolution color image by performing an inverse transformation process on the predicted luminance signal image and the enlarged chrominance signal image.

[0010] The scaling process is preferably a simple scaling process or an upsampling process.

[0011] Super-resolution processing is preferably performed using a convolutional neural network.

[0012] The super-resolution processing preferably includes upsampling or deconvolution.

[0013] Super-resolution processing is preferably performed using U-Net.

[0014] The processor preferably includes a pre-trained model and performs super-resolution processing using the pre-trained model.

[0015] Preferably, the processor includes a training model, the training model includes a set of parameters, and a trained model is generated by updating the parameters. The parameters are updated using a loss value, which is obtained by generating a color-converted image by performing a color space conversion process on a training image containing multiple primary color signals, generating a training luminance signal image by performing a degradation process on the color-converted image, and then comparing the training image with a training predicted luminance signal image generated by performing a super-resolution process based on the training luminance signal image.

[0016] Preferably, the trained model includes multiple parameters, which are updated using a loss value. The loss value is obtained by generating a color-converted image by performing a color space conversion process on a training image containing multiple primary color signals, generating a training luminance signal image by performing a degradation process on the color-converted image, and then comparing the training image with a training predicted luminance signal image generated by performing a super-resolution process based on the training luminance signal image.

[0017] Preferably, the system comprises two distinct networks that each generate two different training predictive luminance signal images by performing super-resolution processing, and the processor obtains two loss values ​​using the training predictive luminance signal images obtained by each of the two networks, and updates multiple parameters using these two loss values.

[0018] It is preferable that the processor performs a single super-resolution process on the image to be processed, which includes a luminance signal image and a chrominance signal image.

[0019] It is preferable to acquire endoscopic images taken using an endoscope as color images.

[0020] The image processing method of the present invention includes steps of: obtaining a color image including a plurality of primary color signals; generating a luminance signal image and a color difference signal image by performing color space conversion processing on the color image; generating a predicted luminance signal image by performing super-resolution processing on the luminance signal image; and generating a super-resolution color image with increased resolution of the color image using the predicted luminance signal image.

[0021] The image processing program of the present invention causes a computer to execute functions of: obtaining a color image including a plurality of primary color signals; generating a luminance signal image and a color difference signal image by performing color space conversion processing on the color image; generating a predicted luminance signal image by performing super-resolution processing on the luminance signal image; and generating a super-resolution color image with increased resolution of the color image using the predicted luminance signal image.

[0022] The endoscope system of the present invention includes an endoscope that captures a subject to generate an endoscope image, a display that displays the endoscope image, and an image processing device having a processor that generates a super-resolution endoscope image with increased resolution of the endoscope image. The processor obtains the endoscope image as a color image, generates a super-resolution endoscope image with increased resolution of the endoscope image, and performs control to display the super-resolution endoscope image on the display.

[0023] Preferably, the processor displays the super-resolution endoscope image on the display and also displays an indicator indicating that the displayed image is the super-resolution endoscope image.

[0024] The image processing device is preferably the image processing device described above.

Advantages of the Invention

[0025] According to the present invention, a super-resolution image with improved resolution can be generated.

Brief Description of the Drawings

[0026] [Figure 1]This is a block diagram showing the functions of an image processing device. [Figure 2] This is a block diagram showing the functions of the image processing unit. [Figure 3] This is an explanatory diagram illustrating the processing flow of the image processing unit. [Figure 4] This is an explanatory diagram illustrating the processing flow of the predictive brightness signal generation model. [Figure 5] This is a block diagram showing the functions of an image processing device having a learning unit. [Figure 6] This is a block diagram showing the functions of the learning department. [Figure 7] This is an explanatory diagram illustrating the learning process for a learning model. [Figure 8] This is an explanatory diagram illustrating the processing flow of a predicted luminance signal generation model that shows the number of bits. [Figure 9] This is a flowchart explaining the processing flow of the image processing unit. [Figure 10] This is a schematic diagram of an endoscope system. [Figure 11] This is an explanatory diagram describing the generation of super-resolution endoscopic images. [Figure 12] This image diagram illustrates the super-resolution endoscopic images and indicators displayed on the screen. [Modes for carrying out the invention]

[0027] The embodiments will be described below with reference to the drawings as appropriate. First, the process leading to the following embodiments will be explained. In recent years, deep learning using multi-layered neural networks has made it possible to perform highly accurate automatic image recognition if data on images and their corresponding answers is available. In the field of segmentation, highly accurate segmentation has become possible through methods such as downsampling and upsampling processes called U-Net, and CNNs (Convolutional Neural Networks) using skip connections. Furthermore, it has also become possible to generate images using a method called GAN.

[0028] Super-resolution techniques are also being researched to restore degraded images that have deteriorated due to a decrease in resolution using these methods. Restoring means restoring the degraded image to its original, high-resolution state before the resolution decrease. In this specification, high resolution means including not only the number of pixels in the image but also high-frequency component information. Therefore, improved resolution means not only increasing the number of pixels but also increasing the perceived resolution through the addition of high-frequency components, color shift reduction, noise reduction, etc. On the other hand, decreased resolution means that the perceived resolution decreases due to a decrease in the number of pixels in the image and / or a decrease in high-frequency component information, etc. Super-resolution refers to improving resolution, the process of performing super-resolution is called super-resolution processing, and the image generated by super-resolution processing is called a super-resolution image. Furthermore, image quality is defined as the quality of the image, including resolution.

[0029] For example, a degraded image is input to a CNN, and by comparing it with the original image and the CNN's output, the CNN learns to improve image quality by increasing resolution. However, while using images containing color information can increase the amount of information, problems such as color shift can occur during high-resolution enhancement. These color shift issues can sometimes cause a decrease in resolution or perceived resolution in the generated super-resolution image.

[0030] The following embodiments describe an image processing apparatus, an image processing method, an image processing program, and an endoscope system that generate super-resolution images with improved resolution.

[0031] An example of the basic configuration of the image processing apparatus of the present invention will be described. The image processing apparatus of the present invention is a personal computer or workstation or the like, on which an application program for realizing a predetermined function is installed. The computer is equipped with a processor, a CPU (Central Processing Unit), memory, and storage, and various functions are realized by the program stored in the storage, etc. The computer may also include a communication unit that can communicate with other computers via a network, a display unit such as a display, an input unit such as a touch panel or keyboard, etc.

[0032] As shown in Figure 1, the image processing device 10 comprises an image acquisition unit 11, an image processing unit 12, and an output unit 13 as functional components. The image acquisition unit 11 has the function of acquiring a color image containing multiple primary color signals. The image processing unit 12 has the function of processing the color image acquired by the image acquisition unit 11 and generating a super-resolution color image. The output unit 13 has the function of outputting the super-resolution color image generated by the image processing unit 12 to a storage unit or display (not shown). Each of these functional components in the image processing device 10 is implemented as a program that makes the computer function.

[0033] The color image acquired by the image acquisition unit 11, which contains multiple primary color signals, can be any color image containing multiple primary color signals, but it is mainly an RGB image containing three primary color signals: an R image consisting of a red signal value R, a G image consisting of a green signal value G, and a B image consisting of a blue signal value B. Note that the primary color signals are not limited to these three colors, but may be other three colors, or two, four, etc. The image acquisition unit 11 is instructed to acquire an RGB image for which the resolution is to be improved.

[0034] The image processing unit 12 processes the RGB image acquired by the image acquisition unit 11 to create a super-resolution color image with increased resolution. The super-resolution color image is an RGB image consisting of an R image, a B image, and a G image.

[0035] As shown in Figure 2, the image processing unit 12 comprises a color space conversion unit 21, a predicted luminance signal generation unit 22, an enlarged image generation unit 23, and a color space inverse conversion unit 24 as functional components. The color space conversion unit 21 has the function of performing color space conversion processing on the RGB image acquired by the image acquisition unit 11 and converting it into a YUV image. The YUV image is an image with luminance signal value Y, and a color difference signal value U and The system includes U images and V images, which are images based on V. The predicted luminance signal generation unit 22 has the function of generating a predicted luminance signal image, which is a super-resolution image of the luminance signal image, using the YUV image. The enlarged image generation unit 23 has the function of performing an enlargement process on the color difference signal image to generate an enlarged color difference signal image. The color space inverse conversion unit 24 has the function of converting the predicted luminance signal image and the enlarged color difference signal image into an RGB image.

[0036] As shown in Figure 3, in the workflow of each functional component of the image processing unit 12, the input RGB image 31 acquired by the image acquisition unit 11 is subjected to super-resolution processing and output as a super-resolution RGB image 35. The super-resolution RGB image 35 consists of a super-resolution R image 35a, a super-resolution G image 35b, and a super-resolution B image 35c. The image processing unit 12 acquires the input RGB image 31 acquired by the image acquisition unit 11. In this embodiment, the input RGB image 31 includes the R image 31a, the G image 31b, and the B image 31c, each with a pixel count of 512 × 512 pixels.

[0037] In this specification, the number of pixels in an image is expressed in pixels. R images and r images represent images with red signal values ​​R and r, respectively; G images and g images represent images with green signal values ​​G and g, respectively; and B images and b images represent images with blue signal values ​​B and b, respectively. Images indicated by uppercase letters such as R, G, and B images have a larger number of pixels than images indicated by lowercase letters such as r, g, and b images.

[0038] The color space conversion unit 21 performs color space conversion processing on the input RGB image 31. Color space conversion processing is the process of converting an RGB image into a luminance signal image and a chrominance signal image. It is known that RGB images can be converted into a luminance signal image and a chrominance signal image. Because RGB images redundantly store the image structure, in the field of image compression, the color space is sometimes converted into a luminance signal and a chrominance signal, with JPEG (Joint Photographic Experts Group) being a typical example. In JPEG, during the encoding process, the RGB image is converted into a luminance signal Y and chrominance signals Cb and Cr, and downsampling of the chrominance signals Cb and Cr, which are said to have little effect on image quality, is performed as (Y:Cb:Cr) (4:2:2), etc., and upsampling is performed during decoding.

[0039] The color space conversion unit 21 can use various color space conversion methods in addition to the format used in JPEG. In this embodiment, the RGB image is converted to a YUV image using a method used in color composite video signals. In this method, the conversion from an RGB image to a YUV image can be performed by the following equations (1) to (3). In equations (1) to (3), R is the red signal value in the RGB image, G is the green signal value in the RGB image, B is the blue signal value in the RGB image, Y is the luminance signal value in the YUV image, U is the color difference signal value of U in the YUV image, and V is the color difference signal value of V in the YUV image. The luminance signal value Y, and the color difference signal values ​​U and V are calculated for each pixel of the input RGB image 31. Note that, as with the RGB image, the Y image and y image are images based on the luminance signal values ​​Y and y, respectively, and the U image and u image are images based on the color difference signal values ​​U and u, respectively. V-picture The image and the v-image represent images with chromatic difference signal values ​​V and v, respectively. Images indicated by uppercase letters such as Y-image, U-image, and V-image have a larger number of pixels than images indicated by lowercase letters such as y-image, u-image, and v-image.

[0040] Y=(0.29900*R)+(0.58700*G)+(0.11400*B) (1) U=(-0.14713*R)+(-0.28886*G)+(0.43600*B) (2) V=(0.61500*R)+(-0.51499*G)+(-0.10001*B) (3)

[0041] In this embodiment, the input RGB image 31 is an RGB image including R image 31a, G image 31b, and B image 31c. Therefore, the luminance signal image and chrominance signal image obtained by performing a color space conversion process on the input RGB image 31 are the input YUV image 32, which includes Y image 32a, U image 32b, and V image 32c, each being 512 × 512 pixels. The predicted luminance signal generation unit 22 receives the input YUV image 32 and processes it.

[0042] The predicted luminance signal generation unit 22 generates a predicted luminance signal image by performing super-resolution processing on the luminance signal image and the chrominance signal image. Since the predicted luminance signal image is an image obtained by performing super-resolution processing on the luminance signal image, etc., it has a higher resolution than the luminance signal image. In this embodiment, the luminance signal image and the chrominance signal image are an input yuv image 32 which includes a y image 32a which is a luminance signal image and a u image 32b and a v image 32c which are chrominance signal images, and the predicted luminance signal image 33 is a Y image 33a which has a higher resolution than the y image 32a. Both the y image 32a and the Y image 33a are images of luminance signals.

[0043] The predicted luminance signal generation unit 22 generates a predicted luminance signal image 33 using a predicted luminance signal generation model 41 (see Figure 4). The predicted luminance signal generation model 41 can be any model that performs super-resolution, but it is preferable that it is a model that performs super-resolution using a CNN because the generated predicted luminance signal image 33 is of good quality, and more preferably that it is a model that performs super-resolution using a U-Net because the generated predicted luminance signal image 33 is of good quality.

[0044] In this specification, "model" refers to a series of algorithms that process input, and includes models that have not undergone training for parameter adjustment, etc., models that have undergone training in at least a part of them, and models that have undergone training to become a trained model, and then have their parameters updated by training the trained model again. Among the "models," "trained model" specifically refers to a model that has undergone training in at least a part of it using training data, etc. Therefore, "trained model" also includes models generated by training a trained model again. The predicted luminance signal generation model 41 is a trained model. The generation of the predicted luminance signal generation model 41 by training will be described later.

[0045] U-Net is a model consisting of a fully convolutional network (FCN) using a CNN, and is used for semantic segmentation. However, it is also known that the U-Net architecture is used to achieve super-resolution (see "Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections" CVPR2016, (USA), 4 / 2016, p.210).

[0046] The predicted luminance signal generation model 41 (see Figure 4) employs a U-Net architecture comprising an encoder, a folding layer, and a decoder. The encoder and decoder each include one or more intermediate layers. The folding layer connects the downstream intermediate layer in the encoder and the upstream intermediate layer in the decoder.

[0047] In the encoder's intermediate layer, a feature map is generated by performing a convolution operation using multiple kernels on the input image, or by performing downsampling to reduce the number of pixels compared to the feature map input to the intermediate layer using pooling, and then generating a feature map by performing a convolution operation using multiple kernels. Therefore, the processing in the encoder can be said to be processing using a CNN. Pooling is a type of downsampling. The feature map is a tensor consisting of the vertical and horizontal size, number of channels, etc., of the convolution result, which represents the features of the input image extracted by each kernel in the convolution operation. In the feature map, the vertical size, horizontal size, and number of channels are collectively referred to as the number of elements. A channel represents the data of constituent elements such as color at each pixel of the image or convolution result, and the number of channels indicates the number of data at each pixel of the image or convolution result. Note that the size in the convolution result is the same as the number of pixels. The convolution result or tensor is a type of image data.

[0048] The aliasing layer functions as both an encoder and a decoder. It performs downsampling by pooling to reduce the vertical and horizontal size of the convolution results included in the feature map input to the aliasing layer, then performs convolution using multiple kernels, and finally performs upsampling by inverse convolution to increase the vertical and horizontal size of the convolution results included in the feature map. Information reduction processing may also be performed on the feature map in the aliasing layer. Information reduction processing is a process to reduce the amount of information, and can employ methods such as data type conversion or reducing the number of elements in the feature map. Therefore, the amount of information can be expressed by the number of elements in the feature map, the number of bits, etc. The number of elements in the feature map can be expressed as a combination of the vertical size, horizontal size, and number of channels of the image, convolution results, etc., as explained below. The number of bits can be determined from the number of elements and the data type.

[0049] In the decoder's intermediate layer, the vertical and horizontal size of the convolution results contained in the feature map is increased by upsampling. Upsampling methods can be those used for image upsampling, such as bilinear, bicubic, average pixel methods, and deconvolution. Deconvolution is preferred as it yields good results. Deconvolution can be a combination of scaling and convolution processes applied to the convolution results in the feature map being processed. The processing in the decoder is the reverse of convolution and pooling, and can therefore be considered a CNN-based process. Furthermore, it is preferable that the upsampling performed on the feature map in the decoder's intermediate layer is different from the upsampling performed on the chrominance signal image, which will be explained later.

[0050] In addition to performing convolution and other operations in each intermediate layer of the encoder, the feature map generated in each intermediate layer of the encoder is passed to the intermediate layer of the decoder at the corresponding level. In U-Net, the decoder can utilize the feature map output from the intermediate layer of the encoder, enabling highly accurate super-resolution. Therefore, it is preferable to set the number of intermediate layers in each level of the decoder so that it can receive feature maps from the intermediate layers of each level of the encoder, and it is even more preferable that the number of intermediate layers in the encoder and decoder be the same. Furthermore, in order to ensure that the number of pixels in the feature map received by the decoder's intermediate layer from the encoder's intermediate layer and the feature map received by the decoder's intermediate layer from the intermediate layer of the upstream level do not differ significantly, it is preferable that the number of pixels increased or decreased by downsampling or upsampling in each intermediate layer of the corresponding level in the encoder and decoder be approximately the same, depending on whether padding is used or not. In addition, in the feature map passed from the encoder's intermediate layer, a portion such as the central part of the feature map from the encoder may be selected and then received by the decoder's intermediate layer. The number of intermediate layers in the encoder and decoder will be determined as appropriate on a case-by-case basis.

[0051] Adjustable parameters in the encoder, decoder, and folded layer processing can be set as appropriate. Examples of adjustable parameters include the number of convolutions, kernel size, number of channels, downsampling or upsampling method, and the size reduced by pooling or increased by deconvolution. In addition to updating the model parameters, training may also include adjusting these adjustable parameters.

[0052] In this embodiment, the predictive luminance signal generation model 41 is a trained model learned using U-Net, and is trained to preferably output a Y image 33a, which is a super-resolution image of the Y image 32a, upon input of the Yuv image 32, and various parameters and other items have been adjusted. The predictive luminance signal generation model 41 (see Figure 4) generates and outputs a super-resolution image with a larger number of pixels than the input Yuv image 32 using an encoder and decoder.

[0053] As shown in Figure 4, the predicted luminance signal generation model 41 of this embodiment is a network consisting of a U-Net architecture comprising an encoder 42 and a decoder 43. In this embodiment, the encoder 42 comprises an intermediate layer 45 consisting of intermediate layers 45a, 45b, 45c, and 45d, and a folded layer 44. The decoder 43 comprises an intermediate layer 46 consisting of intermediate layers 46a, 46b, 46c, and 46d. In Figure 4, the intermediate layers are indicated by diagonal lines. The numbers written below or above the figures representing each layer indicate the number of elements in the feature map or image consisting of the convolution result output by each layer, and represent the vertical size, horizontal size, and number of channels of the image, etc., in that order. When the intermediate layers of the encoder 42 or decoder 43 are not distinguished, they are referred to as intermediate layer 45 or intermediate layer 46.

[0054] The predicted luminance signal generation model 41 may have an input layer 48 and an output layer 49 for processing for subsequent processing. The input layer 48 and output layer 49 perform data type conversions, etc. The input YUV image 32 is first input to the input layer 48 and then input to the first intermediate layer 45a of the encoder 42. The feature map output from the last intermediate layer 46a of the decoder 43 is input to the output layer 49, where processing is performed to create a super-resolution YUV image 47 consisting of a Y image 33a, a U image 33b, and a V image 33c.

[0055] The input YUV image 32 input to the input layer 48 undergoes a data type conversion and is passed to the hidden layer 46a. The input YUV image 32 and the YUV image output by the input layer 48 have vertical dimensions, horizontal dimensions, and number of channels of 512, 512, and 3, respectively.

[0056] In the first hidden layer 45a of the encoder 42, a multi-channel feature map is generated by performing a convolution operation using multiple kernels on the input YUV image 32. The feature maps output by the hidden layer 45a have 512, 512, and 64 elements in the convolution result, respectively. The hidden layer 45a passes the generated feature maps to the hidden layer 45b in the downstream layer of the encoder 42 and to the corresponding hidden layer 46a of the decoder 43.

[0057] In the hidden layer 45b of the downstream layer of encoder 42, the number of pixels in the feature map is reduced by performing pooling on the feature map passed from the hidden layer 45a of the upstream layer. Then, a multi-channel feature map is generated by performing convolution using multiple kernels. The feature maps output by the hidden layer 45b have 256, 256, and 128 elements in the convolution result, respectively. The hidden layer 45b passes the generated feature maps to the hidden layer 45c of the downstream layer of encoder 42 and to the corresponding hidden layer 46b of decoder 43.

[0058] The same process is repeated in each intermediate layer 45 of the encoder 42. Specifically, in the intermediate layer 45c immediately downstream of the intermediate layer 45b, the number of pixels in the feature map is reduced by performing pooling on the feature map passed from the intermediate layer 45b of the upstream layer. Then, a multi-channel feature map is generated by performing convolutional processing using multiple kernels. The intermediate layer 45c passes the generated feature map to the intermediate layer 45d of the encoder 42 and to the corresponding intermediate layer 46c of the decoder 43.

[0059] In the pooling process in the intermediate layer 45 of the encoder 42, the number of pixels is halved. In addition, a 64-channel feature map is generated in the convolution process of the first intermediate layer 45a, and in the convolution process of each intermediate layer 45 other than intermediate layer 45a, a feature map is generated with twice the number of channels as the previous intermediate layer 45. In this way, in the downstream intermediate layer 45d of the encoder 42, a feature map is generated with a number of elements of 64, 64, and 512 based on the input YUV image 32, and the number of pixels or information is reduced. The feature map generated by the downstream intermediate layer 45d of the encoder 42 is passed to the folding layer 44.

[0060] In the aliasing layer 44, both downsampling and upsampling are performed in this order. Therefore, in Figure 4, the aliasing layer 44 is shown as including both an encoder 42 that performs downsampling and a decoder 43 that performs upsampling.

[0061] In the folded layer 44, the number of pixels in the feature map is reduced by performing pooling on the feature map received from the hidden layer 45d. Then, a convolution process using multiple kernels is performed to generate feature maps for multiple channels. The resulting feature maps have 32, 32, and 1024 elements.

[0062] Here, the folded layer 44 may perform an information reduction process to reduce the number of channels in the feature map. In this embodiment, the information reduction process reduces the number of elements in the feature maps, which were 32, 32, and 1024, to 32, 32, and 128.

[0063] In the folded layer 44, feature maps for multiple channels are generated by performing deconvolution using multiple kernels. As a result, the feature maps output by the folded layer 44 have 64, 64, and 512 elements in the convolution result. The feature maps generated by the folded layer 44 are output to the uppermost intermediate layer 46d of the decoder 43.

[0064] In the uppermost hidden layer 46d of the decoder 43, a feature map with multiple channels is generated by performing a deconvolution operation using multiple kernels on the combined feature map passed from the folding layer 44 and the feature map passed from the corresponding hidden layer 45d of the encoder 42. The feature maps output by the hidden layer 46d have 128, 128, and 256 elements as a result of the convolution. The hidden layer 46d passes the generated feature map to the next downstream hidden layer 46c of the decoder 43.

[0065] Next, the downstream hidden layer 46c performs the same processing on the received feature map as the upstream hidden layer 46d performed. That is, the hidden layer 46c of the decoder 43 generates a multi-channel feature map by performing a deconvolution operation using multiple kernels on the combined feature map passed from the upstream hidden layer 46d and the feature map passed from the corresponding hidden layer 45c of the encoder 42. The feature map output by the hidden layer 46c has 256, 256, and 128 elements as the convolution result. The hidden layer 46c passes the generated feature map to the hidden layer 46b of the next downstream layer of the decoder 43. If the hidden layer 46 consists of multiple layers, these processes are repeated in the hidden layer 46 of each layer.

[0066] The same process is repeated in each intermediate layer 46 of the decoder 43. Specifically, in the intermediate layer 46b, which is downstream of the intermediate layer 46c, a feature map with multiple channels is generated by performing deconvolution using multiple kernels on the combined feature map passed from the intermediate layer 46c of the upstream layer and the feature map passed from the corresponding intermediate layer 45b of the encoder 42. The intermediate layer 46b then passes the generated feature map to the intermediate layer 46a, which is downstream of the decoder 43.

[0067] Furthermore, the downstream hidden layer 46a of the decoder 43 may process a combination of the feature map passed from the upstream hidden layer 46b, the feature map passed from the corresponding hidden layer 45a of the encoder 42, and the input yuv image 32. The hidden layer 46a generates a multi-channel feature map by performing deconvolution using multiple kernels on this combination of the three. The feature map output by the hidden layer 46a has 1024, 1024, and 32 elements as the convolution result. The feature map output by the hidden layer 46a is input to the output layer 49.

[0068] The output layer 49 receives the feature map output by the downstream intermediate layer 46a of the decoder 43. The output layer 49 adjusts the number of channels, converts data types, etc., and outputs a super-resolution YUV image 47 with 1024, 1024, and 3 image elements. The super-resolution YUV image 47 includes a Y image 33a, a U image 33b, and a V image 33c, and the three channels correspond to a luminance signal image and two chrominance signal images.

[0069] As described above, the predictive luminance signal generation model 41 is a model designed and trained to preferably output a Y image 33a, which is a super-resolution image of the y image 32a. Therefore, by inputting the input yuv image 32, it can preferably output a Y image 33a, which is a super-resolution image of the y image 32a. The Y image 33a is the predictive luminance signal image 33.

[0070] The enlarged image generation unit 23 generates an enlarged chrominance signal image by performing an enlargement process on the chrominance signal image. The enlargement process performed by the enlarged image generation unit 23 is a different process from super-resolution processing. As shown in Figure 3, in this embodiment, the enlarged image generation unit 23 includes u picture image and v picture image The input YUV image 32 contains u picture image and v picture image The number of pixels has increased compared to U picture image andV picture image An enlarged color difference signal image 34 consisting of these components is generated.

[0071] The enlargement process performed by the enlarged image generation unit 23 can be any process different from the super-resolution process performed to generate the predicted luminance signal image 33, and includes simple enlargement, upsampling, etc. Simple enlargement is a process that simply stretches the vertical and horizontal size of each pixel to increase the number of pixels. Upsampling is a process that increases the number of pixels while calculating and interpolating the pixel values ​​of the increasing pixels. Various methods can be employed in upsampling. Examples of upsampling methods for images include the bilinear method, the bicubic method, and the average pixel method.

[0072] The number of pixels in the enlarged color difference signal image 34 generated by the enlargement process performed by the enlarged image generation unit 23 is the same as the number of pixels in the predicted luminance signal image 33. Therefore, in this embodiment, since the u image 32b and the v image 32c were each 512 × 512 pixels, the vertical and horizontal sizes of each of these images are enlarged by a factor of two, so that the U image 34a and the V image 34b are the same 1024 × 1024 pixels as the y image 32a, which is the predicted luminance signal image 33.

[0073] The color space inverse conversion unit 24 performs a color space inverse conversion on the predicted luminance signal image 33 and the expanded chrominance signal image 34 to generate a super-resolution RGB image 35. In this embodiment, as with the conversion from an RGB image to a YUV image, the following conversion formulas (4) to (6), which are used in color composite video signals, are used. In formulas (4) to (6), R is the red signal value in the RGB image, G is the green signal value in the RGB image, B is the blue signal value in the RGB image, Y is the luminance signal value in the YUV image, U is the chrominance signal value of U in the YUV image, and V is the chrominance signal value of V in the YUV image. According to these formulas, the red signal value R, the green signal value G, and the blue signal value B are mutually convertible with the luminance signal value Y, the chrominance signal value U, and the chrominance signal value V.

[0074] R=(1.00000*Y)+(0.00000*U)+(0.13983*V) (4) G=(1.00000*Y)+(-0.39465*U)+(-0.58060*V) (5) B=(1.00000*Y)+(2.03211*U)+(0.00000*V) (6)

[0075] The output unit 13 outputs the super-resolution RGB image 35, which is a super-resolution image generated by the image processing device 10, to a display unit such as a display or storage device. In this case, the image processing device 10 is connected to the display or storage device. When outputting the super-resolution RGB image 35 for display on a display, it is preferable to display an indicator on the display that the image being displayed is not an image obtained directly from photographing the subject, but rather includes image data predicted by super-resolution processing, before displaying the super-resolution RGB image 35. This is because the super-resolution RGB image 35 is not an image obtained from photographing the subject, but includes image data predicted by a predicted luminance signal image 33, etc., and this is to prevent misleading doctors, etc., during diagnosis, etc.

[0076] As described above, the image processing unit 12 can generate a super-resolution image with improved resolution compared to the input image. This super-resolution image can be a super-resolution image with excellent resolution, suppressing color shift, noise, etc. Normally, when performing super-resolution with a CNN, an RGB image is input and an RGB image is output. In this case, the R image, G image, and B image must be inferred separately during inference, which can sometimes cause color shift. In the image processing unit 12, the RGB image is converted into a luminance signal image and a chrominance signal image, the luminance signal image and the chrominance signal image are input to the CNN, and only the predicted luminance signal image 33, which is the Y image 33a, is output to generate a super-resolution image, and the chrominance signal image is generated by, for example, simple scaling to generate a super-resolution RGB image 35. With the above configuration, the super-resolution RGB image 35 is a super-resolution image with excellent resolution, suppressing color shift, noise, etc.

[0077] Furthermore, in the image processing unit 12, since the aliasing layer 44 of the predicted luminance signal generation model 41 is a model trained based on a model that has undergone information reduction processing, the generated predicted luminance signal image 33 can be a high-resolution image with more high-frequency components added compared to the luminance signal image of the original high-resolution image before the degradation of the input RGB image 31 used for training. Therefore, with the above configuration, the super-resolution RGB image 35 generated by the image processing device 10 having the image processing unit 12 can be a higher-quality image than the original high-resolution image before degradation, even if the number of pixels is the same, because color shift, noise, etc. are suppressed and more high-frequency components are added.

[0078] Next, the generation of the predicted luminance signal generation model 41 (see Figure 4) will be explained. The predicted luminance signal generation model 41 is a trained model generated by training a predicted luminance signal generation learning model (hereinafter referred to as the learning model). As shown in Figure 5, the image processing device 10 includes a learning unit 14 as a functional component when performing training to generate the predicted luminance signal generation model 41. The image processing device 12 uses the predicted luminance signal generation model 41 generated by the learning unit 14.

[0079] The pre-training model includes a network consisting of a series of algorithms with multiple pre-set parameters. A trained model is generated by performing training to adjust these parameters. Note that training the pre-training model may include adjusting other adjustable items within the pre-training model, in addition to adjusting the parameters it contains.

[0080] As shown in Figure 6, the learning unit 14 includes, as a functional component, a degradation processing unit 51 that has the function of performing degradation processing on a source image, which is a learning image containing multiple primary color signals; a comparison unit 52 that has the function of calculating loss by comparing an image processed by the learning model with the source image, which is the learning image; and a feedback control unit 53 that has the function of updating the parameters in the learning model by feeding back the loss calculated by the comparison unit 52 to the learning model. Since the source image is usually an RGB image, the target of the degradation processing performed by the degradation processing unit 51 is the source image that has been converted into a YUV image, which is a color conversion image, by performing color space conversion processing on the RGB image. The degradation processing unit 51a performs degradation processing on the source image, which is a YUV image, such as reducing the number of pixels, to generate a YUV image with reduced resolution.

[0081] The training model only needs to be capable of outputting a super-resolution image of an input image when an image is input to the predictive brightness signal generation model 41 generated through training. Examples of training models that generate super-resolution images include GANs and U-Nets. In this embodiment, a training model that is a modified version of a basic U-Net is used. A basic U-Net is a network that has an encoder and a decoder, each consisting of one network, with the intermediate layers of the encoder and decoder connected.

[0082] As shown in Figure 7, the learning model 61 in this embodiment is a modified version of the U-Net which includes an encoder 42 and decoder 43, which have the same architecture as the predicted brightness signal generation model 41, by adding mainly two components.

[0083] The first point is that it branches off to the folding layer 44 and connects to an independent branch network 62. The branch network 62 performs upsampling separately from the decoder 43 and generates a secondary predicted luminance signal. imageThis generates 64. Note that the branch network 62 is independent and not connected to other networks such as the encoder 42. Therefore, since the processing is different from that in the decoder 43, it is possible to generate a feature map unique to the branch network 62.

[0084] The second point is that the system includes comparison units 52a and 52b, which perform the function of calculating two types of losses by comparing the source image 63b of the YUV image, which is the training data, with the predicted luminance signal image 33 and sub-predicted luminance signal 64 generated by the training model, respectively, and a feedback control unit 53, which performs the function of changing the parameters of the training model to minimize the losses. The predicted luminance signal image 33 and sub-predicted luminance signal 64 generated by the predicted luminance signal generation model 41 and the branch network 62, respectively, included in the training model 61 are training luminance signal images. When comparison units 52a and 52b are not distinguished, the comparison unit 52 is referred to as the comparison unit 52.

[0085] For training the training model 61, a source image 63a of a high-resolution RGB image, which serves as training data, is used. To compare with the predicted luminance signal image 33, which is a super-resolution image generated by the predicted luminance signal generation model 41, or the sub-predicted luminance signal 64, which is a super-resolution image generated by the branching network 62, it is preferable to use an image that is a cropped portion of a high-resolution image for source image 63a or source image 63b. Source image 63a is a color image containing multiple primary color signals, and in this embodiment, it is an RGB image containing R, G, and B images. Note that when source image 63a, which is an RGB image, and source image 63b, which is a YUV image, are not distinguished, they are referred to as source image 63.

[0086] In Figure 7, source image 63a is an RGB image containing R, G, and B images. Source image 63b is a YUV image containing Y, U, and V images, and input yuv image 32 is an image containing one luminance signal and two chrominance signals. Source image 63 and input yuv image are each represented by three rectangular shapes. One rectangle represents an image consisting of one image signal, and a rectangle with a diagonal line represents an image consisting of luminance signals. Circular shapes represent processing units, and rounded rectangular shapes represent each layer of the learning model's processing unit. In the figure, the same sign indicates the same thing.

[0087] The source image 63a, which is an RGB image, is first input to the color space conversion unit 21, where a color space conversion process is performed to obtain the source image 63b, which is a YUV image. Subsequently, a degradation process is performed by the degradation processing unit 51a. In the degradation process and the color space conversion process, the color space conversion process is performed first, followed by the degradation process. This allows the source image 63b, which contains high-resolution luminance signals, to be used as the source image for training. In this embodiment, the source image 63 has a pixel count of 1024 × 1024 pixels.

[0088] The color space conversion process performed by the color space conversion unit 21 is the same as that described in the predicted luminance signal generation model 41 above. The YUV image source image 63b generated by the color space conversion process is sent to the degradation processing unit 51. The degradation processing unit 51 generates a degraded input YUV image 32 by performing a degradation process on the YUV image source image 63b.

[0089] Degradation processing refers to processes that reduce the perceived resolution of an image by reducing the number of pixels and / or reducing the information of high-frequency components. In this embodiment, the source image 63b, which is a YUV image, is converted into an input YUV image 32, which is a degraded image with a reduced number of pixels. The Y image 32a (see Figure 4) of the YUV image is a learning luminance signal image. In this embodiment, the input YUV image 32 has a pixel count of 512 × 512 pixels.

[0090] Degradation processing may include filtering, which involves applying a filter, or noise addition, which involves adding noise. Degradation processing may be performed using one or more of these processes in combination. Since the performance of the super-resolution processing in the trained model, the predicted luminance signal generation model 41, may be affected by the degradation processing, it is preferable to determine the content of the degradation processing according to the desired super-resolution processing.

[0091] The input YUV image 32 is input to the input layer 48 of the decoder 43, and then processed by the intermediate layers 45a, 45b, 45c, and 45d of the decoder 43 before being input to the aliasing layer 44, as described in the predicted luminance signal generation model 41 above.

[0092] As described above, the folded layer 44 in the training model 61 branches off and connects to an independent branch network 62. After processing the received feature map, the folded layer 44 outputs to the uppermost intermediate layer 46d of the decoder 43, and also outputs to the input layer 66 of the branch network 62.

[0093] The processing in the aliasing layer 44 is the same as in the predicted luminance signal generation model 41 (see Figure 4), and both downsampling and upsampling are performed in that order. In the aliasing layer 44, the number of pixels in the feature map is reduced by performing pooling on the feature map received from the hidden layer 45d. Then, a convolutional process using multiple kernels is performed to generate feature maps for multiple channels. The feature maps in this case have 32, 32, and 1024 elements.

[0094] Here, the folded layer 44 performs an information reduction process to reduce the number of channels in the feature map. In this embodiment, the information reduction process reduces the number of elements in the feature maps, which were 32, 32, and 1024, to 32, 32, and 128.

[0095] Subsequently, the feature maps with 32, 32, and 128 elements are sent to the input layer 66 of the branched network and to the next processing in the folding layer 44. In the next processing in the folding layer 44, feature maps for multiple channels are generated by performing deconvolution using multiple kernels. As a result, the feature maps output by the folding layer 44 have 64, 64, and 512 elements as a result of the convolution. The feature maps generated by the folding layer 44 are output to the uppermost intermediate layer 46d of the decoder 43. The feature maps output to the uppermost intermediate layer 46d of the decoder 43 undergo the same processing as in the predicted luminance signal generation model 41 (see Figure 4), passing through each intermediate layer of the decoder 43, and the predicted luminance signal image 33, which is the Y image, is generated.

[0096] In the branched network 62, the feature maps with 32, 32, and 128 elements, received by the input layer 66 from the folding layer 44, are passed from the input layer 66 to the intermediate layer 65 of the branched network 62. In the processing at the intermediate layer 65, similar to the processing at the intermediate layer 46 of the decoder 43 (see Figure 4), feature maps for multiple channels are generated by performing deconvolution using multiple kernels. However, unlike the intermediate layer 46, the processing at the intermediate layer 65 of the branched network 62 does not take input information from other sources, including the intermediate layer 45 of the encoder 42, and is performed solely by the branched network 62.

[0097] In this embodiment, the intermediate layer 65 has four stages: intermediate layer 65a, intermediate layer 65b, intermediate layer 65c, and intermediate layer 65d. In the uppermost intermediate layer 65a of the branched network 62, a feature map with multiple channels is generated by performing deconvolution using multiple kernels on the feature map passed from the folding layer 44. The feature maps output by the intermediate layer 65a have 128, 128, and 256 elements as convolution results. The intermediate layer 65a passes the generated feature map to the intermediate layer 65b of the next downstream layer of the branched network 62.

[0098] Next, the downstream hidden layer 65b performs the same processing on the received feature map as the upstream hidden layer 65a did. That is, the hidden layer 65b of the branched network 62 generates a feature map with multiple channels by performing deconvolution using multiple kernels on the feature map passed from the hidden layer 65a of the upstream layer. The feature maps output by the hidden layer 65b have 256, 256, and 128 elements as a result of the convolution. The hidden layer 65b passes the generated feature map to the hidden layer 65c of the next downstream layer of the branched network 62. If the hidden layer 65 consists of multiple layers, these processes are repeated in the hidden layer 65 of each layer.

[0099] The same process is repeated in each hidden layer 65 of the branched network 62. Specifically, in the hidden layer 65c, which is downstream of hidden layer 65b, a feature map with multiple channels is generated by performing deconvolution using multiple kernels on the feature map passed from the hidden layer 65b of the upstream layer. The hidden layer 65c passes the generated feature map to the hidden layer 65d, which is downstream of the branched network 62. In the hidden layer 65d, a feature map with multiple channels is generated by performing deconvolution using multiple kernels on the feature map passed from the hidden layer 65c of the upstream layer.

[0100] The feature map output by the hidden layer 65d, which is the downstream hidden layer of the branched network 62, has a convolution result with 1024, 1024, and 3 elements. This can be a 1024x1024 pixel YUV image consisting of three channels: Y image, U image, and V image. In this way, the branched network 62 generates a sub-predicted luminance signal image 64, which is a 1024x1024 pixel Y image.

[0101] The predicted luminance signal image 33 and the sub-predicted luminance signal image 64 are compared with the Y image of the training data source image 63b in the comparison unit 52a and comparison unit 52b, respectively, to calculate the loss value. The loss value is a value calculated by the loss function and is an evaluation of how well the predicted luminance signal image 33 and the sub-predicted luminance signal image 64 can correctly reconstruct the source image 63 when compared with the source image 63.

[0102] As loss functions, those used in super-resolution, CNNs, etc., can be employed, and examples include precision, recall, F-score (Dice coefficient), IoU (Jaccard coefficient), sum of squared errors, and cross-entropy error. The choice of loss function may also be determined by calculating two or more different loss values ​​using different loss functions, training with each of them, and then evaluating these two or more loss values ​​using evaluation metrics.

[0103] Furthermore, since the super-resolution images, the predicted luminance signal image 33 and the sub-predicted luminance signal image 64, may have a higher resolution than the Y image of the source image 63b before degradation processing, in addition to using the loss value that minimizes the loss when the predicted luminance signal image 33 and the sub-predicted luminance signal image 64 are identical pixel by pixel to the Y image of the source image 63b before degradation processing, a loss value may also be calculated that minimizes the loss when the predicted luminance signal image 33 and the sub-predicted luminance signal image 64 have a higher resolution than the Y image of the source image 63b before degradation processing. Specifically, by employing unsupervised learning techniques such as GANs, a loss value may be calculated that minimizes the loss when the predicted luminance signal image 33 and the sub-predicted luminance signal image 64 have a higher resolution than the Y image of the source image 63b before degradation processing.

[0104] The feedback control unit 53 uses the two loss values ​​calculated by the comparison unit 52a and the comparison unit 52b to predict brightness. degree The parameters in the training model 61 are updated and adjustment items are adjusted so that the signal image 33 and the sub-predicted luminance signal image 64 have a higher resolution than the Y image of the source image 63b before degradation processing. The feedback control unit 53 can adjust the parameters so that the two loss values ​​are minimized.

[0105] Furthermore, in the predicted luminance signal generation model 41, it is preferable that the predicted luminance signal image 33 and the sub-predicted luminance signal image 64 have a higher resolution than the Y image of the source image 63b before degradation processing. Therefore, the control in the feedback control unit 53 may be adjusted with the goal of not minimizing the loss value. Specifically, a loss value may be set for the case where the super-resolution predicted luminance signal image 33 and the sub-predicted luminance signal image 64 have a higher resolution than the Y image of the source image 63b before degradation processing, and parameters may be adjusted with this as the target.

[0106] During training, this series of steps using the source image 63 is repeated using multiple source images 63. The parameters are adjusted so that the output predicted luminance signal image 33 and sub-predicted luminance signal image 64 have the same or higher resolution than the Y image of the source image 63b before degradation processing. Training is completed when it is evaluated that the results of the output predicted luminance signal image 33 and sub-predicted luminance signal image 64 will not improve even if training is performed further. The loss function described above can be used for evaluation. The image processing unit 12 uses the predicted luminance signal generation model 41 at this point, with its parameters adjusted, as the trained model.

[0107] In the predicted luminance signal generation model 41 (see Figure 4), the input YUV image 32, which includes the Y image 32a, U image 32b, and V image 32c, is used as the input, and as a result of super-resolution processing, a super-resolution YUV image is generated, which includes the Y image 33a, U image 33b, and V image 33c. 47 However, only the Y image 33a, which is the luminance signal image, is used to calculate the loss and perform feedback. Thus, in the predictive luminance signal generation model 41, the input yu includes the luminance signal image and the chrominance signal image. v It is preferable to perform a single super-resolution process on image 32. That is, although only luminance signal images such as Y image 33a are used for calculating the loss value and updating parameters, the predictive luminance signal generation model 41 also receives chrominance signal images other than y image 32a as input and processes them to output super-resolution images. This is because it is possible to generate super-resolution images of the luminance signal image with a greater amount of information.

[0108] Furthermore, as described above, this embodiment includes a network consisting of a prediction luminance signal generation model 41 and a branch network 62, which are two distinct networks that generate two different training prediction luminance signal images by performing super-resolution processing. The comparison unit 52 obtains two loss values ​​by using the training prediction luminance signal image 33 and the sub-prediction luminance signal image 64 obtained by each of these two networks. The parameters of the prediction luminance signal generation model 41 are then updated using these two loss values.

[0109] Furthermore, although the above description explains the case of training a training model 61 that has multiple pre-set parameters, it is also possible to train a trained model that has already completed training and contains multiple parameters again. In this case, the training is the same as when training the training model 61 as described above.

[0110] Furthermore, in the above, while the information reduction process in the aliasing layer 44 was evaluated by the number of elements, the reduction in information may also be evaluated by the total number of bits in the image or feature map. In other words, the information reduction process can be any process that reduces the number of bits. And, regarding the number of bits in the predictive luminance signal generation model 41 (see Figure 4), it is preferable to reduce the amount of information in the aliasing layer 44 to less than the amount of information in the input YUV image 32 input to the predictive luminance signal generation model 41.

[0111] In Figure 8, the number of bits is shown instead of the number of elements in Figure 4. In this embodiment, the information content of the input YUV image 32 is 6,291,456 bits because the number of elements is 5,125,123 and the data type is an 8-bit integer. However, the data type is converted to a floating-point constant type in the input layer 48, and the number of elements in the first half of the folding layer 44 is 32, 32, and 128, resulting in 4,194,304 bits. In the information reduction process in the folding layer 44, any means that can reduce the amount of information expressed in bits, other than reducing the number of elements, can be used as appropriate.

[0112] The process flow for generating a super-resolution image by performing super-resolution processing using the image processing device 10 will be explained. As shown in Figure 9, the image processing unit 12 prepares a trained model, the predicted luminance signal generation model 41, which is generated by training the training model 61 so that super-resolution processing can be performed suitably (step ST110). Using the predicted luminance signal generation model 41, an input RGB image 31 to be subjected to super-resolution processing is prepared, and the color space conversion unit 21 converts the color space Perform the conversion process (step ST120).

[0113] color space The input YUV image 32 generated by the conversion process is processed by the image processing unit 12. When the input YUV image 32 is input to the input layer 48 of the encoder 42 of the predicted luminance signal generation model 41, the predicted luminance signal generation model 41 processes the input YUV image 32 and outputs a predicted luminance signal image 33 based on the input YUV image 32 (step ST130).

[0114] Furthermore, the enlarged image generation unit 23 outputs an enlarged chrominance signal image 34 based on the input YUV image 32 (step ST140). The color space inverse conversion unit 24 acquires the predicted luminance signal image 33 and the enlarged chrominance signal image 34 and performs inverse conversion processing on them (step ST150). The inverse conversion processing generates a super-resolution RGB image 35 (step ST160).

[0115] CNNs often use max pooling or convolution to downsample and extract features while reducing resolution. In this process, it's common to increase the number of channels to maintain information density, and the features output from the intermediate layers often retain information redundantly. Even in CNN-based super-resolution, the encoder maintains information density by increasing the number of channels instead of reducing resolution. Maintaining the information density of the input image, which is closest to the source image and accurately preserves its information, is the key to generating a super-resolution image. However, as a result, feature extraction in the encoder is limited to extracting features necessary to correct the input image to the original image, making it difficult to surpass the image quality of the original.

[0116] Therefore, in the training model 61, information reduction is performed in the aliasing layer 44, the final layer of the encoder 42, so that the encoder 42 can extract more image features. Furthermore, an independent branching network 62 that generates super-resolution is connected, branching off from the aliasing layer 44. The branching network 62 is used only during training. Information reduction in the aliasing layer 44 is achieved, for example, by adjusting the number of channels. Adjusting the number of channels means adjusting the number of elements, as described above. Also, as described above, the input image, the input YUV image 32, has a resolution of 512x512, 3 YUV channels, and is an 8-bit integer image with an information content of 6,291,456 bits. In the aliasing layer 44, the resolution is downsampled to 32x32 vertically and horizontally, so by setting the number of channels to 128, the information content becomes 4,194,394 bits, reducing it by 2,097,062 bits.

[0117] The independent branch network 62 needs to generate the original image based on the reduced amount of information. Therefore, during training, it is possible to encourage this independent branch network 62 to extract more important image features. In addition, the amount of information is preserved because the feature map, which is the feature quantity output from the U-Net encoder 42, is shared by the decoder 43 in the reduced information.

[0118] Therefore, the training model 61 and the predictive luminance signal generation model 41 obtained by training it can generate super-resolution images with improved image quality, such as resolution and perceived resolution. With the above configuration, it is possible to produce super-resolution images with higher image quality than the source image 63 before degradation processing.

[0119] The image processing device 10 may acquire endoscopic images taken using the endoscope as color images. As shown in Figure 10, the endoscope system 70 includes an endoscope 71 that photographs a subject and generates endoscopic images, a processor device 72 that controls the endoscope 71 and a light source device 73, a light source device 73 that emits illumination light to irradiate the subject from the tip of the endoscope 71, a display 74 that displays endoscopic images, etc., and an input device 75 such as a touch panel, keyboard, or mouse used to input information, etc., to the processor device 72. The processor device 72 is also connected via a network 77 to a PACS (Picture Archiving and Communication System) 76 that stores and manages medical images such as endoscopic images acquired by the endoscope.

[0120] As shown in Figure 11, in the endoscope system 70, the image processing device 10 receives the endoscopic image 81 acquired by the endoscope 71 from the processor device 72 or PACS 76, and generates a super-resolution endoscopic image 82 by increasing the resolution of the received endoscopic image 81. The super-resolution endoscopic image 82 is an image obtained by performing super-resolution processing on the endoscopic image 81 by the image processing device 10. In this embodiment, super-resolution processing is performed on a partial endoscopic image 83 obtained by cropping a part of the endoscopic image 81. The setting of the partial endoscopic image 83 may be done by the user of the endoscope system 70, or by the image processing device 10, etc. The super-resolution endoscopic image 82 of the generated partial endoscopic image 83 may be displayed on the display 74 via the processor device 72, or saved to the PACS 76 via the network 77.

[0121] As configured as described above, the endoscopic system 70 can generate a super-resolution endoscopic image 82 of the endoscopic image 81 and display it on the display 74, or save it to the PACS 76 via the network 77.

[0122] Furthermore, as shown in Figure 12, when displaying the super-resolution endoscopic image 82 on the display 74, it is preferable to display both the super-resolution endoscopic image 82 and an indicator 84 on the display 74 to show that what is being displayed is the super-resolution endoscopic image 82. The indicator 84 should be displayed in a manner that allows the user to understand that what is being displayed on the display 74 is the super-resolution endoscopic image 82, as shown in Figure 1. 2 As shown, the super-resolution endoscopic image 82 may display "×4" which indicates how many times it has been magnified compared to the input endoscopic image 81, and the letters "SR" which indicate that it is a super-resolution image.

[0123] In the endoscopic system 70, the image processing device 10 described above is incorporated to process endoscopic images. Expand By generating a large image and displaying it on the display 74, the super-resolution endoscopic image 82 with high resolution can be used by doctors and others as a reference for diagnosis. Endoscopic images are characterized by containing a large amount of red as a color, and not much of other colors. In this case, the image processing device 10 divides the luminance signal image and the chrominance signal image into two color differences. space By performing a conversion and generating a super-resolution endoscopic image 82 using an image that has undergone super-resolution processing only on the luminance signal image, it is possible to generate a desirable super-resolution endoscopic image 82 with excellent resolution and reduced noise. Therefore, the image processing device 10 can be suitably used for super-resolution processing of endoscopic images.

[0124] Each of the above-described functional components is implemented as a program that makes the computer function. Therefore, the image processing program of the present invention is an image processing program that causes the computer to perform the following functions: acquiring a color image containing multiple primary color signals; generating a luminance signal image and a chrominance signal image by performing a color space conversion process on the color image; generating a predicted luminance signal image by performing a super-resolution process on the luminance signal image; and generating a super-resolution color image with increased resolution of the color image using the predicted luminance signal image.

[0125] In the above embodiment, the hardware structure of the processing unit that performs various processes, such as the image acquisition unit 11, image processing unit 12, output unit 13, and learning unit 14, included in the image processing device 10 or processor device 72, is a type of processor as shown below. These types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units; a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array); and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to perform various processes.

[0126] A single processing unit may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as client and server systems. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0127] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit in the form of a combination of circuit elements such as semiconductor devices. [Explanation of Symbols]

[0128] 10 Image Processing Device 11 Image acquisition unit 12 Image Processing Unit 13 Output section 14. Learning Department 21 Color Space Conversion Unit 22 Predicted Brightness Signal Generation Unit 23. Enlarged Image Generation Unit 24 Color Space Inverse Transformation Unit 31 Input RGB image 31a r image 31b g image 31c b image 32 Input YUV images 32a y image 32b u image 32c v image 33 Predicted luminance signal image 33a Y image 33b U image 33c V image 34. Enlarged color difference signal image 34a U image 34b V image 35 Super-resolution RGB images 35a Super-resolution R image 35b Super-resolution G image 35c Super-resolution B image 41 Predictive Brightness Signal Generation Model 42 IOS 43 Decoder 44 Folding Layer 45, 45a, 45b, 45c, 45d intermediate layer 46, 46a, 46b, 46c, 46d Intermediate layer 47 Super-resolution YUV images 48 Input Layers 49 Output Layer 51 Degradation Processing Unit 52, 52a, 52b comparison section 53 Feedback Control Unit 61 Training Models 62 Branch Network 63, 63a, 63b Source Images 64 Subpredicted Brightness Signal Image 70 Endoscopy Systems 71 Endoscopy 72 Processor Units 73 Light source device 74 displays 75 Input Devices 76 PACS 77 Network 81 Endoscopic Images 82 Super-resolution endoscopic images 83 Partial Endoscopic Images 84 Indicators

Claims

1. Equipped with a processor, The aforementioned processor, A color image containing multiple primary color signals is acquired. By performing a color space conversion process on the aforementioned color image, a luminance signal image and a chrominance signal image are generated. A predicted luminance signal image is generated by performing super-resolution processing on the luminance signal image and the chrominance signal image. Using the predicted luminance signal image, a super-resolution color image is generated with increased resolution of the color image. The predictive luminance signal generation model that generates the predictive luminance signal image comprises an encoder, a folded layer, and a decoder. The encoder receives the luminance signal image and the chrominance signal image as input. The aforementioned folded layer connects the downstream intermediate layer of the encoder and the upstream intermediate layer of the decoder, and inputs the feature map output from the encoder to the decoder. The decoder is an image processing device that outputs the super-resolution color image based on the feature map input from the encoder, the luminance signal image, and the chrominance signal image.

2. The aforementioned processor, By performing a magnification process different from the super-resolution process on the aforementioned color difference signal image, an enlarged color difference signal image is generated. The image processing apparatus according to claim 1, which generates the super-resolution color image by performing an inverse transformation process on the predicted luminance signal image and the magnified color difference signal image.

3. The image processing apparatus according to claim 2, wherein the enlargement process is a simple enlargement process or an upsampling process.

4. The image processing apparatus according to claim 1 or 2, wherein the super-resolution processing is a process using a convolutional neural network.

5. The image processing apparatus according to claim 1 or 2, wherein the super-resolution processing includes upsampling or deconvolution processing.

6. The image processing apparatus according to claim 5, wherein the super-resolution processing is a process using U-Net.

7. The aforementioned processor is equipped with a trained model, The image processing apparatus according to claim 1, which performs the super-resolution processing using the trained model.

8. comprising a processor, The aforementioned processor includes a learning model, The aforementioned learning model includes several pre-set parameters, By updating multiple parameters, a trained model is generated that performs super-resolution processing on luminance signal images and chrominance signal images. The aforementioned multiple parameters are updated using the loss value, The loss value is obtained by comparing the training image with a training image, which is generated by performing a color space conversion process on a training image containing multiple primary color signals to generate a color conversion image, then performing a degradation process on the color conversion image to generate a training luminance signal image, and subsequently performing the super-resolution process based on the training luminance signal image to generate a training predicted luminance signal image. The system comprises two different networks that generate two different training prediction luminance signal images by performing the super-resolution processing described above, The aforementioned processor, Two loss values ​​are obtained by using the training prediction luminance signal images obtained by each of the two networks. An image processing device that updates multiple parameters using two of the aforementioned loss values.

9. comprising a processor, A trained model that performs super-resolution processing on luminance signal images and chrominance signal images includes multiple parameters, The aforementioned multiple parameters are updated using the loss value, The loss value is obtained by comparing the training image with a training image, which is generated by performing a color space conversion process on a training image containing multiple primary color signals to generate a color conversion image, then performing a degradation process on the color conversion image to generate a training luminance signal image, and subsequently performing the super-resolution process based on the training luminance signal image to generate a training predicted luminance signal image. The system comprises two different networks that generate two different training prediction luminance signal images by performing the super-resolution processing described above, The aforementioned processor, Two loss values ​​are obtained by using the training prediction luminance signal images obtained by each of the two networks. An image processing device that updates multiple parameters using two of the aforementioned loss values.

10. The aforementioned processor, The image processing apparatus according to claim 1 or 2, wherein the super-resolution processing is performed once on an image to be processed, which includes the luminance signal image and the chrominance signal image.

11. The image processing apparatus according to claim 1 or 2, which acquires an endoscopic image taken using an endoscope as the color image.

12. A step to acquire a color image containing multiple primary color signals, The steps include generating a luminance signal image and a chrominance signal image by performing a color space conversion process on the aforementioned color image, The steps include generating a predicted luminance signal image by performing super-resolution processing on the aforementioned luminance signal image, The step includes generating a super-resolution color image with increased resolution of the color image using the predicted luminance signal image, The predictive luminance signal generation model that generates the predictive luminance signal image comprises an encoder, a folded layer, and a decoder. The encoder receives the luminance signal image and the chrominance signal image as input. The aforementioned folded layer connects the downstream intermediate layer of the encoder and the upstream intermediate layer of the decoder, and inputs the feature map output from the encoder to the decoder. The decoder is an image processing method that outputs the super-resolution color image based on the feature map input from the encoder, the luminance signal image, and the chrominance signal image.

13. On the computer, A function to acquire a color image containing multiple primary color signals, The function generates a luminance signal image and a chrominance signal image by performing a color space conversion process on the aforementioned color image. A function to generate a predicted luminance signal image by performing super-resolution processing on the aforementioned luminance signal image, This system performs the function of generating a super-resolution color image with increased resolution using the predicted luminance signal image. The predictive luminance signal generation model that generates the predictive luminance signal image comprises an encoder, a folded layer, and a decoder. The encoder receives the luminance signal image and the chrominance signal image as input. The aforementioned folded layer connects the downstream intermediate layer of the encoder and the upstream intermediate layer of the decoder, and inputs the feature map output from the encoder to the decoder. The decoder is an image processing program that outputs the super-resolution color image based on the feature map input from the encoder, the luminance signal image, and the chrominance signal image.

14. An endoscope that photographs a subject and generates an endoscopic image, A display for displaying the endoscopic image, The system includes an image processing device having a processor that generates super-resolution endoscopic images with increased resolution of the endoscopic images, The image processing apparatus is the image processing apparatus described in claim 1, The aforementioned processor, The aforementioned endoscopic image is acquired as a color image, As the super-resolution color image, a super-resolution endoscopic image is generated by increasing the resolution of the endoscopic image. An endoscope system that controls the display to show the super-resolution endoscopic image.

15. The endoscopic system according to claim 14, wherein the processor displays the super-resolution endoscopic image on the display and displays an indicator indicating that the displayed image is the super-resolution endoscopic image.

Images