Image processing method and electronic device

Abstract

To provide an image processing method and electronic device that can provide physicians with useful reference information while keeping costs down. [Solution] The processing unit generates a first intermediate image by performing a difference operation on a low-resolution image using a super-resolution model. The processing unit generates a second intermediate image by performing a convolution activation operation on the low-resolution image using a super-resolution model. The processing unit generates a high-resolution image by performing an addition operation on the first and second intermediate images.

Description

Technical Field

[0004] , ,

[0006] , , , , ,

[0007] , , , , , ,

[0005]

[0001] This application relates to an image processing method and an electronic device that apply super resolution imaging based on deep learning to endoscopic images.

Background Art

[0002] An endoscope inspection device is a technology mainly developed for the purpose of inspecting the inside of the body. It penetrates into the human body through various routes and observes the internal state of the human body. The endoscopic image taken by the endoscope detection device is used as a reference for a doctor to judge the state of the disease.

[0003] Also, regarding endoscopic images, since a high-resolution image contains more detailed information than a low-resolution image, the amount of information that a doctor can refer to increases.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the current technology, in order to obtain a high-resolution image, high-end shooting devices such as Olympus EVIS X1 and CV1500 are required, and these devices have the problem of being expensive.

[0005] On the other hand, in order to improve the resolution of endoscopic images, interpolation methods such as linear interpolation and bicubic interpolation have also been used in conventional methods. However, with these interpolation methods, image blurring occurs and effective reference information cannot be provided to doctors.

Means for Solving the Problems

[0006] In view of the above problems, the present invention has the following configurations.

[0007] That is, an image processing method using a processing device and a storage device, The processing device inputs the low-resolution image stored in the storage device into the super-resolution model. The processing device generates a first intermediate image by performing a difference process on the low-resolution image using the super-resolution model. The processing apparatus generates a second intermediate image by performing a convolution activation process on the low-resolution image using the super-resolution model. The processing device generates a high-resolution image by performing an addition operation on the first intermediate image and the second intermediate image.

[0008] Furthermore, the aforementioned low-resolution image is a real-time image generated by a testing device that inspects the target object.

[0009] Furthermore, the test apparatus is an endoscopic system, the low-resolution image is a low-resolution endoscopic image, and the high-resolution image is a high-resolution endoscopic image.

[0010] Furthermore, the low-resolution image is a low-resolution endoscopic image, and the high-resolution image is a high-resolution endoscopic image.

[0011] Furthermore, the difference processing is performed on the low-resolution image using a bilinear interpolation method, thereby generating the second intermediate image.

[0012] Furthermore, the convolutional activation process generates the second intermediate image by performing detailed processing on multiple convolutional layers and activation function layers.

[0013] Furthermore, the first intermediate image and the second intermediate image have the same image size.

[0014] Furthermore, the super-resolution model can perform a convolution activation process on the low-resolution image, and then perform a depth space transformation process, resulting in a second intermediate image having the same image size as the first intermediate image.

[0015] Furthermore, an electronic device including a storage device and a processing device, The storage device stores low-resolution images, The processing device is electrically connected to the storage device and incorporates a super-resolution model. The processing device inputs the low-resolution image to the super-resolution model and uses the super-resolution model to perform differential processing and convolution activation processing on the low-resolution image, respectively, thereby generating a first intermediate image and a second intermediate image. In addition, it performs an addition operation on the first intermediate image and the second intermediate image to generate a high-resolution image.

[0016] Furthermore, the electronic device is electrically connected to the testing apparatus, which inspects the object and generates a real-time image, and the real-time image is the low-resolution image.

[0017] Furthermore, the test apparatus is an endoscopic system, the low-resolution image is a low-resolution endoscopic image, and the high-resolution image is a high-resolution endoscopic image.

[0018] Furthermore, the low-resolution image is a low-resolution endoscopic image, and the high-resolution image is a high-resolution endoscopic image.

[0019] Furthermore, the super-resolution model generates the second intermediate image by performing the difference processing on the low-resolution image using a bilinear interpolation method.

[0020] Furthermore, the convolutional activation process further includes performing detailed processing on multiple convolutional layers and activation function layers to generate the second intermediate image.

[0021] Also, the first intermediate image and the second intermediate image have the same image size.

[0022] In addition, the super-resolution model can perform a depth space conversion process after performing a convolutional activation process on the low-resolution image, and the generated second intermediate image has the same image size as the first intermediate image. [Advantages of the Invention]

[0023] As described above, the image processing method and the electronic device of the present application do not need to rely on expensive hardware, and can realize high-resolution images that could only be obtained by high-end devices through the calculation of artificial intelligence (AI) software. Moreover, the generated high-resolution images have sharp edges and are less likely to be blurred. Therefore, when the technology of the present application is applied to endoscopic images, high-resolution images from 4K to 8K can be obtained while using a device with a 1080P output source. [Brief Description of the Drawings]

[0024] [Figure 1] It is a block diagram of an electronic device according to an embodiment of the present invention. [Figure 2] It is a flowchart of an image processing method according to an embodiment of the present invention. [Figure 3] It is a block diagram of an electronic device connected to a test device according to an embodiment of the present invention. [Modes for Carrying Out the Invention]

[0025] Hereinafter, embodiments of the present application will be described with reference to the drawings. In the drawings of the embodiments, some components or structures are omitted in order to clearly show the technical features of the present invention.

[0026] In these drawings, the same reference numerals represent the same or functionally related components or circuits.

[0027] Terms such as "First," "Second," etc., may be used herein to describe various components, parts, areas, or functions, but it should be understood that no component, part, area, or function should be limited by these terms, and that these terms are used solely to distinguish one component, part, area, or function from other components, parts, areas, or functions.

[0028] The explanation will be given with reference to Figure 1. Here, Figure 1 is a block diagram of an electronic device according to one embodiment of the present invention. The electronic device 10 includes a processing unit 12 and a storage device 14. The storage device 14 stores one or more low-resolution images (not shown).

[0029] The processing unit 12 is electrically connected to the storage device 14 and incorporates a super-resolution model 121. The processing unit 12 reads low-resolution images from the storage device 14 and inputs these low-resolution images to the super-resolution model 121, which performs image processing.

[0030] The processing unit 12 uses the super-resolution model 121 to perform difference processing and convolution activation processing on the low-resolution image, respectively, to generate a first intermediate image and a second intermediate image, and then performs an addition operation on the first intermediate image and the second intermediate image to generate a high-resolution image.

[0031] In one embodiment, referring to Figure 1, the electronic device 10 further includes a display device 16. The display device 16 is electrically connected to the processing device 12, which can further provide a user interface (not shown) displayed on the display device 16, and the entire image processing process is performed via the user interface.

[0032] After the super-resolution model 121 generates a high-resolution image, the processing unit 12 can further display the high-resolution image directly on the user interface of the display device 16 so that the user can view it.

[0033] In one embodiment, the electronic device 10 is an independent computing-capable electronic device (computer device) such as a personal computer, notebook computer, or tablet computer, but is not limited to these.

[0034] In one embodiment, the processing unit 12 may be a central processing unit (CPU), an embedded controller (EC), or other general-purpose or dedicated microprocessor, microcontroller, microcontroller unit (MCU), digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), or a functionally related component thereto, or a combination of the above components, but is not limited to these.

[0035] In one embodiment, the storage device 14 may store any type of fixed or removable random access memory (RAM), flash memory, hard disk drive (HDD), solid state drive (SSD), or functionally related components, or combinations thereof, for any images, photographs, or data required by the processing device 12, but the present invention is not limited to these.

[0036] The explanation will be given with reference to Figures 1 and 2. Here, Figure 2 is a flowchart of an image processing method according to one embodiment of the present invention. In the electronic device 10, the processing unit 12 may execute the image processing method using software, and as shown in step S10 of Figure 2, the processing unit 12 inputs a low-resolution image to the super-resolution model 121. Here, the matrix size of the low-resolution image is described as [1000 × 1000 × 3], but it is not limited to this.

[0037] As shown in step S12, the processing unit 12 performs difference processing on the low-resolution image using the super-resolution model 121 to generate a first intermediate image. In one embodiment, the difference processing uses bilinear interpolation to process the low-resolution image and enlarge the image to generate the first intermediate image. In this case, the matrix size of the first intermediate image after difference processing is [4000 × 4000 × 3].

[0038] Furthermore, as shown in step S14, the processing unit 12 performs a convolution activation process using the super-resolution model 121 on the same low-resolution image to generate a second intermediate image.

[0039] In one embodiment, the convolutional activation process includes executing 16 convolutional layers and an activation function layer, the activation function layer being a Parametric Rectified Linear Unit (PReLU), which performs detailed processing on a low-resolution image to generate a second intermediate image, in which case the matrix size of the second intermediate image after the convolutional activation process is [1000 × 1000 × 48].

[0040] As shown in step S16, after performing convolution activation on the low-resolution image, depth space transformation is applied to the second intermediate image using the super-resolution model 121, and the second intermediate image generated after processing has the same image size as the first intermediate image.

[0041] At this point, the matrix size of the second intermediate image after depth space transformation is converted to [4000×4000×3], and this image size is the same as that of the first intermediate image.

[0042] Finally, as shown in step S18, the processing unit 12 generates a high-resolution image by adding the first intermediate image and the second intermediate image using the super-resolution model 121, and the matrix size of the high-resolution image at this time is [4000 × 4000 × 3].

[0043] Therefore, in this application, a 1000 x 1000 RGB color image (low-resolution image) can be enlarged to a 4x higher resolution image after the above image processing has been applied.

[0044] In one embodiment, the low-resolution image is a low-resolution endoscopic image, and the high-resolution image is a high-resolution endoscopic image.

[0045] In one embodiment, referring to Figure 3, the electronic device 10 is further electrically connected to the test device 18. For example, the processing unit 12 is connected to the test device 18 via a high-resolution multimedia interface (HDMI®) or a serial digital interface (SDI). The test device 18 inspects the object and generates a real-time image, which is a low-resolution image.

[0046] Of these, test device 18 is an endoscopic system such as a detection device for colonoscopy. In this case, since the object is the intestine, the low-resolution image is a low-resolution endoscopic image, and the high-resolution image is a high-resolution endoscopic image.

[0047] Furthermore, in addition to using the real-time images generated by the test device 18 as low-resolution images when inspecting the object, the real-time images generated by the test device 18 when inspecting the object can also be stored in the storage device 14 as low-resolution images to be processed. Next, the processing device 12 reads the low-resolution images from the storage device 14 and performs subsequent image processing.

[0048] Refer to Figure 1 for explanation. Before performing image processing using the super-resolution model 121, the processing unit 12 first performs a model training step.

[0049] During training, both high-resolution and low-resolution images are used as training data. High-resolution images range in resolution from 1080P to 4K, and there are no restrictions on image type; the corresponding image type can be selected based on the application area of ​​the super-resolution model 121.

[0050] For example, when applying the super-resolution model 121 to an endoscope system, endoscope images can be used as training data. Low-resolution images are acquired by using high-resolution images, reducing their resolution, adding noise, and compressing them using software.

[0051] Next, the training data is input into the super-resolution model 121 for training. Low-resolution images are used as input, the output of the super-resolution model 121 is used as output, and high-resolution images are used as labeled training data.

[0052] This iterative learning process yields a trained super-resolution model 121, which can then be used in image processing to convert low-resolution images into high-resolution images.

[0053] As described above, the image processing method and electronic device of the present invention do not require reliance on expensive hardware, and by using artificial intelligence (AI) software calculations, high-resolution images that could previously only be obtained with high-end hardware can be realized. The resulting high-resolution images have sharp edges and are less prone to blurring.

[0054] Therefore, when the present invention is applied to endoscopic images, high-resolution images from 4K to 8K can be acquired with a device equipped with a 1080P output source, effectively providing physicians with more accurate reference information. Furthermore, the computing level of the processing unit used by the electronic device only requires that the built-in graphics card of the electronic computer runs smoothly.

[0055] The embodiments described above are merely illustrative of the technical idea and features of the present application. Their purpose is to enable those skilled in the art to understand and implement the present invention accordingly, and they should not be used to limit the technical scope of the present application. That is, all equivalent changes or modifications made in accordance with the spirit expressed herein remain within the scope of this patent application. [Explanation of Symbols]

[0056] 10 Electronic equipment 12 Processing Unit 121 Super-Resolution Models 14 Storage devices 16 Display device 18 Test equipment S10~S18 Step

Claims

1. An image processing method using a processing device and a storage device, The processing device inputs the low-resolution image stored in the storage device into the super-resolution model. The processing device generates a first intermediate image by performing a difference process on the low-resolution image using the super-resolution model. The processing apparatus generates a second intermediate image by performing a convolution activation process on the low-resolution image using the super-resolution model. The processing device generates a high-resolution image by performing an addition operation on the first intermediate image and the second intermediate image. An image processing method characterized by the following features.

2. The aforementioned low-resolution image is a real-time image generated by a testing device that inspects the target object. The image processing method according to claim 1, characterized in that

3. The aforementioned test apparatus is an endoscope system, The aforementioned low-resolution image is a low-resolution endoscopic image. The aforementioned high-resolution image is a high-resolution endoscopic image. The image processing method according to claim 2, characterized in that...

4. The aforementioned low-resolution image is a low-resolution endoscopic image. The aforementioned high-resolution image is a high-resolution endoscopic image. The image processing method according to claim 1, characterized in that

5. The difference processing generates the second intermediate image by performing processing on the low-resolution image using a bilinear interpolation method. The image processing method according to claim 1, characterized in that

6. The aforementioned convolutional activation process includes performing detailed processing on multiple convolutional layers and activation function layers to generate the second intermediate image. The image processing method according to claim 1, characterized in that

7. The first intermediate image and the second intermediate image have the same image size. The image processing method according to claim 1, characterized in that

8. The super-resolution model can perform a convolution activation process on the low-resolution image, and then further perform a depth space transformation process. The generated second intermediate image has the same image size as the first intermediate image. The image processing method according to claim 7, characterized in that...

9. An electronic device including a storage device and a processing device, The storage device stores low-resolution images, The aforementioned processing device is electrically connected to the storage device and incorporates a super-resolution model. The processing device inputs the low-resolution image to the super-resolution model, uses the super-resolution model to perform difference processing and convolution activation processing on the low-resolution image, respectively, thereby generating a first intermediate image and a second intermediate image, and simultaneously performs addition operations on the first intermediate image and the second intermediate image to generate a high-resolution image. An electronic device characterized by the following features.

10. The aforementioned electronic device is electrically connected to the test apparatus. The aforementioned testing apparatus inspects the object and generates real-time images, and The real-time image is the low-resolution image. The electronic device according to claim 9, characterized in that...

11. The aforementioned test apparatus is an endoscope system, The aforementioned low-resolution image is a low-resolution endoscopic image. The aforementioned high-resolution image is a high-resolution endoscopic image. The electronic device according to claim 10, characterized in that...

12. The aforementioned low-resolution image is a low-resolution endoscopic image. The aforementioned high-resolution image is a high-resolution endoscopic image. The electronic device according to claim 10, characterized in that...

13. The super-resolution model generates the second intermediate image by performing the difference processing on the low-resolution image using a bilinear interpolation method. The electronic device according to claim 9, characterized in that...

14. The convolutional activation process further includes performing detailed processing on multiple convolutional layers and activation function layers to generate the second intermediate image. The electronic device according to claim 9, characterized in that...

15. The first intermediate image and the second intermediate image have the same image size. The electronic device according to claim 9, characterized in that...

16. The super-resolution model can perform a convolution activation process on the low-resolution image, and then further perform a depth space transformation process. The generated second intermediate image has the same image size as the first intermediate image. The electronic device according to claim 15, characterized in that...

Images