Endoscope-based super-resolution image generation method, device and electronic equipment

Abstract

The present invention provides an endoscope-based super-resolution image generation method, device, and electronic device, relating to the field of endoscope technology. The method comprises obtaining an endoscope-enhanced image; separating the endoscope-enhanced image into an illumination image and a reflection image; performing super-resolution enhancement on the illumination image and the reflection image to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image, respectively; and restoring the endoscope super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image. The present invention can perform super-resolution enhancement on the illumination information and reflection information in the image separately, improving the problems of artifacts and poor reconstruction while ensuring processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopes, and in particular to an endoscope-based super-resolution image generation method, device and electronic equipment. Background Art

[0002] With the development of technologies in fields such as materials, mechanics, electronics, optics, and imaging, the pace of replacement of medical device products is gradually accelerating. In the field of endoscopy, traditional multiplexed endoscopes are gradually being replaced by disposable endoscopes because they use relatively expensive imaging structural materials and significantly increase the probability of cross-infection among patients. However, the performance of current disposable endoscopes is far from reaching the level of multiplexed endoscopes, especially in terms of imaging quality. Multiplexed products use high-resolution, high-signal-to-noise ratio imaging chips (such as CCD sensors), and their imaging performance is significantly better than that of disposable endoscopes. In addition, in application areas such as ureteroscopy and bronchoscopy, the use of higher-resolution imaging sensors will significantly increase the structural size, which will significantly affect the performance of the endoscope. Therefore, super-resolution enhancement of low-resolution sensors has very high application value in the field of endoscopic image processing.

[0003] Although existing technologies have applied super-resolution enhancement technology to endoscopic imaging, they still have the following problems:

[0004] 1. Super-resolution imaging methods use expensive imaging structures such as high signal-to-noise ratio sensors, which limits the application scope of super-resolution;

[0005] 2. Super-resolution imaging methods that use inexpensive imaging structures such as monocular sensors do not fully consider the lighting and imaging scene characteristics of the endoscope, making it difficult to achieve reliable super-resolution enhancement effects. In other words, super-resolution methods produce undesirable features such as image artifacts, resulting in unacceptable consequences for medical images such as image distortion.

[0006] Therefore, a method, device and electronic device for generating super-resolution images based on an endoscope are proposed. Summary of the Invention

[0007] This specification provides an endoscope-based super-resolution image generation method, device and electronic equipment, which can perform super-resolution enhancement on the illumination information and reflection information in the image respectively, and improve the problems of artifacts and poor reconstruction while ensuring processing efficiency.

[0008] This specification provides a method for generating super-resolution images based on an endoscope, comprising:

[0009] Acquiring endoscopic enhanced images;

[0010] separating the endoscope enhanced image into an illumination image and a reflection image;

[0011] Performing super-resolution enhancement on the illumination image and the reflection image to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image respectively;

[0012] An endoscopic super-resolution image is restored based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image.

[0013] Optionally, performing super-resolution enhancement on the illumination image to obtain an illumination super-resolution enhanced image includes:

[0014] Inputting the illumination image into the convolution layer for feature extraction to obtain features of different receptive fields;

[0015] Using residual dense blocks of different scales to identify the features of the different receptive fields, a preliminary result feature matrix is ​​obtained;

[0016] Inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix;

[0017] The intermediate result matrix is ​​input into the upsampling network for amplification processing to obtain an illumination super-resolution enhanced image.

[0018] Optionally, performing super-resolution enhancement on the reflected image to obtain a reflected super-resolution enhanced image includes:

[0019] Performing bicubic linear interpolation on the reflected image to obtain a second intermediate result matrix;

[0020] The second intermediate result matrix is ​​input into the convolution layer to obtain a reflection super-resolution enhanced image.

[0021] Optionally, the restoring the endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image includes:

[0022] Gamma enhancement is performed on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image to restore an endoscopic super-resolution image.

[0023] Optionally, the residual dense block includes a primary residual dense block convolution layer with a stride of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a stride of 2 and a convolution kernel of 3×3, a ReLU residual dense block convolution layer with a stride of 4 and a convolution kernel of 5×5, an upsampling operation layer with an upsampling multiple of 2, an upsampling operation layer with an upsampling multiple of 4, a convolution kernel of 3×3 and a magnification of 2 dilated convolution, and a convolution kernel of 3×3 and a magnification of 2 dilated convolution.

[0024] Optionally, the features of the different receptive fields are identified using residual dense blocks of different scales to obtain a preliminary result feature matrix, including:

[0025] The features of the different receptive fields are respectively passed through a primary residual dense block convolution layer with a step size of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a step size of 2 and a convolution kernel of 3×3, and a ReLU residual dense block convolution layer with a step size of 4 and a convolution kernel of 5×5, to obtain a feature matrix with a size of 1 / 2 of the endoscopic enhanced image and a feature matrix with a size of 1 / 4 of the endoscopic enhanced image;

[0026] Passing the feature matrix having a size of 1 / 2 of the endoscopic enhanced image and the feature matrix having a size of 1 / 4 of the endoscopic enhanced image through the upsampling operation layer with an upsampling factor of 2 and the upsampling operation layer with an upsampling factor of 4, respectively, to obtain a feature matrix enlarged by 2 times and a feature matrix enlarged by 4 times;

[0027] The feature matrix enlarged by 2 times and the feature matrix enlarged by 4 times are respectively subjected to dilated convolution with a convolution kernel of 3×3 and a magnification of 2, and dilated convolution with a convolution kernel of 3×3 and a magnification of 2 to obtain preliminary result feature matrices.

[0028] Optionally, inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix includes:

[0029] The dense feature fusion layer includes a convolution layer with a step size of 1 and a convolution kernel of 1×1, a convolution layer with a step size of 1 and a convolution kernel of 3×3, and a convolution layer with a step size of 1 and a convolution kernel of 5×5;

[0030] Pass the preliminary result feature matrix through the convolution layer with a step size of 1 and a convolution kernel of 1×1, the convolution layer with a step size of 1 and a convolution kernel of 3×3, and the convolution layer with a step size of 1 and a convolution kernel of 5×5, respectively, to obtain a multi-scale result feature matrix;

[0031] The multi-scale result feature moments are connected to form a first intermediate result matrix.

[0032] This specification provides an endoscope-based super-resolution image generation device, comprising:

[0033] an acquisition module, for acquiring an endoscope enhanced image;

[0034] a separation module, configured to separate the endoscope enhanced image into an illumination image and a reflection image;

[0035] an enhancement module, configured to perform super-resolution enhancement on the illumination image and the reflection image, respectively, to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image;

[0036] A restoration module is used to restore the endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image.

[0037] Optionally, performing super-resolution enhancement on the illumination image to obtain an illumination super-resolution enhanced image includes:

[0038] Inputting the illumination image into the convolution layer for feature extraction to obtain features of different receptive fields;

[0039] Using residual dense blocks of different scales to identify the features of the different receptive fields, a preliminary result feature matrix is ​​obtained;

[0040] Inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix;

[0041] The intermediate result matrix is ​​input into the upsampling network for amplification processing to obtain an illumination super-resolution enhanced image.

[0042] Optionally, performing super-resolution enhancement on the reflected image to obtain a reflected super-resolution enhanced image includes:

[0043] Performing bicubic linear interpolation on the reflected image to obtain a second intermediate result matrix;

[0044] The second intermediate result matrix is ​​input into the convolution layer to obtain a reflection super-resolution enhanced image.

[0045] Optionally, the restoring the endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image includes:

[0046] Gamma enhancement is performed on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image to restore an endoscopic super-resolution image.

[0047] Optionally, the residual dense block includes a primary residual dense block convolution layer with a stride of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a stride of 2 and a convolution kernel of 3×3, a ReLU residual dense block convolution layer with a stride of 4 and a convolution kernel of 5×5, an upsampling operation layer with an upsampling multiple of 2, an upsampling operation layer with an upsampling multiple of 4, a convolution kernel of 3×3 and a magnification of 2 dilated convolution, and a convolution kernel of 3×3 and a magnification of 2 dilated convolution.

[0048] Optionally, the features of the different receptive fields are identified using residual dense blocks of different scales to obtain a preliminary result feature matrix, including:

[0049] The features of the different receptive fields are respectively passed through a primary residual dense block convolution layer with a step size of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a step size of 2 and a convolution kernel of 3×3, and a ReLU residual dense block convolution layer with a step size of 4 and a convolution kernel of 5×5, to obtain a feature matrix with a size of 1 / 2 of the endoscopic enhanced image and a feature matrix with a size of 1 / 4 of the endoscopic enhanced image;

[0050] Passing the feature matrix having a size of 1 / 2 of the endoscopic enhanced image and the feature matrix having a size of 1 / 4 of the endoscopic enhanced image through the upsampling operation layer with an upsampling factor of 2 and the upsampling operation layer with an upsampling factor of 4, respectively, to obtain a feature matrix enlarged by 2 times and a feature matrix enlarged by 4 times;

[0051] The feature matrix enlarged by 2 times and the feature matrix enlarged by 4 times are respectively subjected to dilated convolution with a convolution kernel of 3×3 and a magnification of 2, and dilated convolution with a convolution kernel of 3×3 and a magnification of 2 to obtain preliminary result feature matrices.

[0052] Optionally, inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix includes:

[0053] The dense feature fusion layer includes a convolution layer with a step size of 1 and a convolution kernel of 1×1, a convolution layer with a step size of 1 and a convolution kernel of 3×3, and a convolution layer with a step size of 1 and a convolution kernel of 5×5;

[0054] Pass the preliminary result feature matrix through the convolution layer with a step size of 1 and a convolution kernel of 1×1, the convolution layer with a step size of 1 and a convolution kernel of 3×3, and the convolution layer with a step size of 1 and a convolution kernel of 5×5, respectively, to obtain a multi-scale result feature matrix;

[0055] The multi-scale result feature moments are connected to form a first intermediate result matrix.

[0056] This specification also provides an electronic device, wherein the electronic device includes:

[0057] A processor; and a memory storing processor-executable messages, wherein the executable messages, when executed, cause the processor to perform any of the methods described above.

[0058] This specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, any of the above methods is implemented.

[0059] In the present invention, super-resolution enhancement can be performed on the illumination information and reflection information in the disposable endoscope image respectively, thereby improving the problems of artifacts and poor reconstruction while ensuring processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0061] Figure 1 A schematic diagram of the principle of an endoscope-based super-resolution image generation method provided in an embodiment of this specification;

[0062] Figure 2 A schematic diagram of the structures of two image processing systems provided in the embodiments of this specification;

[0063] Figure 3 A diagram of a network structure for super-resolution enhancement of illuminated images provided in an embodiment of this specification;

[0064] Figure 4 A schematic structural diagram of an endoscope-based super-resolution image generation device provided in an embodiment of this specification;

[0065] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification;

[0066] Figure 6 A schematic diagram of a computer-readable medium provided in accordance with an embodiment of this specification. DETAILED DESCRIPTION

[0067] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0068] The following is combined with Figure 1-6The exemplary embodiments of the present invention are described more fully. However, the exemplary embodiments can be implemented in various forms, and the present invention should not be construed as being limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, and more conveniently conveys the inventive concept to those skilled in the art. In the figures, the same reference numerals represent the same or similar elements, components, or parts, and thus their repeated description will be omitted.

[0069] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0070] In the description of specific embodiments, the features, structures, characteristics, or other details of the present invention are described to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from practicing the technical solutions of the present invention without one or more of the specific features, structures, characteristics, or other details.

[0071] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0072] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0073] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0074] Figure 1 A schematic diagram of the principle of an endoscope-based super-resolution image generation method provided in an embodiment of this specification may include:

[0075] S110: Acquire an endoscope enhanced image;

[0076] In a specific embodiment of the present invention, Figure 2As shown, the image processing system includes an image acquisition unit, an image processor, and a display. The image acquisition unit captures images of the disposable endoscope inspection scene, decodes them using a specific format (such as MIPI), and generates a processed image. This processed image is then input to the SOC controller in the image processor, where the FPGA initiates CFA, curve enhancement, defogging, and time-domain filtering operations to generate an enhanced endoscope image.

[0077] S120: Separating the endoscope enhanced image into an illumination image and a reflection image;

[0078] In a specific embodiment of the present invention, Retinex is used to separate the enhanced endoscope image into an illuminated image and a reflected image, representing the high-frequency texture and light and dark structures in the image, respectively. Retinex is a commonly used image enhancement algorithm whose core concept is to adjust the contrast and brightness of the image while preserving image detail.

[0079] S130: Performing super-resolution enhancement on the illumination image and the reflection image respectively to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image;

[0080] like Figure 2 As shown, the illumination image and the reflection image enter the illumination super-resolution enhancement network and the reflection super-resolution enhancement network deployed on the FPGA respectively, and obtain the illumination super-resolution enhanced image and the reflection super-resolution enhanced image respectively.

[0081] Of course, the illumination image and the reflection image can also be transferred into the DDR memory via the SOC controller or directly through DMA, and forwarded to the neural network computing module (including GPU, TPU, NPU and other computing architectures) via the SOC controller or DMA to perform super-resolution enhancement to obtain illumination super-resolution enhanced images and reflection super-resolution enhanced images.

[0082] Optionally, performing super-resolution enhancement on the illumination image to obtain an illumination super-resolution enhanced image includes:

[0083] Inputting the illumination image into the convolution layer for feature extraction to obtain features of different receptive fields;

[0084] Using residual dense blocks of different scales to identify the features of the different receptive fields, a preliminary result feature matrix is ​​obtained;

[0085] Inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix;

[0086] The intermediate result matrix is ​​input into the upsampling network for amplification processing to obtain an illumination super-resolution enhanced image.

[0087] Optionally, the residual dense block includes a primary residual dense block convolution layer with a stride of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a stride of 2 and a convolution kernel of 3×3, a ReLU residual dense block convolution layer with a stride of 4 and a convolution kernel of 5×5, an upsampling operation layer with an upsampling multiple of 2, an upsampling operation layer with an upsampling multiple of 4, a convolution kernel of 3×3 and a magnification of 2 dilated convolution, and a convolution kernel of 3×3 and a magnification of 2 dilated convolution.

[0088] Optionally, the features of the different receptive fields are identified using residual dense blocks of different scales to obtain a preliminary result feature matrix, including:

[0089] The features of the different receptive fields are respectively passed through a primary residual dense block convolution layer with a step size of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a step size of 2 and a convolution kernel of 3×3, and a ReLU residual dense block convolution layer with a step size of 4 and a convolution kernel of 5×5, to obtain a feature matrix with a size of 1 / 2 of the endoscopic enhanced image and a feature matrix with a size of 1 / 4 of the endoscopic enhanced image;

[0090] Passing the feature matrix having a size of 1 / 2 of the endoscopic enhanced image and the feature matrix having a size of 1 / 4 of the endoscopic enhanced image through the upsampling operation layer with an upsampling factor of 2 and the upsampling operation layer with an upsampling factor of 4, respectively, to obtain a feature matrix enlarged by 2 times and a feature matrix enlarged by 4 times;

[0091] The feature matrix enlarged by 2 times and the feature matrix enlarged by 4 times are respectively subjected to dilated convolution with a convolution kernel of 3×3 and a magnification of 2, and dilated convolution with a convolution kernel of 3×3 and a magnification of 2 to obtain preliminary result feature matrices.

[0092] Optionally, inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix includes:

[0093] The dense feature fusion layer includes a convolution layer with a step size of 1 and a convolution kernel of 1×1, a convolution layer with a step size of 1 and a convolution kernel of 3×3, and a convolution layer with a step size of 1 and a convolution kernel of 5×5;

[0094] Pass the preliminary result feature matrix through the convolution layer with a step size of 1 and a convolution kernel of 1×1, the convolution layer with a step size of 1 and a convolution kernel of 3×3, and the convolution layer with a step size of 1 and a convolution kernel of 5×5, respectively, to obtain a multi-scale result feature matrix;

[0095] The multi-scale result feature moments are connected to form a first intermediate result matrix.

[0096] In a specific embodiment of the present invention, Figure 3 As shown, the upsampling network sets the upsampling factor based on the scale magnification. In disposable endoscopes, this magnification does not exceed 4. Taking a magnification of 2 as an example, the size of the first intermediate result matrix generated by the previous processing is doubled. A dilated convolution with a 3×3 kernel and a magnification of 2 and a dilated convolution with a 5×5 kernel and a magnification of 4 are used to collect features from the scaled feature matrix. The two dilated convolution results can be improved by removing max pooling and removing residual connections to obtain an illumination super-resolution enhanced image.

[0097] By using residual dense blocks of different scales, the receptive field of the feature extraction network can be effectively improved, the artifact problem caused by scale increase can be controlled, and the execution performance of the subsequent upsampling network can be improved.

[0098] Optionally, performing super-resolution enhancement on the reflected image to obtain a reflected super-resolution enhanced image includes:

[0099] Performing bicubic linear interpolation on the reflected image to obtain a second intermediate result matrix;

[0100] The second intermediate result matrix is ​​input into the convolution layer to obtain a reflection super-resolution enhanced image.

[0101] In a specific embodiment of the present invention, bicubic interpolation is a more complex interpolation method that can create smoother image edges than bilinear interpolation. Bicubic interpolation is commonly used in some image processing software, printer drivers, and digital cameras to magnify the original image or certain areas of the original image.

[0102] S140: Restoring an endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image.

[0103] Optionally, the S140 includes:

[0104] Gamma enhancement is performed on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image to restore an endoscopic super-resolution image.

[0105] In a specific embodiment of the present invention, Figure 2 As shown, the endoscope super-resolution image is further transmitted to the video output module for format conversion, and finally output to the display in a specific format (including DVI, VGA, LVDS, S-Video, etc.).

[0106] Of course, the illumination super-resolution enhanced image and the reflection super-resolution enhanced image can also be transmitted back to the FPGA by the neural network computing module, and subsequent operations such as gamma enhancement are completed. Finally, the result image is formed through image restoration, and the endoscopic super-resolution image is finally output to the display through the video output module.

[0107] In the present invention, super-resolution enhancement can be performed on the illumination information and reflection information in the disposable endoscope image respectively, thereby improving the problems of artifacts and poor reconstruction while ensuring processing efficiency.

[0108] Figure 4 This is a schematic diagram of the structure of an endoscope-based super-resolution image generation device provided in an embodiment of this specification. The device may include:

[0109] An acquisition module 10 is used to acquire an endoscope enhanced image;

[0110] a separation module 20 for separating the endoscope enhanced image into an illumination image and a reflection image;

[0111] an enhancement module 30 for performing super-resolution enhancement on the illumination image and the reflection image to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image respectively;

[0112] The restoration module 40 is configured to restore the endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image.

[0113] Optionally, performing super-resolution enhancement on the illumination image to obtain an illumination super-resolution enhanced image includes:

[0114] Inputting the illumination image into the convolution layer for feature extraction to obtain features of different receptive fields;

[0115] Using residual dense blocks of different scales to identify the features of the different receptive fields, a preliminary result feature matrix is ​​obtained;

[0116] Inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix;

[0117] The intermediate result matrix is ​​input into the upsampling network for amplification processing to obtain an illumination super-resolution enhanced image.

[0118] Optionally, performing super-resolution enhancement on the reflected image to obtain a reflected super-resolution enhanced image includes:

[0119] Performing bicubic linear interpolation on the reflected image to obtain a second intermediate result matrix;

[0120] The second intermediate result matrix is ​​input into the convolution layer to obtain a reflection super-resolution enhanced image.

[0121] Optionally, the restoring the endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image includes:

[0122] Gamma enhancement is performed on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image to restore an endoscopic super-resolution image.

[0123] Optionally, the residual dense block includes a primary residual dense block convolution layer with a stride of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a stride of 2 and a convolution kernel of 3×3, a ReLU residual dense block convolution layer with a stride of 4 and a convolution kernel of 5×5, an upsampling operation layer with an upsampling multiple of 2, an upsampling operation layer with an upsampling multiple of 4, a convolution kernel of 3×3 and a magnification of 2 dilated convolution, and a convolution kernel of 3×3 and a magnification of 2 dilated convolution.

[0124] Optionally, the features of the different receptive fields are identified using residual dense blocks of different scales to obtain a preliminary result feature matrix, including:

[0125] The features of the different receptive fields are respectively passed through a primary residual dense block convolution layer with a step size of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a step size of 2 and a convolution kernel of 3×3, and a ReLU residual dense block convolution layer with a step size of 4 and a convolution kernel of 5×5, to obtain a feature matrix with a size of 1 / 2 of the endoscopic enhanced image and a feature matrix with a size of 1 / 4 of the endoscopic enhanced image;

[0126] Passing the feature matrix having a size of 1 / 2 of the endoscopic enhanced image and the feature matrix having a size of 1 / 4 of the endoscopic enhanced image through the upsampling operation layer with an upsampling factor of 2 and the upsampling operation layer with an upsampling factor of 4, respectively, to obtain a feature matrix enlarged by 2 times and a feature matrix enlarged by 4 times;

[0127] The feature matrix enlarged by 2 times and the feature matrix enlarged by 4 times are respectively subjected to dilated convolution with a convolution kernel of 3×3 and a magnification of 2, and dilated convolution with a convolution kernel of 3×3 and a magnification of 2 to obtain preliminary result feature matrices.

[0128] Optionally, inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix includes:

[0129] The dense feature fusion layer includes a convolution layer with a step size of 1 and a convolution kernel of 1×1, a convolution layer with a step size of 1 and a convolution kernel of 3×3, and a convolution layer with a step size of 1 and a convolution kernel of 5×5;

[0130] Pass the preliminary result feature matrix through the convolution layer with a step size of 1 and a convolution kernel of 1×1, the convolution layer with a step size of 1 and a convolution kernel of 3×3, and the convolution layer with a step size of 1 and a convolution kernel of 5×5, respectively, to obtain a multi-scale result feature matrix;

[0131] The multi-scale result feature moments are connected to form a first intermediate result matrix.

[0132] The functions of the device in the embodiment of the present invention have been described in the above method embodiment. Therefore, for details not fully described in this embodiment, please refer to the relevant description in the above embodiment and will not be repeated here.

[0133] Based on the same inventive concept, an embodiment of this specification also provides an electronic device.

[0134] The following describes an electronic device embodiment of the present invention, which can be considered a specific physical implementation of the method and apparatus embodiments of the present invention described above. Details described in the electronic device embodiment of the present invention should be considered supplementary to the above-mentioned method or apparatus embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above-mentioned method or apparatus embodiments.

[0135] Figure 5 This is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Figure 5 The electronic device 300 according to this embodiment of the present invention will be described. Figure 5 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0136] like Figure 5 As shown, electronic device 300 is implemented as a general-purpose computing device. Components of electronic device 300 may include, but are not limited to, at least one processing unit 310, at least one storage unit 320, a bus 330 connecting various system components (including storage unit 320 and processing unit 310), a display unit 340, and the like.

[0137] The storage unit stores program codes that can be executed by the processing unit 310, so that the processing unit 310 performs the steps according to various exemplary embodiments of the present invention described in the above processing method section of this specification. For example, the processing unit 310 can perform the following steps: Figure 1 Steps shown.

[0138] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 3201 and / or a cache memory unit 3202 , and may further include a read-only memory unit (ROM) 3203 .

[0139] The storage unit 320 may also include a program / utility 3204 having a set (at least one) of program modules 3205, such program modules 3205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0140] Bus 330 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0141] The electronic device 300 may also communicate with one or more external devices 400 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable viewers to interact with the electronic device 300, and / or any device that enables the electronic device 300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 350. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 360. The network adapter 360 may communicate with other modules of the electronic device 300 through the bus 330. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] Through the description of the above embodiments, it is easy for those skilled in the art to understand. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of messages to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the present invention. When the computer program is executed by a data processing device, the computer-readable medium can implement the above method of the present invention, that is: Figure 1 The method shown.

[0143] Figure 6 A schematic diagram of a computer-readable medium provided in accordance with an embodiment of this specification.

[0144] accomplish Figure 1 The computer program of the method shown can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0145] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a message execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0146] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the viewer computing device, partially on the viewer device, as a stand-alone software package, partially on the viewer computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the viewer computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0147] In summary, the present invention can be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that general data processing equipment such as a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program for implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0148] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0149] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0150] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for generating super-resolution images based on endoscope, characterized in that: include: Acquiring endoscopic enhanced images; separating the endoscope enhanced image into an illumination image and a reflection image; The illumination image and the reflection image are respectively subjected to super-resolution enhancement to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image, including: inputting the illumination image into a convolution layer for feature extraction to obtain features of different receptive fields; using residual dense blocks of different scales to identify the features of the different receptive fields to obtain a preliminary result feature matrix; inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix; inputting the intermediate result matrix into an upsampling network for amplification processing to obtain an illumination super-resolution enhanced image; performing bicubic linear interpolation on the reflection image to obtain a second intermediate result matrix; and inputting the second intermediate result matrix into a convolution layer to obtain a reflection super-resolution enhanced image. Restoring an endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image includes: performing Gamma enhancement on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image to restore the endoscopic super-resolution image.

2. The endoscope-based super-resolution image generation method according to claim 1, wherein: The residual dense block includes a primary residual dense block convolution layer with a step size of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a step size of 2 and a convolution kernel of 3×3, a ReLU residual dense block convolution layer with a step size of 4 and a convolution kernel of 5×5, an upsampling operation layer with an upsampling multiple of 2, an upsampling operation layer with an upsampling multiple of 4, a convolution kernel of 3×3 and a magnification of 2 hollow convolution, and a convolution kernel of 3×3 and a magnification of 2 hollow convolution.

3. The endoscope-based super-resolution image generation method according to claim 2, wherein: The features of the different receptive fields are identified by using residual dense blocks of different scales to obtain a preliminary result feature matrix, including: The features of the different receptive fields are respectively passed through a primary residual dense block convolution layer with a step size of 1 and a convolution kernel of 1×1, an intermediate residual dense block convolution layer with a step size of 2 and a convolution kernel of 3×3, and a ReLU residual dense block convolution layer with a step size of 4 and a convolution kernel of 5×5, to obtain a feature matrix with a size of 1 / 2 of the endoscopic enhanced image and a feature matrix with a size of 1 / 4 of the endoscopic enhanced image; Passing the feature matrix having a size of 1 / 2 of the endoscopic enhanced image and the feature matrix having a size of 1 / 4 of the endoscopic enhanced image through the upsampling operation layer with an upsampling factor of 2 and the upsampling operation layer with an upsampling factor of 4, respectively, to obtain a feature matrix enlarged by 2 times and a feature matrix enlarged by 4 times; The feature matrix enlarged by 2 times and the feature matrix enlarged by 4 times are respectively subjected to dilated convolution with a convolution kernel of 3×3 and a magnification of 2, and dilated convolution with a convolution kernel of 3×3 and a magnification of 2 to obtain preliminary result feature matrices.

4. The endoscope-based super-resolution image generation method according to claim 3, wherein: The step of inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix includes: The dense feature fusion layer includes a convolution layer with a step size of 1 and a convolution kernel of 1×1, a convolution layer with a step size of 1 and a convolution kernel of 3×3, and a convolution layer with a step size of 1 and a convolution kernel of 5×5; Pass the preliminary result feature matrix through the convolution layer with a step size of 1 and a convolution kernel of 1×1, the convolution layer with a step size of 1 and a convolution kernel of 3×3, and the convolution layer with a step size of 1 and a convolution kernel of 5×5, respectively, to obtain a multi-scale result feature matrix; The multi-scale result feature moments are connected to form a first intermediate result matrix.

5. An endoscope-based super-resolution image generation device, characterized in that: include: an acquisition module, for acquiring an endoscope enhanced image; a separation module, configured to separate the endoscope enhanced image into an illumination image and a reflection image; An enhancement module is used to perform super-resolution enhancement on the illumination image and the reflection image, respectively, to obtain an illumination super-resolution enhanced image and a reflection super-resolution enhanced image, including: inputting the illumination image into a convolution layer for feature extraction to obtain features of different receptive fields; using residual dense blocks of different scales to identify the features of the different receptive fields to obtain a preliminary result feature matrix; inputting the preliminary result feature matrix into a dense feature fusion layer to obtain a first intermediate result matrix; inputting the intermediate result matrix into an upsampling network for amplification processing to obtain an illumination super-resolution enhanced image; performing bicubic linear interpolation on the reflection image to obtain a second intermediate result matrix; and inputting the second intermediate result matrix into a convolution layer to obtain a reflection super-resolution enhanced image. A restoration module is used to restore an endoscopic super-resolution image based on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image, including: performing Gamma enhancement on the illumination super-resolution enhanced image and the reflection super-resolution enhanced image to restore the endoscopic super-resolution image.

6. An electronic device, wherein: The electronic device includes: A processor; and a memory storing processor-executable messages, which, when executed, cause the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method of any one of claims 1 to 4 is implemented.

Images