Endoscope image correction method and related device

Abstract

The invention provides an endoscope image correction method and a related device, and the method comprises the steps: determining a mirror reflection region of an endoscope image, and obtaining a shielded image based on the mirror reflection region; correcting the shielded image to obtain a first corrected image of the endoscope; performing artifact removal on the first corrected image of the endoscope to obtain a second corrected image of the endoscope; and performing weighted fusion based on the first corrected image of the endoscope and the second corrected image of the endoscope to obtain a third corrected image of the endoscope. According to the method and the device, the problems of poor image correction quality, low accuracy, low stability and high training cost during endoscope image correction can be solved.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, and in particular to an endoscopic image correction method and related apparatus. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] An endoscope is a long, thin tubular instrument with an optical lens and a light source. When it is inserted into a human cavity or body cavity, the light source illuminates the target area, and the optical lens is responsible for capturing the image information of that area. However, the endoscopic imaging process is easily affected by a variety of factors, resulting in problems such as image distortion, noise, and insufficient contrast. Therefore, it is necessary to correct the endoscopic image.

[0004] However, in related technologies, there are problems such as poor image correction quality, inadequate accuracy, unstable stability, and excessively high training costs when correcting endoscopic images. Summary of the Invention

[0005] In view of this, the purpose of this disclosure is to propose an endoscopic image correction method and related apparatus, which at least to some extent solves one of the technical problems in the related art.

[0006] To achieve the above objectives, a first aspect of the exemplary embodiments of this disclosure provides an endoscopic image correction method, the method comprising:

[0007] Determine the specular reflection area of ​​the endoscopic image, and obtain the masked image based on the specular reflection area;

[0008] The obscured image is corrected to obtain the first corrected endoscopic image;

[0009] Artifact removal is performed on the first corrected image of the endoscope to obtain the second corrected image of the endoscope;

[0010] The endoscope third corrected image is obtained by weighted fusion of the first corrected image and the second corrected image.

[0011] Based on the same inventive concept, a second aspect of the exemplary embodiments of this disclosure provides an endoscopic image correction apparatus, comprising:

[0012] The obscured image determination module is configured to determine the specular reflection area of ​​the endoscope image and obtain the obscured image based on the specular reflection area;

[0013] The first corrected image determination module is configured to correct the occluded image to obtain the first corrected endoscope image;

[0014] The second corrected image determination module is configured to remove artifacts from the first corrected image of the endoscope to obtain the second corrected image of the endoscope.

[0015] The third corrected image determination module is configured to perform weighted fusion based on the first corrected image of the endoscope and the second corrected image of the endoscope to obtain the third corrected image of the endoscope.

[0016] Based on the same inventive concept, a third aspect of the exemplary embodiments of this disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in the first aspect.

[0017] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described in the first aspect.

[0018] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of this disclosure provides a computer program product including computer program instructions that, when run on a computer, cause the computer to perform the method as described in the first aspect.

[0019] As can be seen from the above description, the endoscopic image correction method and related apparatus provided in this disclosure include: determining the specular reflection area of ​​an endoscope image; obtaining an occluded image based on the specular reflection area; correcting the occluded image to obtain a first corrected endoscopic image; removing artifacts from the first corrected endoscopic image to obtain a second corrected endoscopic image; and performing weighted fusion of the first corrected endoscopic image and the second corrected endoscopic image to obtain a third corrected endoscopic image. This disclosure can solve the problems of poor image correction quality, inadequate accuracy, unstable stability, and excessively high training costs in endoscopic image correction. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A schematic diagram illustrating an application scenario of the endoscopic image correction method provided as an exemplary embodiment of this disclosure;

[0022] Figure 2 A schematic flowchart of an endoscopic image correction method provided for an exemplary embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of an endoscopic image correction device provided as an exemplary embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of the structure of an electronic device hardware provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this application's technical solution, based on the prompt message.

[0027] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this application. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0029] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0030] To make the objectives, technical solutions, and advantages of this disclosure clearer, the principles and spirit of this disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0031] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0032] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "a" or "an" preceding an element does not exclude the existence of multiple such elements.

[0033] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments.

[0034] As described in the background section, related technologies suffer from problems such as poor image correction quality, inadequate accuracy, unstable performance, and excessively high training costs when correcting endoscopic images. Specifically, current methods for detecting specular reflection regions in endoscopic images mainly fall into three categories: color space-based methods, deep learning-based methods, and methods based on dual-color reflectance models. However, due to significant differences in brightness, contrast, and tissue texture between images acquired by different types of medical devices, existing methods generally suffer from poor versatility, high computational cost, and high false detection rates. Most methods are only effective under specific conditions (such as moderate brightness and flat texture), and perform poorly under low illumination, high exposure, or complex textured surface conditions, making it difficult to adapt to diverse endoscopic image sequences and color variations.

[0035] Among color space-based methods, these approaches analyze the distribution characteristics of images in different color spaces (such as Hue-Saturation-Value (HSV), color coding (YUV, where Y represents the luminance component and U and V represent the chrominance components), and Red-Green-Blue (RGB)) to detect potential highlight areas. They have relatively low computational complexity and certain advantages in practical applications. However, because they heavily rely on empirical parameters, they cannot achieve adaptive adjustment in different scenarios, leading to a significant decrease in detection accuracy and insufficient robustness when the overall image brightness is too high or too low.

[0036] In methods based on the two-color reflectance model, the reflectance component in an image is decomposed into diffuse reflection and specular reflection, representing the intrinsic color of the object and specular highlights, respectively. This method is widely used in reflectance modeling of natural images, but it has limitations in endoscopic images. Commonly bright areas in endoscopic images are mainly composed of strong specular reflection caused by mucus, with extremely low or even absent diffuse reflection components. This prevents the two-color reflectance model from effectively separating the reflectance components, thus affecting the detection and restoration of highlights.

[0037] Among deep learning-based methods, deep learning has been widely used in recent years to recover specular reflection regions. However, its training process is complex and usually relies on a large amount of manually labeled training data, which is not conducive to model development.

[0038] In summary, existing methods mostly focus on single tasks, lacking the ability to collaboratively process various endoscopic imaging defects. Furthermore, they exhibit weak generalization ability, high false positive rates, and high resource consumption when dealing with variations in image color, brightness, and structure, making them difficult to meet clinical needs. Although deep learning methods offer higher recognition accuracy under complex conditions and do not rely on traditional model assumptions, they are highly dependent on large amounts of labeled data, making training complex and costly, thus limiting practical applications.

[0039] To address the aforementioned problems, this disclosure provides an endoscopic image correction method and related apparatus, the method comprising:

[0040] The method involves identifying the specular reflection region in an endoscopic image, obtaining an occluded image based on the specular reflection region, correcting the occluded image to obtain a first corrected endoscopic image, removing artifacts from the first corrected endoscopic image to obtain a second corrected endoscopic image, and performing a weighted fusion of the first and second corrected endoscopic images to obtain a third corrected endoscopic image. This method not only effectively repairs overexposed specular reflection regions in images, improving image quality, but also enhances the accuracy and stability of subsequent computer vision tasks (such as segmentation), while reducing reliance on large amounts of training data, decreasing model development costs, and improving the feasibility and efficiency of clinical applications.

[0041] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0042] refer to Figure 1 This is a schematic diagram illustrating an application scenario of the endoscopic image correction method provided in an exemplary embodiment of this disclosure.

[0043] This application scenario includes an endoscope 101, a terminal device 102, and a server 103. The endoscope 101, terminal device 102, and server 103 can all be connected via wired or wireless communication networks to achieve data exchange.

[0044] Terminal device 102 may be an electronic device located close to the user side, possessing data transmission and multimedia input / output functions, including but not limited to desktop computers, mobile phones, portable computers, tablet computers, media players, smart wearable devices, personal digital assistants (PDAs), or other electronic devices capable of performing the aforementioned functions. This electronic device may include a processor and a display screen with touch input functionality. The display screen is used to present a graphical user interface (GUI), which can display an application interface. The processor is used to process application data, generate the GUI, and control the display of the GUI on the screen.

[0045] Server 103 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0046] In some exemplary embodiments, the endoscopic image correction method may be run on terminal device 102 or server 103.

[0047] When the endoscope image correction method is running on server 103, server 103 is used to provide endoscope image correction services to users of terminal device 102.

[0048] Server 103 determines the specular reflection area of ​​the endoscope image, wherein the specular reflection area of ​​the endoscope image is obtained by the endoscope 101, and server 103 obtains the obscured image based on the specular reflection area;

[0049] Server 103 corrects the obscured image to obtain the first corrected endoscopic image;

[0050] Server 103 performs artifact removal on the first corrected image of the endoscope to obtain the second corrected image of the endoscope.

[0051] After performing weighted fusion on the first and second corrected images of the endoscope to obtain the third corrected image of the endoscope, the server 103 transmits the third corrected image of the endoscope to the terminal device 102.

[0052] It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the implementation of this disclosure is not limited in any way. On the contrary, the implementation of this disclosure can be applied to any applicable scenario.

[0053] refer to Figure 2 An endoscopic image correction method, the method comprising the following steps:

[0054] Step S210: Determine the specular reflection area of ​​the endoscope image, and obtain the masked image based on the specular reflection area.

[0055] In this step, by accurately identifying specular reflection areas, it is possible to clearly determine which parts of the image are affected by specular reflection. Specular reflection areas typically exceed the dynamic range of the imaging sensor due to excessively high light intensity, leading to loss of image details. Generating an occluded image based on these specular reflection areas provides a clear target region for subsequent image inpainting, allowing the inpainting algorithm to focus on these damaged parts and thus more effectively restore the image's structure and details.

[0056] In practice, endoscopic images refer to:

[0057] An endoscope is a medical device used to examine internal organs. It typically consists of an insertion section, an operating section, and an image processing system. The endoscope is inserted into the area to be examined, and the operating section controls its direction and position, allowing for clear observation of the target tissue. The endoscope's imaging system captures images of the examined area in real time. These images undergo preliminary processing by the endoscope's image processing system before being transmitted to an external display device.

[0058] As a specific embodiment, endoscopic image I can be represented in the following form:

[0059] I=∑k a (x,y)|L|+∑k s (x,y)|L|+∑k d (x,y)(N(x,y)·L);

[0060] Where, k a (x,y) represents the ambient light reflection component, k s (x,y) represents the specular reflection coefficient, k d (x,y) represents the diffuse reflection coefficient, N(x,y) is the surface normal vector, and L is the illumination direction.

[0061] Among them, the ambient light reflectance component refers to:

[0062] The ambient light reflectance component refers to the amount of light reflected from an object's surface under ambient light (i.e., scattered light from indirect light sources). It reflects the basic brightness level of an object's surface under the influence of surrounding ambient light and is contrasted with the reflectance component from direct light sources.

[0063] The specular reflection coefficient refers to:

[0064] The specular reflectance coefficient is a parameter that refers to the ability of an object's surface to reflect specular light under specific lighting conditions.

[0065] The diffuse emission coefficient refers to:

[0066] The specular reflection coefficient is a parameter that characterizes the ability of an object's surface to reflect specular light. It reflects the proportion of incident light reflected by the object's surface under specific lighting conditions.

[0067] Among them, surface normal vector information refers to:

[0068] This describes the vector data of the normal direction of an object's surface at each point. A normal is a vector perpendicular to the surface. Surface normal vector information allows for a precise description of the object's surface geometry and directional characteristics, enabling the calculation of lighting reflection effects, including specular and diffuse reflection, in lighting models.

[0069] In practical implementation, the specular reflection area refers to:

[0070] In endoscopic images, the smooth surface of tissues causes specular reflection of light, resulting in bright and overexposed areas. These areas typically have high light intensity, exceeding the dynamic range of the imaging sensor, leading to loss of image detail and noticeable artifacts.

[0071] In some embodiments, obtaining the masked image based on the specular reflection area includes:

[0072] A pixel-level binary mask is determined for the specular reflection area, and the endoscope image is masked based on the pixel-level binary mask to obtain the masked image.

[0073] In practical implementation, a pixel-level binary mask refers to:

[0074] A two-dimensional array of the same size as the image, where each element is either 0 or 1, used to mark whether each pixel in the image belongs to a specific region.

[0075] In specific implementation, a pixel-level binary mask is determined for the specular reflection area, and the endoscope image is masked based on the pixel-level binary mask to obtain the masked image.

[0076] The specular reflection region is extracted using the DUCKNet segmentation network, and a pixel-level binary mask M is generated. s ∈{0,1} H×W Then, based on the prime-level binary mask M... s Generate occluded image I masked :

[0077] I masked =IM s ⊙I;

[0078] Where H is the height of the endoscopic image, W is the width of the endoscopic image, and ⊙ represents element-wise multiplication, which is the result of multiplying corresponding elements of two matrices or arrays of the same shape.

[0079] Step S220: Correct the obscured image to obtain the first corrected endoscopic image.

[0080] In this step, the obscured image is corrected to generate the first corrected endoscopic image. This process lays the foundation for subsequent artifact removal and image fusion, helping to improve the overall quality of the endoscopic image and making it more suitable for subsequent image processing and analysis tasks.

[0081] In some embodiments, the obscured image is corrected to obtain a first corrected endoscopic image, including:

[0082] The pixel-level binary mask and the masked image are input into an image restoration network for image correction to obtain the first corrected image of the endoscope.

[0083] In specific implementation, the pixel-level binary mask and the obscured image are input into an image inpainting network for image correction to obtain the first corrected image of the endoscope.

[0084] Since specular reflection areas are typically concentrated in smooth tissue surfaces, their light intensity often exceeds the dynamic range of imaging sensors. LaMa-guided repair can significantly reduce specular reflection interference. The specular reflection distribution can be approximated as:

[0085] ∑k s (x,y)|L|≈∑M s (x,y)|L|;

[0086] The pixel-level binary mask and the masked image are input into the LaMa inpainting network for structure restoration and filling of the highlighted areas, resulting in the specular reflection corrected image, i.e., the first corrected image of the endoscope (I). noSpec :

[0087] I noSpec =INP(Stack(I masked M s ));

[0088] Where Stack represents the image I to be masked. masked And the binary mask M s Stacked along the channel dimension to form a multi-channel input; INP represents a pre-trained image inpainting network, such as LaMa.

[0089] Step S230: Remove artifacts from the first corrected image of the endoscope to obtain the second corrected image of the endoscope.

[0090] In this step, artifacts (such as halos and blurring caused by overexposure) in the highlighted areas of the first corrected endoscopic image are processed and removed. Specialized image processing algorithms or models can identify and eliminate these artifacts, thereby restoring image details and structure and reducing visual interference caused by highlighted areas. The resulting second corrected endoscopic image shows a significant improvement in visual quality and information integrity.

[0091] In some embodiments, artifact removal is performed on the first corrected endoscopic image to obtain a second corrected endoscopic image, including:

[0092] Feature extraction is performed on the first corrected image of the endoscope to obtain a low-dimensional feature representation of the first corrected image of the endoscope;

[0093] Conditional information of the endoscopic image is obtained, and the low-dimensional feature representation and the conditional information are combined to obtain a conditional feature representation;

[0094] Random noise information is obtained, and the second corrected image of the endoscope is obtained based on the conditional feature representation and the random noise information.

[0095] In specific implementation, feature extraction is performed on the first corrected image of the endoscope to obtain a low-dimensional feature representation of the first corrected image of the endoscope:

[0096] The first corrected image of the endoscope is transformed using an encoder and further processed using a feature extractor to obtain a low-dimensional feature representation of the first corrected image of the endoscope.

[0097] In practical implementation, the condition information refers to:

[0098] Conditional information is an additional input condition used to guide the image generation or restoration process. This condition can contain various types of information, depending on the application scenario and model design. This conditional information can be obtained from endoscopic images, other relevant data, or user input.

[0099] In specific implementation, the conditional information of the endoscopic image is obtained, and the low-dimensional feature representation and the conditional information are combined to obtain the conditional feature representation:

[0100] Conditional feature representations are obtained by combining low-dimensional feature representations with conditional information using conditional mapping functions.

[0101] In specific implementation, the random noise information is obtained, and the second corrected image of the endoscope is obtained based on the conditional feature representation and the random noise information as follows:

[0102] A decoder is used to transform the conditional feature representation and random noise information, thereby generating a diffuse reflection corrected image, i.e., the second corrected image for endoscopy. The random noise information is generated using a standard normal distribution. This random noise introduces randomness into the image generation process, increasing the diversity and naturalness of the generated image.

[0103] As a specific implementation, the StableDelight image-to-image transformation model is used to transform I... noSpec Global artifact removal is performed to obtain the second corrected endoscopic image I′, which can be obtained in the following way:

[0104]

[0105] Here, Enc() represents the encoder, whose function is to process the first corrected image I. noSpec This is converted into a low-dimensional feature representation. The encoder is typically a convolutional neural network (CNN) capable of extracting high-level features from the image; This refers to the feature extractor, whose function is to further process the features output by the encoder. This process may include operations such as feature mapping and feature fusion to generate feature representations that are more suitable for subsequent generation tasks. Typically, it is a neural network with the following parameters: In, where μ θ This represents a conditional mapping function, whose function is to map the feature representation to the condition t. + Combined, a conditional feature representation is generated, condition t + This may include time steps, category labels, or other auxiliary information to guide the generation process. The function takes θ as its parameter; ∈ represents a random noise vector; Dec represents the decoder, which converts the conditional feature representation and random noise information ∈ into the final endoscopic second-corrected image I′. The decoder is typically a deconvolutional CNN, capable of mapping low-dimensional features back to a high-dimensional image space.

[0106] Step S240: Perform weighted fusion on the first corrected image of the endoscope and the second corrected image of the endoscope to obtain the third corrected image of the endoscope.

[0107] In this step, the first corrected endoscopic image is fused with the second corrected image after artifact removal processing to combine the advantages of both. The fusion process uses specific algorithms or strategies, such as weighted averaging and mask-guided techniques, to balance the detail information in the first corrected endoscopic image and the artifact removal effect in the second corrected image. The resulting third corrected endoscopic image preserves key anatomical structures while eliminating multi-source reflection interference, improving the overall consistency and realism of the image. This provides higher-quality image input for subsequent medical diagnosis and computer vision tasks, enhancing the usability and accuracy of the image.

[0108] In some embodiments, the weighted fusion of the first corrected endoscopic image and the second corrected endoscopic image to obtain the third corrected endoscopic image includes:

[0109] Weight extraction is performed on the first corrected image of the endoscope to obtain a continuous value weighted image;

[0110] The endoscope first corrected image and the endoscope second corrected image are weighted and fused based on the continuous value weighted image to obtain the endoscope third corrected image.

[0111] In specific implementation, the method for extracting weights from the first corrected image of the endoscope to obtain a continuous weighted image is as follows:

[0112] The low-intensity artifact region Mask is detected in the first corrected image of the endoscope, and then a continuous value weighted image W is generated.

[0113] In specific implementation, the endoscope third corrected image is obtained by weighted fusing the first corrected image and the second corrected image based on the continuous value weighted image.

[0114] The second corrected image I of the endoscope is calculated using a continuously weighted image W. ′ First Correction Diagram I with Endoscope noSpec Weighted fusion is performed to obtain the third corrected image I from the endoscope. out :

[0115] I out =W⊙I′+(1-W)⊙I noSpec ;

[0116] This fusion strategy can adaptively process regions with strong and weak artifacts, preserving key anatomical structures in the image while eliminating multi-source reflection interference, thus improving the consistency and realism of the overall image.

[0117] In the above exemplary embodiments, the method for obtaining the third corrected image of the endoscope was introduced. The specific method for obtaining the continuous value weighted image is further described below:

[0118] Weight extraction is performed on the first corrected image of the endoscope to obtain a continuous value weighted image, including:

[0119] The first corrected image of the endoscope is converted to obtain a grayscale image;

[0120] The grayscale image is inspected to obtain artifact regions;

[0121] The artifact regions are normalized to obtain the continuous value weighted image.

[0122] In specific implementation, the first corrected image of the endoscope is converted to obtain a grayscale image; the artifact region is obtained by detecting the grayscale image as follows:

[0123] First, the first corrected image I from the endoscope... noSpec Convert to grayscale image G, then detect artifact regions (Mask) using threshold function Θ and dilation operation:

[0124] Mask = DILATE(IND(G,Θ));

[0125] Here, IND(G,Θ) represents an indicator function used to detect artifact regions in the grayscale image G that are below the threshold Θ and mark them as 1, while other regions are marked as 0. Subsequently, the marked regions are expanded by the dilation operation DILATE to ensure the integrity and connectivity of the artifact regions, thereby generating the artifact region Mask.

[0126] In specific implementation, the artifact region is normalized to obtain the continuous value weighted image in the following way:

[0127] Normalization generates a continuous-valued weighted graph W∈[0,1] H×W :

[0128]

[0129] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0130] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides an endoscopic image correction device.

[0132] refer to Figure 3 The endoscopic image correction device includes:

[0133] The occluded image determination module 310 is configured to determine the specular reflection area of ​​the endoscope image and obtain the occluded image based on the specular reflection area;

[0134] The first corrected image determination module 320 is configured to correct the occluded image to obtain the endoscope first corrected image;

[0135] The second corrected image determination module 330 is configured to remove artifacts from the first corrected image of the endoscope to obtain the second corrected image of the endoscope.

[0136] The third corrected image determination module 340 is configured to perform weighted fusion based on the first corrected image of the endoscope and the second corrected image of the endoscope to obtain the third corrected image of the endoscope.

[0137] In this exemplary embodiment, the occluded image determination module 310 is specifically configured as follows:

[0138] The specular reflection area of ​​the endoscope image is determined, and a pixel-level binary mask of the specular reflection area is determined. The endoscope image is masked based on the pixel-level binary mask to obtain the masked image.

[0139] In this exemplary embodiment, the first corrected image determination module 320 is specifically configured as follows:

[0140] The pixel-level binary mask and the masked image are input into an image restoration network for image correction to obtain the first corrected image of the endoscope.

[0141] In this exemplary embodiment, the second corrected image determination module 330 is specifically configured as follows:

[0142] Feature extraction is performed on the first corrected image of the endoscope to obtain a low-dimensional feature representation of the first corrected image of the endoscope; conditional information of the endoscope image is obtained, and the low-dimensional feature representation and the conditional information are combined to obtain a conditional feature representation; random noise information is obtained, and the second corrected image of the endoscope is obtained based on the conditional feature representation and the random noise information.

[0143] In this exemplary embodiment, the third corrected image determination module 340 is specifically configured as follows:

[0144] The first corrected image of the endoscope is converted to obtain a grayscale image; the grayscale image is detected to obtain artifact regions; the artifact regions are normalized to obtain the continuous value weighted image; the first corrected image of the endoscope and the second corrected image of the endoscope are weighted and fused based on the continuous value weighted image to obtain the third corrected image of the endoscope.

[0145] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0146] The apparatus of the above embodiments is used to implement the corresponding endoscopic image correction method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0147] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the endoscopic image correction method described in any of the above embodiments.

[0148] Figure 4 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0149] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0150] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0151] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0152] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0153] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0154] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0155] The electronic devices described above are used to implement the corresponding endoscopic image correction methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0156] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the endoscopic image correction method as described in any of the above embodiments.

[0157] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0158] The aforementioned non-transitory computer-readable storage media can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0159] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the endoscopic image correction method as described in any of the embodiments in the exemplary method section above, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0160] Based on the same inventive concept, corresponding to the endoscopic image correction method described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the endoscopic image correction method. Corresponding to the execution entity for each step in each embodiment of the endoscopic image correction method, the processor executing the corresponding step can belong to the corresponding execution entity.

[0161] The computer program product of the above embodiments is used to cause the computer and / or the processor to execute the endoscopic image correction method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0162] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, method, or computer program product. Therefore, this disclosure can be implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this disclosure can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0163] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0164] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0165] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0166] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0168] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0169] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0170] Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0172] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0173] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0174] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0175] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0176] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

[0177] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for correcting endoscopic images, characterized in that, include: Determine the specular reflection area of ​​the endoscopic image, and obtain the masked image based on the specular reflection area; The obscured image is corrected to obtain the first corrected endoscopic image; Artifact removal is performed on the first corrected image of the endoscope to obtain the second corrected image of the endoscope; The endoscope third corrected image is obtained by weighted fusion of the first corrected image and the second corrected image.

2. The method according to claim 1, characterized in that, The process of obtaining the masked image based on the specular reflection area includes: A pixel-level binary mask is determined for the specular reflection area, and the endoscope image is masked based on the pixel-level binary mask to obtain the masked image.

3. The method according to claim 2, characterized in that, The step of correcting the obscured image to obtain the first corrected endoscopic image includes: The pixel-level binary mask and the masked image are input into an image restoration network for image correction to obtain the first corrected image of the endoscope.

4. The method according to claim 1, characterized in that, The process of removing artifacts from the first corrected image of the endoscope to obtain the second corrected image of the endoscope includes: Feature extraction is performed on the first corrected image of the endoscope to obtain a low-dimensional feature representation of the first corrected image of the endoscope; Conditional information of the endoscopic image is obtained, and the low-dimensional feature representation and the conditional information are combined to obtain a conditional feature representation; Random noise information is obtained, and the second corrected image of the endoscope is obtained based on the conditional feature representation and the random noise information.

5. The method according to claim 1, characterized in that, The step of weighted fusion of the first corrected endoscopic image and the second corrected endoscopic image to obtain the third corrected endoscopic image includes: Weight extraction is performed on the first corrected image of the endoscope to obtain a continuous value weighted image; The endoscope first corrected image and the endoscope second corrected image are weighted and fused based on the continuous value weighted image to obtain the endoscope third corrected image.

6. The method according to claim 5, characterized in that, The step of extracting weights from the first corrected image of the endoscope to obtain a continuous weighted image includes: The first corrected image of the endoscope is converted to obtain a grayscale image; The grayscale image is inspected to obtain artifact regions; The artifact region is normalized to obtain the continuous value weighted image.

7. An endoscopic image correction device, characterized in that, include: The obscured image determination module is configured to determine the specular reflection area of ​​the endoscope image and obtain the obscured image based on the specular reflection area; The first corrected image determination module is configured to correct the occluded image to obtain the first corrected endoscope image; The second corrected image determination module is configured to remove artifacts from the first corrected image of the endoscope to obtain the second corrected image of the endoscope. The third corrected image determination module performs weighted fusion of the first corrected endoscope image and the second corrected endoscope image to obtain the third corrected endoscope image.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer program instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images