Image defogging method for real-time endoscopic imaging and endoscope system

Abstract

The present disclosure provides an image defogging method for real-time endoscopic imaging, comprising: generating a to-be-processed image sequence based on original endoscopic imaging acquired in real time, the to-be-processed image sequence comprising the original endoscopic imaging, and each to-be-processed image in the to-be-processed image sequence having a contrast different from each other; extracting an original imaging feature map of the original endoscopic imaging based on a first convolutional neural network, and extracting a smoke feature map sequence of the to-be-processed image sequence based on a second convolutional neural network; generating a final smoke feature map corresponding to the original endoscopic imaging based on the smoke feature map sequence; and obtaining an imaging feature map after defogging based on the original imaging feature map and the final smoke feature map. The present disclosure also provides an endoscopic imaging system, an electronic device, a readable storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to an image dehazing method, image dehazing device, endoscope system, electronic device, storage medium and program product for real-time endoscopic imaging. Background Technology

[0002] Endoscopic surgery is often performed in a smoky environment. For example, procedures such as ultrasonic scalpel cutting generate smoke, which can affect the image quality of endoscopic imaging and even the surgeon's progress.

[0003] To remove haze from endoscopic images and improve image quality, several image dehazing algorithms have emerged in the existing technology.

[0004] For example, Chinese patent document CN114881896A discloses a real-time dehazing algorithm for endoscopic images, which includes the following main steps:

[0005] The process involves: acquiring the original image obtained by an endoscope; performing dark channel processing on the original image to obtain a dark channel image; calculating the atmospheric brightness value and transfer function of the three primary color channels based on the original image and the dark channel image; correcting the three primary color values ​​of each pixel in the original image based on the atmospheric brightness value and transfer function to obtain a dehazed image; wherein, the dark channel processing steps include: acquiring the minimum gray level among the three primary color values ​​of each pixel within a fixed window centered on the pixel to be processed and with a preset pixel distance as the radius; counting the occurrence frequency of each minimum gray level within the fixed window; changing the gray level of the pixel to be processed to the first gray level; the first gray level is the lowest gray level among the minimum gray levels with an occurrence frequency greater than a preset frequency within the fixed window; and changing the gray level of each pixel to be processed to obtain the dark channel image.

[0006] For example, Chinese patent document CN114638767B discloses a method for removing smoke from laparoscopic images based on generative adversarial networks. This method uses a smoke mask segmentation network to process laparoscopic image samples to obtain a smoke mask image. The laparoscopic image samples and the smoke mask image are then input into a smoke removal network. A multi-level smoke feature extractor extracts features from the laparoscopic image samples to obtain light smoke feature vectors and dense smoke feature vectors. Based on the light smoke feature vectors, dense smoke feature vectors, and the smoke mask image, the smoke information is filtered using the masking effect while retaining the laparoscopic image, resulting in a smoke-free laparoscopic image.

[0007] For example, Chinese patent document CN113066026B discloses a method for smoke purification of endoscopic images based on deep neural networks. The main steps are as follows: using Render to simulate various situations of smoke appearance during surgery, smoke is randomly added to laparoscopic images to obtain training, testing, and validation datasets for the model; adding Laplacian image pyramids to each layer of the encoder to fuse images, and inputting the training images into the encoder to extract high-dimensional features; adding a CBAM attention mechanism to the last five layers of the decoder to restore the image features extracted by the encoder to the input image size through the decoder; using the synthetic image containing smoke as the training set and the original image as the training set label, the network is trained, and the network layers obtain the corresponding parameters through backpropagation.

[0008] However, the existing image dehazing algorithms have relatively complex dehazing processes, and the image processing speed needs to be optimized, while the dehazing effect needs to be improved. Summary of the Invention

[0009] This disclosure provides a novel image dehazing method, image dehazing device, endoscope system, electronic device, storage medium, and program product for real-time endoscopic imaging.

[0010] According to one aspect of this disclosure, an image dehazing method for real-time endoscopic imaging is provided, comprising:

[0011] A sequence of images to be processed is generated based on the raw endoscopic images acquired in real time. The sequence of images to be processed includes the raw endoscopic images, and each image in the sequence of images to be processed has a different contrast.

[0012] The original imaging feature map of the original endoscope image is extracted based on the first convolutional neural network, and the smoke feature map sequence of the image sequence to be processed is extracted based on the second convolutional neural network.

[0013] A final smoke feature map corresponding to the original endoscopic image is generated based on the smoke feature map sequence;

[0014] Based on the original imaging feature map and the final smoke feature map, a defogging imaging feature map is obtained.

[0015] In the image dehazing method according to at least one embodiment of the present disclosure, in S102, each image to be processed in the image sequence to be processed has the same geometric dimensions.

[0016] According to at least one embodiment of the image dehazing method of the present disclosure, the contrast of each image to be processed other than the original endoscopic image of the image sequence to be processed is higher than the contrast of the original endoscopic image.

[0017] According to at least one embodiment of the image dehazing method of the present disclosure, the contrast of each image in the image sequence to be processed is an arithmetic progression.

[0018] According to at least one embodiment of the image dehazing method of this disclosure, both the first convolutional neural network and the second convolutional neural network are trained convolutional neural networks.

[0019] An image dehazing method according to at least one embodiment of the present disclosure generates a sequence of images to be processed based on real-time acquired raw endoscopic images, including:

[0020] Obtain the imaging contrast (C1) of the raw endoscopic image;

[0021] A contrast increment (ΔC) is generated based on the imaging contrast (C1);

[0022] A sequence of images to be processed is generated based on the contrast increment (△C) and the preset number of images to be processed.

[0023] According to at least one embodiment of the image dehazing method of this disclosure, the contrast increment (ΔC) is not less than 1 / 2 of the imaging contrast (C1) of the original endoscopic image and not greater than the imaging contrast (C1) of the original endoscopic image.

[0024] An image dehazing method according to at least one embodiment of the present disclosure generates a final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence, comprising:

[0025] The smoke feature maps in the smoke feature map sequence are weighted and summed to obtain the final smoke feature map.

[0026] According to at least one embodiment of the image dehazing method of this disclosure, a weighted summation process is performed on each smoke feature map of the smoke feature map sequence, including:

[0027] A weighted weight (Wn) is generated for each smoke feature map based on the image contrast of the image to be processed corresponding to each smoke feature map;

[0028] The weighted summation process is performed based on the weighted weights (Wn) of each smoke feature map.

[0029] According to at least one embodiment of the image dehazing method of this disclosure, weighted weights are generated for each smoke feature map based on the following equation:

[0030] Wn*Cn = constant;

[0031] Where Wn is the weighted weight of each smoke feature map, and Cn is the contrast of the image to be processed corresponding to each smoke feature map.

[0032] An image dehazing method according to at least one embodiment of the present disclosure, which obtains a dehazed image feature map based on the original imaging feature map and the final smoke feature map, includes:

[0033] The original imaging feature map and the final smoke feature map are subtracted based on pixel values ​​to obtain the dehazed imaging feature map.

[0034] According to at least one embodiment of the image dehazing method of this disclosure, the first convolutional neural network is trained in the first stage based on a first dataset of foggy images from an endoscope.

[0035] Among them, each foggy endoscope image in the first endoscope foggy image dataset is a composite image of a fog-free endoscope image and a pure smoke image;

[0036] Specifically, the first endoscope foggy image dataset is obtained by superimposing the same pure smoke image onto random regions of each endoscope fog-free image to obtain each endoscope foggy image.

[0037] According to at least one embodiment of the image dehazing method of this disclosure, each endoscope dehazing image includes at least different endoscope dehazing images, preferably, each endoscope dehazing image is different from the others.

[0038] According to at least one embodiment of the image dehazing method of this disclosure, the first convolutional neural network performs feature map extraction on the first endoscope foggy image dataset based at least on a self-attention mechanism for a first stage of training.

[0039] According to at least one embodiment of the image dehazing method of this disclosure, the first convolutional neural network is trained in a second stage based on a second dataset of foggy endoscopic images;

[0040] Among them, each foggy endoscope image in the second endoscope foggy image dataset is a composite image of a fog-free endoscope image and a pure smoke image;

[0041] The second endoscopic foggy image dataset is obtained by superimposing distinct pure smoke images onto random regions of each endoscopic fog-free image to obtain each endoscopic foggy image.

[0042] According to at least one embodiment of the image dehazing method of this disclosure, the first convolutional neural network extracts feature maps from the second endoscopic fogged image dataset based at least on a self-attention mechanism for the second stage of training.

[0043] According to at least one embodiment of the image dehazing method of this disclosure, multiple different first endoscopic fogged image datasets are constructed based on different pure smoke images, and the first convolutional neural network is trained multiple times in the first stage.

[0044] According to at least one embodiment of the image dehazing method of this disclosure, the second convolutional neural network is trained in the first stage based on a first dataset of hazy endoscopic images and a dataset of hazy-free endoscopic images;

[0045] Wherein, each foggy endoscope image in the first foggy endoscope image dataset is a composite image of each fog-free endoscope image and a pure smoke image in the fog-free endoscope image dataset;

[0046] Specifically, the first endoscope foggy image dataset is obtained by superimposing the same pure smoke image onto random regions of each endoscope fog-free image to obtain each endoscope foggy image.

[0047] According to at least one embodiment of the image dehazing method of this disclosure, the endoscope dehazing image dataset includes at least different endoscope dehazing images, preferably, the individual endoscope dehazing images are different from each other.

[0048] According to at least one embodiment of the image dehazing method of this disclosure, the second convolutional neural network performs feature map extraction and subtraction operations on the first endoscope foggy image dataset and the endoscope fog-free image dataset based on at least a cross-attention mechanism to perform a first stage of training for smoke feature map extraction.

[0049] According to at least one embodiment of the image dehazing method of this disclosure, the second convolutional neural network is trained in the second stage based on a second endoscope foggy image dataset and an endoscope fog-free image dataset;

[0050] Wherein, each foggy endoscope image in the second foggy endoscope image dataset is a composite image of each fog-free endoscope image and a pure smoke image in the fog-free endoscope image dataset;

[0051] The second endoscopic foggy image dataset is obtained by superimposing distinct pure smoke images onto random regions of each endoscopic fog-free image to obtain each endoscopic foggy image.

[0052] According to at least one embodiment of the image dehazing method of this disclosure, the second convolutional neural network performs feature map extraction and subtraction operations on the second hazy endoscope image dataset and the hazy-free endoscope image dataset based on a cross-attention mechanism to perform a second stage of training for smoke feature map extraction.

[0053] According to at least one embodiment of the image dehazing method of this disclosure, multiple different first endoscopic fogged image datasets are constructed based on different pure smoke images, and the second convolutional neural network is trained multiple times in the first stage.

[0054] An image dehazing apparatus according to at least one embodiment of the present disclosure includes:

[0055] An image sequence generation unit generates a sequence of images to be processed based on real-time acquired raw endoscopic images. The sequence of images to be processed (In) includes the raw endoscopic images, and each image in the sequence of images to be processed has a different contrast.

[0056] A first convolutional neural network extracts the original imaging feature map of the original endoscope image;

[0057] A second convolutional neural network extracts a smoke feature map sequence from the image sequence to be processed;

[0058] A smoke feature map generation module generates a final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence.

[0059] A defogging imaging feature map generation module obtains a defogging imaging feature map based on the original imaging feature map and the final smoke feature map.

[0060] According to another aspect of this disclosure, an endoscopic imaging system is provided, comprising:

[0061] An image acquisition device for acquiring raw endoscopic images in real time; and a processor for performing an image dehazing method according to any embodiment of the present disclosure on the raw endoscopic images by executing a computer program.

[0062] According to yet another aspect of this disclosure, an electronic device is provided, comprising:

[0063] The memory stores execution instructions;

[0064] A processor that executes execution instructions stored in the memory, causing the processor to perform an image dehazing method according to any embodiment of the present disclosure.

[0065] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement an image dehazing method according to any embodiment of this disclosure.

[0066] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement an image dehazing method according to any embodiment of this disclosure. Attached Figure Description

[0067] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0068] Figure 1 This is a schematic flowchart of an image dehazing method for real-time endoscopic imaging according to one embodiment of the present disclosure.

[0069] Figure 2 A schematic diagram of a process for generating a sequence of images to be processed based on real-time acquired raw endoscopic images is shown as an embodiment of the present disclosure.

[0070] Figure 3 The diagram illustrates a flowchart of a weighted summation process for each smoke feature map in a smoke feature map sequence according to one embodiment of the present disclosure.

[0071] Figure 4 A schematic diagram illustrating the training process of a first convolutional neural network according to an embodiment of the present disclosure is shown.

[0072] Figure 5 A schematic diagram illustrating the training process of a second convolutional neural network according to one embodiment of this disclosure is shown.

[0073] Figure 6 This is a schematic block diagram of an image defogging device for real-time endoscopic imaging, which employs a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0074] Explanation of reference numerals in the attached figures

[0075] 1000 Image Dehazing Device

[0076] 1002 Image Sequence Generation Unit

[0077] 1004 First Convolutional Neural Network

[0078] 1006 Second Convolutional Neural Network

[0079] 1008 Smoke Feature Map Generation Module

[0080] 1010 Dehazing Imaging Feature Map Generation Module

[0081] 1100 bus

[0082] 1200 processor

[0083] 1300 memory

[0084] 1400 Other circuits. Detailed Implementation

[0085] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0086] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0087] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0088] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0089] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0090] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0091] The following text combines Figures 1 to 6 The present disclosure provides a detailed description of the image dehazing method and the image dehazing device for real-time endoscopic imaging.

[0092] Figure 1 This is a schematic flowchart of an image dehazing method for real-time endoscopic imaging according to one embodiment of the present disclosure.

[0093] refer to Figure 1 In some embodiments of this disclosure, the image dehazing method S100 for real-time endoscopic imaging includes:

[0094] S102. Generate a sequence of images to be processed based on the real-time acquired raw endoscopic images. The sequence of images to be processed includes the raw endoscopic images, and each image to be processed in the sequence of images to be processed has a different contrast. The sequence of images to be processed is denoted by In, where n is a natural number greater than or equal to 1.

[0095] S104. Extract the original imaging feature map of the original endoscope image based on the first convolutional neural network, and extract the smoke feature map sequence of the image sequence to be processed based on the second convolutional neural network.

[0096] S106. Generate the final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence;

[0097] S108. Obtain the defogging imaging feature map based on the original imaging feature map and the final smoke feature map.

[0098] This disclosure takes into account that smoke in endoscopic surgery within a human cavity can be considered as a background within the human cavity environment. Contrast will affect the prominence of smoke in endoscopic imaging. This disclosure adjusts the contrast of real-time endoscopic imaging through an algorithm during the real-time endoscopic imaging process, generating a series of images with different contrasts. Based on this series of images with different contrasts, smoke features are extracted to obtain a more accurate smoke feature map, which is then removed from the real-time endoscopic imaging, thereby obtaining endoscopic imaging without smoke.

[0099] Endoscopic imaging is generally a two-dimensional image (video frame). The characteristics of a two-dimensional image... Figure 1 Feature maps are typically extracted using 2D convolutional neural networks (CNNs), such as the ResNet-50 CNN (which has 7x7 filters, or convolutional kernels). The pixel values ​​of an image are processed through the convolutional kernels of a CNN to obtain the feature map.

[0100] The first and second convolutional neural networks described above are both trained convolutional neural networks.

[0101] In this disclosure, the number of images to be processed in the generated image sequence, i.e., the value of n, can be adjusted appropriately, such as 5, 6, 7, etc.

[0102] In the image dehazing method S100 and S102 of this disclosure, each image in the image sequence to be processed has the same geometric dimensions. That is, the image dehazing algorithm of this disclosure does not change the geometric dimensions of the original endoscopic imaging when generating the image sequence to be processed, so as to avoid consuming too much computing power and affecting the processing speed of the image dehazing algorithm.

[0103] In some embodiments of this disclosure, in S102 of the image dehazing method S100 of this disclosure, the contrast of each image to be processed other than the original endoscopic image of the image sequence to be processed is higher than the contrast of the original endoscopic image I1.

[0104] This disclosure sets the contrast of each image to be processed, other than the original endoscopic image I1, to be higher than the contrast of the original endoscopic image. This avoids the smoke feature map sequence extracted by the second convolutional neural network from including image features other than smoke, and avoids removing human tissue image pixels from the endoscopic image during smoke feature removal, which would cause image distortion.

[0105] In the image dehazing method S100 of this disclosure, preferably, in S102, the contrast of each image to be processed in the image sequence to be processed is an arithmetic sequence.

[0106] According to a preferred embodiment of this disclosure, in S102 of the image dehazing method S100, a sequence of images to be processed is generated based on the real-time acquired raw endoscopic images, including the following steps:

[0107] S1022. Obtain the imaging contrast (C1) of the original endoscopic image;

[0108] S1024. Generate contrast increment (△C) based on imaging contrast (C1);

[0109] S1026. Generate a sequence of images to be processed based on the contrast increment (△C) and the preset number of images to be processed.

[0110] Figure 2 A schematic diagram of a process for generating a sequence of images to be processed based on real-time acquired raw endoscopic images is shown as an embodiment of the present disclosure.

[0111] The following is an example of obtaining the final smoke feature map based on the generated smoke feature map sequence:

[0112] The original endoscopic image is denoted as I1. The image sequence to be processed generated based on the real-time acquired original endoscopic image I1 is I1, I2, I3, I4, I5...In. The number of image sequences to be processed can be adjusted appropriately.

[0113] For example, the contrast C1 of the original endoscopic image I1 can be used as a contrast increment to obtain I2, I3, I4, I5...In.

[0114] In this disclosure, in order to make the contrast difference of the image sequence to be processed sufficient to generate the smoke feature map of the corrected original endoscopic image (i.e. the final smoke feature map described above), this disclosure preferably sets the contrast increment to at least 1 / 2 of the contrast C1 of the original endoscopic image I1.

[0115] To avoid excessive contrast in the last image of the image sequence, which could lead to image distortion, the contrast increment should be less than or equal to C1.

[0116] Those skilled in the art, inspired by the technical solutions disclosed herein, can make appropriate adjustments to the contrast increment, all of which fall within the protection scope of this disclosure.

[0117] In some embodiments of this disclosure, preferably, the contrast increment ΔC described above is not less than 1 / 2 of the imaging contrast C1 of the original endoscopic imaging, and not greater than the imaging contrast C1 of the original endoscopic imaging.

[0118] In some embodiments of this disclosure, S106 of the image dehazing method S100 of this disclosure, generating a final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence, includes: performing a weighted summation process on each smoke feature map in the smoke feature map sequence to obtain the final smoke feature map.

[0119] In some embodiments of this disclosure, the weighted summation of the smoke feature maps in the smoke feature map sequence described above includes:

[0120] S1062. Obtain the image contrast Cn of the image to be processed corresponding to each smoke feature map;

[0121] S1064. Generate a weighted weight Wn for each smoke feature map based on the image contrast of the image to be processed corresponding to each smoke feature map;

[0122] S1066. Perform weighted summation based on the weighted weights Wn of each smoke feature map.

[0123] Figure 3 The diagram illustrates a flowchart of a weighted summation process for each smoke feature map in a smoke feature map sequence according to one embodiment of the present disclosure.

[0124] It should be noted that, since each smoke feature map in the smoke feature map sequence of this disclosure is obtained based on images to be processed with different contrasts, and in some embodiments of this disclosure, the contrasts of these images to be processed are arranged in an increasing arithmetic sequence based on the contrast of the original endoscopic imaging. The greater the difference between the contrast and the original endoscopic imaging, the greater the difference between the smoke feature map of the image to be processed and the smoke feature map of the original endoscopic imaging. Therefore, in some embodiments of this disclosure, when performing weighted summation on each smoke feature map in the smoke feature map sequence, this disclosure sets the weight of each smoke feature map to weight (Wn) * contrast of the image to be processed corresponding to the smoke feature map (Cn) = constant (A), and then normalizes the final smoke feature map obtained after weighted summation.

[0125] According to a preferred embodiment of this disclosure, in the image dehazing method S100 of this disclosure, weighted weights are generated for each smoke feature map based on the following equation:

[0126] Wn*Cn = constant;

[0127] Where Wn is the weighted weight of each smoke feature map, and Cn is the contrast of the image to be processed corresponding to each smoke feature map.

[0128] In some embodiments of this disclosure, in the image dehazing method S100 of this disclosure, preferably, S108, obtaining the dehazed image feature map based on the original imaging feature map and the final smoke feature map, includes: performing a pixel-value-based subtraction operation on the original imaging feature map and the final smoke feature map to obtain the dehazed image feature map.

[0129] For the image dehazing method S100 of this disclosure, the first convolutional neural network for extracting the original imaging feature map of the original endoscope image is trained in the first stage based on the first endoscope foggy image dataset.

[0130] Among them, each foggy endoscope image in the first endoscope foggy image dataset is a composite image of a fog-free endoscope image and a pure smoke image.

[0131] Specifically, the first endoscopic foggy image dataset is obtained by superimposing the same pure smoke image onto random regions of each endoscopic fog-free image.

[0132] In this disclosure, each fog-free endoscope image used to synthesize the first foggy endoscope image dataset is an endoscope image taken in a smoke-free environment.

[0133] It should be noted that in the image dehazing method S100 of this disclosure, when the pure smoke image is superimposed onto a random area of ​​each hazy-free endoscope image, the pure smoke image is completely superimposed onto the random area of ​​each hazy-free endoscope image.

[0134] In the image dehazing method S100 of this disclosure, each endoscope dehazing image includes at least different endoscope dehazing images, preferably, each endoscope dehazing image is different from the others.

[0135] In some embodiments of this disclosure, the first convolutional neural network of the image dehazing method S100 of this disclosure performs feature map extraction on a first endoscope foggy image dataset based at least on a self-attention mechanism for first-stage training.

[0136] Furthermore, according to a preferred embodiment of the present disclosure, the first convolutional neural network of the present disclosure is further trained in a second stage based on a second dataset of foggy endoscopic images.

[0137] Among them, each foggy endoscope image in the second endoscope foggy image dataset is a composite image of a fog-free endoscope image and a pure smoke image.

[0138] In this process, distinct pure smoke images are superimposed onto random regions of each endoscope fog-free image to obtain each endoscope foggy image, thus obtaining the second endoscope foggy image dataset.

[0139] Figure 4 A schematic diagram illustrating the training process of a first convolutional neural network according to an embodiment of the present disclosure is shown.

[0140] In some embodiments of this disclosure, preferably, the first convolutional neural network of the image dehazing method S100 of this disclosure performs feature map extraction on the second endoscopic fogged image dataset based at least on a self-attention mechanism for second-stage training.

[0141] In some embodiments of this disclosure, preferably, multiple different datasets of first endoscopic fogged images are constructed based on different pure smoke images to perform multiple first-stage training on the first convolutional neural network.

[0142] For example, based on M different pure smoke images, construct M different datasets of foggy images from the first endoscope, and perform M first-stage trainings on the first convolutional neural network, where M is greater than or equal to 2.

[0143] For the image dehazing method S100 of this disclosure, preferably, the second convolutional neural network used to extract the smoke feature map sequence of the image sequence to be processed is trained in the first stage based on the first endoscope foggy image dataset and the endoscope fog-free image dataset.

[0144] Among them, each foggy endoscope image in the first endoscope foggy image dataset is a composite image of each fog-free endoscope image and a pure smoke image in the endoscope fog-free image dataset.

[0145] Specifically, the first endoscopic foggy image dataset is obtained by superimposing the same pure smoke image onto random regions of each endoscopic fog-free image.

[0146] In this disclosure, each fog-free endoscope image used to synthesize the first foggy endoscope image dataset is an endoscope image taken in a smoke-free environment.

[0147] It should be noted that in this disclosure, when superimposing a pure smoke image onto a random area of ​​each endoscope fog-free image, the pure smoke image is completely superimposed onto the random area of ​​each endoscope fog-free image.

[0148] In some embodiments of this disclosure, preferably, the endoscope fog-free image dataset includes at least different endoscope fog-free images, and preferably, the individual endoscope fog-free images are different from each other.

[0149] In some embodiments of this disclosure, the second convolutional neural network performs feature map extraction and subtraction operations on the first endoscope foggy image dataset and the endoscope non-fog image dataset based on a cross-attention mechanism to perform the first stage training for smoke feature map extraction.

[0150] Furthermore, according to a preferred embodiment of the present disclosure, the second convolutional neural network of the present disclosure is further trained in a second stage based on a second dataset of foggy endoscope images and a dataset of fog-free endoscope images.

[0151] Among them, each foggy endoscope image in the second endoscope foggy image dataset is a composite image of each fog-free endoscope image and a pure smoke image in the endoscope fog-free image dataset.

[0152] In this process, distinct pure smoke images are superimposed onto random regions of each endoscope fog-free image to obtain each endoscope foggy image, thus obtaining the second endoscope foggy image dataset.

[0153] Figure 5 A schematic diagram illustrating the training process of a second convolutional neural network according to one embodiment of this disclosure is shown.

[0154] In some embodiments of this disclosure, preferably, the second convolutional neural network performs feature map extraction and subtraction operations on the second endoscope foggy image dataset and the endoscope non-fog image dataset based at least on a cross-attention mechanism to perform a second stage of training for smoke feature map extraction.

[0155] According to a preferred embodiment of the present disclosure, in the image dehazing method S100 of the present disclosure, multiple different first endoscope foggy image datasets are constructed based on different pure smoke images to perform multiple first-stage training on the second convolutional neural network.

[0156] This disclosure does not impose any specific limitations on the specific network architecture of the first and second convolutional neural networks described above. Under the guidance of the technical solutions disclosed herein, those skilled in the art can select existing convolutional neural network models as the first and second convolutional neural networks, all of which fall within the protection scope of this disclosure.

[0157] Accordingly, this disclosure also provides an image dehazing device for real-time endoscopic imaging. In some embodiments of this disclosure, the image dehazing device 1000 for real-time endoscopic imaging includes:

[0158] The image sequence generation unit 1002 generates a sequence of images to be processed based on the raw endoscopic images acquired in real time. The sequence of images to be processed (In) includes the raw endoscopic images, and each image to be processed in the sequence of images to be processed has a different contrast.

[0159] The first convolutional neural network 1004 extracts the original imaging feature map of the original endoscope image.

[0160] The second convolutional neural network 1006 extracts the smoke feature map sequence of the image sequence to be processed;

[0161] The smoke feature map generation module 1008 generates a final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence.

[0162] The defogging imaging feature map generation module 1010 obtains the defogging imaging feature map based on the original imaging feature map and the final smoke feature map.

[0163] The image defogging device for real-time endoscopic imaging disclosed herein can be implemented based on a computer software program architecture.

[0164] Figure 6 This is a schematic block diagram of an image defogging device for real-time endoscopic imaging, which employs a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0165] The image dehazing device 1000 may include corresponding modules that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart may be performed by a corresponding module, and the device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0166] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0167] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0168] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0169] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0170] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0171] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0172] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0173] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0174] This disclosure also provides an endoscopic imaging system, including: an image acquisition device for acquiring raw endoscopic images in real time; and a processor that executes an image dehazing method according to any embodiment of this disclosure on the raw endoscopic images by executing a computer program.

[0175] The image acquisition device disclosed herein can be an existing image acquisition device for endoscope systems, and may include CCD sensors, CMOS sensors, etc., without particular limitation. The processor may be included in a computer with a display.

[0176] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform an image dehazing method according to any embodiment of this disclosure.

[0177] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement an image dehazing method according to any embodiment of this disclosure.

[0178] This disclosure also provides a computer program product, including a computer program / instruction, characterized in that the computer program / instruction, when executed by a processor, implements an image dehazing method according to any embodiment of this disclosure.

Claims

1. An image dehazing method for real-time endoscopic imaging, characterized in that, include: A sequence of images to be processed is generated based on the raw endoscopic images acquired in real time. The sequence of images to be processed includes the raw endoscopic images, and each image in the sequence of images to be processed has a different contrast. The original imaging feature map of the original endoscope image is extracted based on the first convolutional neural network, and the smoke feature map sequence of the image sequence to be processed is extracted based on the second convolutional neural network. Generating a final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence includes: performing a weighted summation process on each smoke feature map in the smoke feature map sequence to obtain the final smoke feature map; and Based on the original imaging feature map and the final smoke feature map, a defogging imaging feature map is obtained; The image sequence to be processed is generated based on the raw endoscopic images acquired in real time, including: Obtain the imaging contrast of the raw endoscopic image; Generate a contrast increment based on the imaging contrast; and A sequence of images to be processed is generated based on the contrast increment and the preset number of images to be processed. The smoke feature map sequence is subjected to a weighted summation process, including: Weighted weights are generated for each smoke feature map based on the image contrast of the image to be processed corresponding to each smoke feature map; and The weighted summation process is performed based on the weighted weights of each smoke feature map.

2. The image dehazing method according to claim 1, characterized in that, Each image in the sequence of images to be processed has the same geometric dimensions.

3. The image dehazing method according to claim 1, characterized in that, The contrast of each image to be processed, excluding the original endoscopic image, is higher than the contrast of the original endoscopic image.

4. The image dehazing method according to claim 3, characterized in that, The contrast of each image in the image sequence to be processed is an arithmetic progression.

5. The image dehazing method according to any one of claims 1 to 4, characterized in that, Both the first convolutional neural network and the second convolutional neural network are trained convolutional neural networks.

6. The image dehazing method according to claim 1, characterized in that, The contrast increment is not less than 1 / 2 of the imaging contrast of the original endoscopic image, and not greater than the imaging contrast of the original endoscopic image.

7. The image dehazing method according to claim 1, characterized in that, Obtaining a dehazed imaging feature map based on the original imaging feature map and the final smoke feature map includes: performing a pixel-value-based subtraction operation on the original imaging feature map and the final smoke feature map to obtain the dehazed imaging feature map.

8. An image dehazing device for real-time endoscopic imaging, characterized in that, include: An image sequence generation unit generates a sequence of images to be processed based on real-time acquired raw endoscopic images. The sequence of images to be processed includes the raw endoscopic images, and each image in the sequence of images to be processed has a different contrast. A first convolutional neural network extracts the original imaging feature map of the original endoscope image; A second convolutional neural network extracts a smoke feature map sequence from the image sequence to be processed; A smoke feature map generation module generates a final smoke feature map corresponding to the original endoscopic image based on the smoke feature map sequence, including: performing a weighted summation process on each smoke feature map in the smoke feature map sequence to obtain the final smoke feature map; and A defogging imaging feature map generation module, which obtains a defogging imaging feature map based on the original imaging feature map and the final smoke feature map; The image sequence to be processed is generated based on the raw endoscopic images acquired in real time, including: Obtain the imaging contrast of the raw endoscopic image; Generate a contrast increment based on the imaging contrast; and A sequence of images to be processed is generated based on the contrast increment and the preset number of images to be processed. The smoke feature map sequence is subjected to a weighted summation process, including: Weighted weights are generated for each smoke feature map based on the image contrast of the image to be processed corresponding to each smoke feature map; and The weighted summation process is performed based on the weighted weights of each smoke feature map.

9. An endoscopic imaging system, characterized in that, include: An image acquisition device, wherein the image acquisition device is used to acquire raw endoscopic images in real time; as well as A processor that performs the image dehazing method of any one of claims 1 to 7 on a raw endoscopic image by executing a computer program.

10. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the image dehazing method according to any one of claims 1 to 7.

11. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the image dehazing method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image dehazing method according to any one of claims 1 to 7.

Images