Endoscope image freezing processing method and related product

Abstract

Embodiments of the present application provide an endoscope image freezing processing method and related products. The method comprises: in a super-resolution imaging mode, in response to an image freezing instruction, acquiring a first frozen image in an endoscope video stream; inputting the first frozen image into a generation network for image enhancement to obtain a first super-resolution image, the optical resolution of the first super-resolution image being higher than that of the first frozen image; and obtaining a second frozen image according to the first super-resolution image and displaying the second frozen image in a user interface. According to the above technical solution, when the endoscope video stream is frozen, a second frozen image with higher signal-to-noise ratio and clarity can be displayed, wherein key details such as blood vessel texture are enhanced, providing users with more rich image information.

Description

Technical Field

[0001] This invention relates to the field of endoscopy technology, and more specifically to an endoscopy image freezing processing method, an endoscopy device, a storage medium, and a computer program product. Background Technology

[0002] Endoscopy is a commonly used diagnostic tool in modern medicine, widely applied in the diagnosis and treatment of diseases of the digestive and respiratory tracts. During the examination, the endoscopic video stream can be displayed in real time on the endoscope's display device. When the user sees an image of interest, they can pause the current view using the freeze image function to analyze the image details.

[0003] In some existing endoscopic image freezing methods, the current frame image in the endoscopic video stream can be directly frozen and displayed to the user. However, due to limitations in the aperture and aperture of the endoscope, a significant amount of high-frequency signal components are lost in endoscopic imaging, resulting in poor clarity for each frame in the endoscopic video stream. Furthermore, during actual endoscopic operation, the endoscope may be in motion when the user triggers the image freeze command, leading to motion blur in the directly frozen current frame image. Additionally, the natural movements of human organs, such as breathing and heartbeat, can make it difficult for the imaging object to remain completely still relative to the endoscope, further exacerbating the blurring of the frozen current frame image.

[0004] Based on this, the frame with the highest resolution can be selected from the video stream within a certain period before and after the freeze point as the freeze frame image. While this method can avoid the problem of lower image quality caused by endoscope movement to some extent, the resolution of the frozen image is still limited by the imaging performance of the endoscope.

[0005] Therefore, how to effectively improve the quality of frozen frame images under the limitation of endoscopic imaging performance has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] The present invention is proposed in view of the above-mentioned problems. According to one aspect of the present invention, a method for freezing endoscopic images is provided, comprising:

[0007] In super-resolution imaging mode, in response to an image freeze command, a first frozen image is acquired from the endoscopic video stream;

[0008] A first frozen image is input into a generative network for image enhancement to obtain a first super-resolution image, the optical resolution of which is higher than that of the first frozen image; and

[0009] A second frozen image is obtained based on the first super-resolution image, and the second frozen image is displayed in the user interface.

[0010] For example, the number of pixels in the first super-resolution image is the same as the number of pixels in the first frozen image.

[0011] For example, the generative network is trained by constructing a training dataset and performing a first operation multiple times based on the training dataset, wherein the training dataset includes multiple sets of corresponding high-resolution training images and low-resolution training images;

[0012] The first operation includes:

[0013] Low-resolution training images are input into the generative network to obtain a second super-resolution image;

[0014] The first loss function value of the generative network is calculated based on the second super-resolution image and the high-resolution training image, and the generative network is trained based on at least the first loss function value.

[0015] For example, the first loss function values ​​include pixel loss function values ​​and feature loss function values.

[0016] Based on the second super-resolution image and the high-resolution training image, the first loss function value of the generative network is calculated, including:

[0017] Calculate the pixel loss function value based on the pixel difference between the second super-resolution image and the high-resolution training image;

[0018] The second super-resolution image and the high-resolution training image are respectively input into the trained feature extraction network to output their respective feature extraction results;

[0019] The feature loss function value is calculated based on the feature extraction results of the second super-resolution image and the feature extraction results of the high-resolution training image.

[0020] The first loss function value is calculated based on the pixel loss function value and the feature loss function value.

[0021] For example, the first operation is performed alternately with a second operation for training the discriminative network, wherein the second operation includes:

[0022] The second super-resolution image is input into the discriminant network to output the true probability of the second super-resolution image, and the high-resolution training image is input into the discriminant network to output the true probability of the high-resolution training image.

[0023] Based on the true probability of the second super-resolution image and the true probability of the high-resolution training image, the second loss function value of the discriminant network is calculated, and the discriminant network is trained based on the second loss function value.

[0024] The first operation also includes:

[0025] The third loss function value of the generator network is calculated based on the true probability of the second super-resolution image, and the training of the generator network is also based on the third loss function value.

[0026] For example, a training dataset is constructed, including:

[0027] Obtain the original high-resolution training images;

[0028] The original high-resolution training image is degraded to obtain a corresponding low-resolution training image, wherein the number of pixels in the original high-resolution training image and the number of pixels in the low-resolution training image are the same. The degradation process includes blurring and / or a first noise-adding process.

[0029] For example, constructing the training dataset also includes:

[0030] Obtain the original high-resolution training images;

[0031] The original high-resolution training images are subjected to a second noise-adding process to obtain noisy high-resolution training images, wherein the high-resolution training images in the training dataset are noisy high-resolution training images.

[0032] For example, obtaining a second frozen image based on a first super-resolution image includes:

[0033] Based on the first super-resolution image and the first frozen image, the difference image is determined;

[0034] A second frozen image is obtained based on the first frozen image and the difference image.

[0035] For example, obtaining a second frozen image based on a first frozen image and a difference image includes:

[0036] The differential image is subjected to parameter tuning, and a second frozen image is obtained based on the first frozen image and the parameter-tuned differential image;

[0037] The parameter tuning process for the difference image includes:

[0038] Gain adjustment is performed on the differential image based on the enhancement intensity coefficient; and / or,

[0039] Perform filtering operations on the difference image.

[0040] For example, the user interface also includes a parameter setting area, which includes a first operable control and / or a second operable control. The first operable control is used to set the enhancement intensity coefficient in response to the user's first parameter setting operation, and the second operable control is used to set the processing parameters of the filtering operation in response to the user's second parameter setting operation.

[0041] According to another aspect of the present invention, an endoscope device is also provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the endoscope image freezing processing method as described above.

[0042] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, which, when executed, are used to perform the endoscopic image freezing processing method as described above.

[0043] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions which, when executed, are used to perform the endoscopic image freezing processing method as described above.

[0044] According to the above technical solution, in super-resolution imaging mode, upon receiving an image freeze command, a first frozen image can be acquired from the endoscopic video stream. This first frozen image is then input into a generative network for image enhancement to generate a first super-resolution image. A second frozen image is then generated and displayed based on the first super-resolution image. Since the optical resolution of the first super-resolution image is higher than that of the first frozen image, the second frozen image generated from the first super-resolution image also has a higher optical resolution than the first frozen image. Therefore, when the endoscopic video stream is frozen, a second frozen image with a higher signal-to-noise ratio and higher clarity can be displayed, enhancing key details such as vascular texture and providing the user with richer image information.

[0045] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0046] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0047] Figure 1 A schematic flowchart of an endoscopic image freezing processing method according to an embodiment of this application is shown;

[0048] Figure 2 A schematic diagram of a user interface according to an embodiment of this application is shown; and

[0049] Figure 3 A schematic block diagram of an endoscopic device according to one embodiment of this application is shown. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0051] To at least partially solve the above-mentioned technical problems, an endoscopic image freezing processing method is provided according to an embodiment of the present invention. Figure 1 A schematic flowchart of an endoscopic image freezing processing method 100 according to an embodiment of this application is shown. Figure 1 As shown, method 100 may include, but is not limited to, the following steps S110, S130 and S150.

[0052] Step S110: In super-resolution imaging mode, in response to an image freeze command, a first frozen image is acquired in the endoscopic video stream.

[0053] Optionally, the super-resolution imaging mode can be turned on or off in response to the user's mode setting operation. For example, during an endoscopy examination, an endoscopic video stream can be generated in real time. If an image of interest is detected in the video stream, an image freeze command can be generated by triggering a device such as the freeze button on the endoscope handle or a foot switch. Under the action of the image freeze command, a first frozen image can be captured from the endoscopic video stream.

[0054] Optionally, the first frozen image can be the current frame image, that is, the frame image that is being displayed when the user triggers the image freeze command. Alternatively, the first frozen image can also be another frame image besides the current frame image, for example, the frame image with the highest clarity among several frames before and after the current frame image.

[0055] When the super-resolution imaging mode is off, a first frozen image can be acquired from the endoscopic video stream in response to an image freeze command and displayed directly on the user interface. Alternatively, the first frozen image acquired from the endoscopic video stream can be processed in other ways and then the processed image can be displayed on the user interface. This application does not specifically limit this.

[0056] Step S130: The first frozen image is input into the generative network for image enhancement to obtain a first super-resolution image, wherein the optical resolution of the first super-resolution image is higher than that of the first frozen image.

[0057] Research has found that the clarity of endoscopic images is mainly affected by optical resolution and spatial sampling rate (determined by the number of pixels in the image). In practical applications, most endoscopes can meet the requirements for spatial sampling rate, and some endoscopes even exhibit oversampling. Therefore, the fundamental reason affecting the clarity of endoscopic images lies in insufficient optical resolution. However, optical resolution is mainly affected by the optical imaging performance of the endoscope, and improving the optical imaging performance of endoscopes is difficult and costly. Therefore, in order to solve the technical problems described in the background art, according to embodiments of the present invention, it is desirable to perform enhancement processing on the first frozen image to obtain an image with higher optical resolution.

[0058] Compared to natural images, endoscopic images have relatively limited color information. In the medical field, endoscopic images serve as an auxiliary tool for disease diagnosis, containing a wealth of important details, such as vascular texture. Using traditional image processing methods like filtering to improve the clarity of endoscopic images may struggle to balance noise reduction and sharpening, and can easily result in lost details, affecting diagnostic results. Furthermore, endoscopic images are typically imaged under low-light, high-gain conditions, resulting in significantly higher noise levels than natural images. Traditional image processing methods are prone to producing artifacts, further reducing the clarity of endoscopic images. Therefore, according to embodiments of the present invention, a generative network is used to enhance a first frozen image to obtain a first super-resolution image.

[0059] Compared to the first frozen image, the first super-resolution image has a higher optical resolution. In other words, when an ideal spot on the object is imaged, it covers a larger number of pixels in the first frozen image, such as 3 pixels. However, in the first super-resolution image, a smaller number of pixels correspond to the same ideal spot, such as 1 or 2 pixels. Therefore, the first super-resolution image can provide the user with clearer image details compared to the first frozen image.

[0060] It is understandable that the higher the optical resolution, the more accurately the image can reproduce the details of the imaged object without distortion. Generally, the spatial frequency response (SFR) can be used to characterize the optical resolution of an image. For example, in one example, the first super-resolution image compared to the first frozen image: when the SFR value is 50%, the corresponding object-space spatial angular frequency can be increased by 7.95%, and when the SFR value is 30%, the corresponding object-space spatial angular frequency can be increased by 7.75%.

[0061] For example, when super-resolution imaging mode is enabled, the first frozen image acquired above can be input into a pre-trained generative network. The generative network can be a neural network used to upscale a low-optical-resolution image to a high-optical-resolution image. This neural network can have any reasonable network structure capable of fitting a mapping between the distribution of low-resolution images and the distribution of high-resolution images. Based on the image features and mapping relationships learned during training, the generative network can enhance the first frozen image, thereby outputting a first super-resolution image.

[0062] In some embodiments, the generating network may employ a Residual Local Feature Network (RLFN). After the first frozen image undergoes a series of operations such as convolution, activation, and pooling in the generating network, a first super-resolution image is output.

[0063] Optionally, the network structure of the generative network is optimized for the characteristics of endoscopic images, including an enhanced spatial attention module. This module rapidly reduces the spatial and channel dimensions of the feature map by sequentially using 1x1 convolutional layers, 3x3 convolutional layers with a stride of 2, and 7x7 max-pooling layers with a stride of 4. This allows for faster aggregation of image features and accelerates the model's inference time. Within the enhanced spatial attention module, a sigmoid function can be used to generate attention weights based on the aggregated image features. Multiplying these attention weights by the feature map pixels adjusts the range of activation values ​​in the feature map. This, to some extent, reduces the training difficulty of the generative network.

[0064] Step S150: Obtain the second frozen image based on the first super-resolution image and display the second frozen image in the user interface.

[0065] It is understood that the second frozen image is the frozen image generated in response to the aforementioned image freeze command and ultimately displayed to the user for observation.

[0066] Figure 2 A schematic diagram of a user interface 200 according to one embodiment of this application is shown. Figure 2 As shown, the user interface 200 may include an image display area 210. For example, when the super-resolution imaging mode is enabled, the first super-resolution image can be directly used as the second frozen image, or the second frozen image can be obtained after post-processing the first super-resolution image. For instance, the first super-resolution image output by the generating network may have high sharpness, which may not meet the user's expectations or requirements; for example, the user may prefer a softer output image. The user can select different post-processing operations as needed to further adjust the enhanced first super-resolution image to obtain a second frozen image with better display effects or that better suits their observation habits. Thus, while further optimizing the visual effect of the image, it can also better match the user's endoscopic image observation habits and clinical diagnostic needs.

[0067] Image display area 210 can be dedicated to displaying the second frozen image. That is, the user can directly observe the image after image enhancement processing in image display area 210. In conjunction with the foregoing, when super-resolution imaging mode is off, images directly captured from the endoscopic video stream can be directly displayed in image display area 210.

[0068] According to the above technical solution, in super-resolution imaging mode, upon receiving an image freeze command, a first frozen image can be acquired from the endoscopic video stream. This first frozen image is then input into a generative network for image enhancement to generate a first super-resolution image. A second frozen image is then generated and displayed based on the first super-resolution image. Since the optical resolution of the first super-resolution image is higher than that of the first frozen image, the second frozen image generated from the first super-resolution image also has a higher optical resolution than the first frozen image. Therefore, when the endoscopic video stream is frozen, a second frozen image with a higher signal-to-noise ratio and higher clarity can be displayed, enhancing key details such as vascular texture and providing the user with richer image information.

[0069] For example, the number of pixels in the first super-resolution image is the same as the number of pixels in the first frozen image.

[0070] For endoscopic imaging technology, the number of pixels in each frame of the endoscopic video stream is sufficient to represent the detailed features of the imaged object. Therefore, while improving the optical resolution of the first frozen image, there is little need to perform super-resolution processing on the first frozen image to increase the number of pixels. More importantly, research has found that if endoscopic images are simultaneously super-resolution processed to increase the number of pixels, it is more likely to generate high-frequency false textures that do not actually exist in the endoscopic image, interfering with the user's diagnosis and potentially even causing misdiagnosis.

[0071] Therefore, in this embodiment of the application, when enhancing the first frozen image, the optical resolution is increased while the number of pixels remains unchanged. This not only ensures that the first super-resolution image presents more detailed features visually, but also avoids the introduction of high-frequency false textures that may interfere with the doctor's diagnosis, thus ensuring the accuracy of the presented endoscopic image.

[0072] For example, step S150, obtaining the second frozen image based on the first super-resolution image, may include steps S151 and S152.

[0073] In step S151, a difference image is determined based on the first super-resolution image and the first frozen image.

[0074] For example, each pixel in the difference image can represent the corresponding pixel difference between the first super-resolution image and the first frozen image. Specifically, the pixel difference between each pixel in the first super-resolution image and the first frozen image can be calculated to obtain the difference image. The specific calculation formula can be:

[0075] Diff = G(g) - g,

[0076] Where Diff represents the difference image, G(g) represents the first super-resolution image, and g represents the first frozen image. The difference image reflects the intensity of image enhancement performed by the generative network on the first frozen image.

[0077] In step S152, a second frozen image is obtained based on the first frozen image and the difference image.

[0078] For example, after calculating the difference image, it can be rationalized according to the actual application requirements, such as parameter tuning. Then, the pixels of the first frozen image are superimposed on the corresponding pixel positions of the processed difference image to obtain the second frozen image.

[0079] Since the first frozen image and the first super-resolution image are the input and output of the generator network, respectively, a difference image can be directly obtained based on their pixel differences. The difference image can provide information about local changes in the image. Determining the second frozen image based on the difference image and the first frozen image can improve the detail visibility of the second frozen image. Furthermore, by performing parameter tuning on the difference image and obtaining the second frozen image based on the parameter-tuned difference image and the first frozen image, the degree of improvement in detail visibility can be adjusted to adapt to more diverse observation needs.

[0080] For example, the parameter tuning process may include: adjusting the gain of the difference image based on the enhancement intensity coefficient.

[0081] For example, the enhancement intensity coefficient α can control the degree of enhancement of the first super-resolution image relative to the first frozen image, where 0 ≤ α ≤ 1. In some embodiments, three enhancement intensity coefficients can be set, for example, low, medium, and high. Specifically, α can be set to, for example, 0.33, 0.66, and 1.0 respectively. When α is small, the gain adjustment of the difference image is small, the image enhancement degree is low, the image is closer to the original first frozen image, and the image is softer; when α is large, the gain adjustment of the difference image is large, the image enhancement degree is high, the image details and sharpness are more significantly improved, and the image is sharper. Users can select an appropriate enhancement intensity coefficient according to specific observation needs and / or habits. Thus, the obtained second frozen image Fuse = g + α * Diff.

[0082] For example, the above parameter tuning process may include: performing filtering operations on the difference image.

[0083] Filtering processes include low-pass filtering, high-pass filtering, and band-pass filtering. By selecting different filters and setting appropriate cutoff frequencies, specific frequency bands in the differential signal can be filtered out, further optimizing image details and noise suppression. In one embodiment, a high-pass filter is used to process the differential image, setting three cutoff frequencies: low, medium, and high, corresponding to cutoff frequencies of 0.3, 0.6, and 1.0, respectively. This can filter out low- and mid-frequency components in the differential image. For example, when the cutoff frequency is set to medium, low-frequency signals with frequencies less than 0.6 in the differential image can be filtered out, effectively reducing over-enhancement in the first super-resolution image, preserving high-frequency detail information, and suppressing texture edge overshoot artifacts in the first super-resolution image. Thus, the obtained second frozen image, Fuse = g + Filter(Diff), where Filter(Diff) represents the result of filtering the differential image Diff.

[0084] In some embodiments, parameter tuning of the difference image may include both the gain adjustment operation described above and the filtering operation. The gain adjustment and filtering operations for the difference image have been described in detail above, and for brevity, they will not be repeated here. In this embodiment, the obtained second frozen image Fuse = g + α * Filter(Diff).

[0085] Through the above methods, the image enhancement information generated by the generative network can be reasonably fused with the original first frozen image to obtain the final image (i.e., the second frozen image) that meets the user's needs.

[0086] For example, see again Figure 2 To facilitate users in adjusting image enhancement effects independently, the user interface 200 may also include a parameter setting area 220.

[0087] The parameter setting area 220 may include a first operable control 221. The first operable control 221 is used to set the aforementioned enhancement intensity coefficient in response to a user's first parameter setting operation. In some embodiments, the first operable control 221 may be in the form of a slider, with the slider's value range corresponding to three enhancement intensity coefficients: low, medium, and high. The user can set the enhancement intensity coefficient by dragging the slider, thereby controlling the degree of image enhancement. When the slider is at the low end, it corresponds to a low enhancement intensity coefficient, resulting in a lower degree of image enhancement; when the slider is at the medium end, it corresponds to a medium enhancement intensity coefficient, resulting in a moderate degree of image enhancement; and when the slider is at the high end, it corresponds to a high enhancement intensity coefficient, resulting in a higher degree of image enhancement. Optionally, the first operable control 221 may also be any reasonable form, such as a text box or an adjustment bar. When the first operable control 221 is a text box, the user can set the enhancement intensity coefficient by directly entering a number, for example, through the first parameter setting operation.

[0088] For example, the parameter setting area 220 may further include a second operable control 222. The second operable control 222 is used to set the processing parameters of the filtering operation in response to a user's second parameter setting operation. In some embodiments, the second operable control 222 may take the form of a combination of a drop-down menu and a text box. The drop-down menu can be used to select the type of filter, such as low-pass filtering, high-pass filtering, band-pass filtering, etc.; the text box is used to set the cutoff frequency of the filter. By performing a second parameter setting operation, such as selecting the filter type from the drop-down menu and entering a suitable cutoff frequency in the text box, the user can personalize the image filtering effect.

[0089] Therefore, the parameter setting area is set up in a visual way on the user interface, making it convenient for users to adjust relevant parameters according to their personalized needs.

[0090] For example, the generative network can be trained by constructing a training dataset in step S210 and performing the first operation multiple times based on the training dataset in step S220. The training dataset includes multiple sets of corresponding high-resolution training images and low-resolution training images.

[0091] For example, step S210, which constructs the training dataset, may include steps S211 and S212.

[0092] Step S211: Obtain the original high-resolution training image.

[0093] Step S212: Degrade the original high-resolution training image to obtain the corresponding low-resolution training image.

[0094] For example, a magnifying endoscope with a shorter focal length or a larger entrance pupil diameter can be used to capture the original high-resolution training image. This original high-resolution training image is denoted as f. This type of endoscope imaging exhibits weaker optical diffraction, a smaller Airy disk diameter, and higher optical resolution. Exemplarily, multiple images at different angles and magnifications can be acquired in various scenarios to cover different situations that may occur during endoscopic examinations. Then, the acquired original high-resolution training image is degraded to generate a corresponding low-resolution training image k. Using the corresponding high-resolution training image f and the low-resolution training image k as a set of training data, a training dataset can be constructed.

[0095] Optionally, the degradation processing may include blurring and / or a first noise-adding process. This blurring can be achieved using a blurring convolution kernel. The choice of the blurring convolution kernel h is crucial. Different types of blurring convolution kernels produce different degrees and types of blurring effects. Taking a Gaussian blurring convolution kernel as an example, its parameters, such as standard deviation, directly affect the degree of blurring. By setting different Gaussian blurring convolution kernel sizes and standard deviations, the blurring effects caused by lens aberrations, tissue motion, and other factors during endoscopic imaging are simulated. Considering the characteristics of endoscopic images, blurring convolution kernel parameters that can better simulate the blurring situations that may occur during endoscopic imaging are selected. This yields a low-resolution training image k = f*h.

[0096] For example, obtaining a low-resolution training image k by degrading the original high-resolution training image f can also be achieved through a first noise-adding process, i.e., adding additive noise a to the original high-resolution training image f, where k = f + a. The additive noise a also simulates the noise characteristics of endoscopic images. Endoscopic images are typically imaged under low-light, high-gain conditions, resulting in significant noise intensity and complex noise types, potentially including Gaussian noise, salt-and-pepper noise, etc. When generating noise in the low-resolution training images of the training dataset, mixed noise can be generated based on the noise statistical characteristics of actual endoscopic images to closely approximate the noise situation of real endoscopic images. For example, Gaussian noise can be generated first at a certain ratio, and then salt-and-pepper noise can be randomly added to some pixel locations, with the noise intensity precisely controlled by adjusting parameters such as the variance of the noise.

[0097] Optionally, the degradation process may also include blurring and first noise addition, resulting in a low-resolution training image k = f*h + a. The specific processing steps can be found in the previous text and will not be repeated here.

[0098] It is worth noting that the degradation process of the original high-resolution training image in the embodiments of this application does not include a downsampling process. Therefore, the number of pixels in the original high-resolution training image is the same as the number of pixels in the degraded low-resolution training image. In the embodiments of this application, the spatial sampling rate of the imaging system for the obtained high-resolution image is the same as the spatial sampling rate of the imaging system for obtaining the super-resolution image, i.e., the physical pixel spacing of the imaging sensor is the same. Therefore, the degradation process does not include a downsampling process. The difference lies in the optical resolution of the imaging system; both high resolution and low resolution refer to optical resolution. As mentioned earlier, the spatial resolution of endoscopic images acquired by the image sensor of an endoscope device usually has a margin of error, so there is no need to improve the spatial resolution of the endoscopic image when enhancing the image. Furthermore, improving the spatial resolution of the endoscopic image may introduce high-frequency false textures, affecting the accuracy of diagnosis.

[0099] For example, step S210, which constructs the training dataset, may also include steps S213 and S214.

[0100] Step S213: Obtain the original high-resolution training image.

[0101] This step S213 is similar to step S211 above, and will not be repeated here for the sake of brevity.

[0102] Step S214: Perform a second noise-adding process on the original high-resolution training image to obtain a noise-added high-resolution training image, wherein the high-resolution training image in the training dataset is a noise-added high-resolution training image.

[0103] It should be noted that the role of the generative network is to enhance image detail features, transforming low-resolution images into high-resolution images. If high-resolution and low-resolution training images with a large signal-to-noise ratio difference are used as training datasets during training, the trained generative network may exhibit excessive denoising, resulting in the loss of significant image detail features. Endoscopic images typically require the preservation of as much of this important, subtle texture detail as possible. Therefore, according to embodiments of this application, a second noise-adding process is applied to the original high-resolution training image. The intensity of the added noise b can be adjusted using the coefficient β to balance the denoising capability and detail preservation capability of the generative network. The resulting noisy high-resolution training image f′=f+β*b. Optionally, for each set of high-resolution and low-resolution training images, the additive noise a added by the first noise-adding process on the original high-resolution training image and the noise b added by the second noise-adding process on the original high-resolution training image are equal, i.e., a=b, in order to adjust the signal-to-noise ratio difference between the high-resolution and low-resolution images. When the coefficient β is small, such as close to 0, the generative network will focus more on learning to remove noise during training. This may lead to over-smoothing of the image, resulting in the loss of some subtle details and textures. Conversely, when the coefficient β is large, close to 1, the generative network's ability to suppress noise is relatively weak, but it can better preserve the original details of the image. In some embodiments, the coefficient β can be set to a reasonable value, such as greater than 0.5 and less than 0.9, so that when processing endoscopic images, the generative network can effectively remove noise while preserving important subtle details such as blood vessel textures to the greatest extent, providing strong support for accurate diagnosis by the user. Therefore, the high-resolution training images mentioned below can all be noisy high-resolution training images.

[0104] Step S220, which involves performing the first operation multiple times based on the training dataset, may include steps S221 and S222.

[0105] Step S221: Input the low-resolution training image into the generator network to obtain the second super-resolution image.

[0106] For example, in the first stage of training the generator network, a low-resolution training image k from the constructed training dataset is input into the generator network so that the generator network outputs the corresponding second super-resolution image G(k).

[0107] This step S221 is similar to step S130 above, and will not be repeated here for the sake of brevity.

[0108] In step S222, based on the second super-resolution image and the high-resolution training image, the first loss function value of the generator network is calculated, and the generator network is trained based on at least the first loss function value.

[0109] For example, high-resolution training images can be used as labels. A first loss function value is calculated based on the difference between the second super-resolution image output by the generator network and the label—the high-resolution training image. Based on the calculated first loss function value, the parameters of the generator network can be adjusted and optimized using the Adaptive Moment Estimation (Adam) optimizer. The Adam optimizer can adaptively adjust the learning rate, quickly converging to the optimal solution during training. In one embodiment, the initial learning rate can be set to 1e-4, iterating through all training data 400 times. To gradually decay the learning rate during training, a learning rate adjustment strategy of 1e-4*(1-(epoch / 400)^0.9) is adopted, where epoch represents the current training epoch number. During each training iteration, eight image patches are selected from the training dataset for training, i.e., a batch size of 8, which can balance training efficiency and memory usage to some extent. The training image patch size can be set to 192*192, a size that contains sufficient image information for the model to learn rich image features without causing excessive computation and affecting training speed. Optionally, image patches of size 192*192 can be cropped from the high-resolution and low-resolution training images in the original training dataset as described above, and used as training data. The cropped image patch can be the one containing the most information from the original image. In each training round, the Adam optimizer backpropagates the parameters of the generator network based on the first loss function value, updates the parameters of the generator network, and allows the generator network to continuously learn, gradually reducing the first loss function value and improving the accuracy and quality of image generation.

[0110] The above technical solution utilizes a training dataset to train the generator network. This enables the generator network to output a first super-resolution image whose enhancement effect better matches the expected enhancement of the endoscopic image when the first frozen image is input into it.

[0111] For example, the first loss function value may include pixel loss function value and feature loss function value. In one embodiment, step S222, which calculates the first loss function value of the generator network based on the second super-resolution image and the high-resolution training image, may include steps S222a to S222d.

[0112] In step S222a, the pixel loss function value is calculated based on the pixel difference between the second super-resolution image and the high-resolution training image.

[0113] For example, the pixel loss function value is calculated by comparing the pixel value difference of the corresponding pixel in the second super-resolution image and the high-resolution training image. The formula for calculating the L1 loss function can be: Where m×n represents the dimensions of the second super-resolution image and the high-resolution training image. G(k(i,j)) represents the pixel value at position (i,j) in the second super-resolution image output by the generator network, and f′(i,j) represents the pixel value at position (i,j) in the high-resolution training image. The L1 loss function can intuitively reflect the average error in pixel values ​​between the second super-resolution image and the high-resolution training image, making the pixel values ​​of the output image of the generator network as close as possible to the pixel values ​​of the high-resolution training image during training. That is, the smaller the L1 loss function value, the closer the pixel values ​​at corresponding spatial positions of the second super-resolution image and the high-resolution training image are.

[0114] Step S222b: Input the second super-resolution image and the high-resolution training image into the trained feature extraction network respectively to output their respective feature extraction results.

[0115] For example, a 16-layer Visual Geometric Group Network (VGG16) can be used as a feature extraction network. The VGG16 network includes multiple convolutional and pooling layers, enabling it to extract feature information from images at different scales. Optionally, natural images can be used as the training dataset to train the VGG16 network. The second super-resolution image and the high-resolution training image are then input into the trained VGG16 network, respectively, and the feature extraction results for the second super-resolution image and the high-resolution training image are output, respectively.

[0116] Step S222c: Calculate the feature loss function value based on the feature extraction results of the second super-resolution image and the feature extraction results of the high-resolution training image.

[0117] For example, based on the feature extraction results of both the second super-resolution image and the high-resolution training image, the feature loss function value LPIPS is calculated at the feature level. The feature loss function value can be calculated using the mean squared error (MSE) loss function, and the calculation formula is as follows:

[0118]

[0119] Where, φ k H represents the feature map of the k-th layer extracted by the trained VGG16 network. k and W k These represent the height and width of the feature map at layer k, respectively.

[0120] Step S222d: Calculate the first loss function value based on the pixel loss function value and the feature loss function value.

[0121] For example, the pixel loss function value and the feature loss function value can be weighted and summed according to certain weights to obtain the first loss function value. In one embodiment, the weights of the pixel loss function value and the feature loss function value can be set to 1:1, that is, the formula for calculating the first loss function value is:

[0122] L G =L1+LPIPS.

[0123] In the above technical solution, the first loss function includes not only pixel loss function values ​​but also feature loss function values. A user's perception of an endoscopic image depends not only on pixel-level differences but also, and more significantly, on image content, structure, and semantics. In this solution, the feature loss function values ​​are used to minimize the differences between the features of the second super-resolution image and the features of the high-resolution training image, and this is used to train the generative network. Therefore, this generative network can comprehensively consider the differences in images at both the pixel and feature levels, thereby generating a more realistic first super-resolution image.

[0124] For example, a discriminative network can be introduced, which, together with the generator network, forms an adversarial generative network. Based on the first stage of training, the performance of the generator network can be further optimized by combining the training of the discriminative network. Specifically, the first operation for training the generator network can be executed alternately with the second operation for training the discriminative network. The second operation may include steps S310 and S320.

[0125] In step S310, the second super-resolution image is input into the discriminant network to output the true probability of the second super-resolution image, and the high-resolution training image is input into the discriminant network to output the true probability of the high-resolution training image.

[0126] In some embodiments, the discriminative network can be implemented using a semantic segmentation network (U-Net). The U-Net network structure includes a decoder and an encoder, which can effectively extract and utilize the contextual information of the image to accurately determine the authenticity of the input image. For example, the second super-resolution image output by the generator network and the high-resolution training image are respectively input into the discriminative network. The discriminative network analyzes each pixel in the input image and outputs the true probability of each pixel. For the high-resolution training image, the true probability label for each pixel is set to 1, indicating that the pixel comes from a real sample; for the second super-resolution image, the true probability label for each pixel is set to 0, indicating that the pixel value was generated by the generator network.

[0127] Step S320: Calculate the second loss function value of the discriminant network based on the true probability of the second super-resolution image and the true probability of the high-resolution training image, and train the discriminant network based on the second loss function value.

[0128] For example, the second loss function value can be calculated using the cross-entropy loss function, and the calculation formula is as follows:

[0129] L D =-E Y～f′ [log(σ(D(Y)))]-E x～G(k) [log(1-σ(D(x)))],

[0130] Where σ represents the Sigmoid function, E represents the expected value, Y represents the high-resolution training image, x represents the second super-resolution image, and D(Y) and D(x) represent the true probabilities of the high-resolution training image and the second super-resolution image output by the discriminant network, respectively. Optionally, the parameters of the discriminant network are adjusted to minimize the value of the second loss function. This enables the discriminant network to more accurately distinguish between real images and images output by the generator network. During the training of the discriminant network, the network parameters of the generator network remain unchanged so that the focus is on training the performance of the discriminant network.

[0131] For example, the first operation may further include step S222e, calculating the third loss function value of the generator network based on the true probability of the second super-resolution image, wherein training the generator network is also based on the third loss function value.

[0132] For example, the third loss function value L adv =-E x～G(k) [log(σ(D(x)))]. The third loss function value represents the adversarial loss between the generator network and the discriminator network. The purpose of the third loss function is to make the image generated by the generator network more realistic and closer to a real high-resolution image. In the above embodiment, the total loss function value L of the generator network is... G =L1+LPIPS+ω*L adv Here, ω represents the weight coefficient of the third loss function value. By adjusting this weight coefficient ω, the optimization direction of the generator network between pixel accuracy and image realism can be balanced. In some embodiments, the value of ω is set to 0.05, so that the generator network can balance the relationship between image generation accuracy and adversarial ability against the discriminative network, generating more realistic images.

[0133] During the training of the generative network, the parameters of the discriminative network remain unchanged. Similarly, in one embodiment, the initial learning rate can be set to 1e-4, and all training data can be trained iteratively 200 times, using a learning rate adjustment strategy of 1e-4*(1-(epoch / 200)^0.9). During each training iteration, eight image patches are selected from the training dataset, i.e., the batch size is 8, which can balance training efficiency and memory usage to some extent. The training image patch size can be set to 192*192, a size that contains sufficient image information for the model to learn rich image features without causing excessive computation and affecting training speed. In each training round, the Adam optimizer backpropagates the parameters of the generative network based on the total loss function value, updating the parameters of the generative network so that it continuously learns and gradually improves the quality and realism of the generated images. After training, the weight parameters of the generative network are saved for processing frozen images in practical applications.

[0134] Those skilled in the art will understand that, in the above description, at least one specific calculation method is given for each loss function value. However, the disclosed specific calculation methods are merely examples and do not constitute a limitation of the present invention. Those skilled in the art can use other loss functions to calculate the above loss function values.

[0135] According to another aspect of the present invention, an endoscopic device is also provided. Figure 3 A schematic block diagram of an endoscope device 300 according to one embodiment of this application is shown. Figure 3 As shown, the endoscope device 300 may include a processor 310 and a memory 320, wherein the memory 320 stores computer program instructions, which are executed by the processor to perform the endoscope image freezing processing method 100 as described above. The endoscope device 300 may be, for example, an endoscope image processor or an integrated unit combining an endoscope light source and an image processor.

[0136] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, which, when executed, are used to perform the endoscopic image freezing processing method described above. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0137] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions which, when executed, are used to perform the endoscopic image freezing processing method as described above.

[0138] Those skilled in the art can understand the specific implementation scheme and beneficial effects of the above-mentioned endoscope device, storage medium and computer program product by reading the relevant description of the endoscopic image freezing processing method. For the sake of brevity, they will not be described in detail here.

[0139] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0142] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0143] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0144] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0145] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0146] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the endoscopic image freezing processing apparatus according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0147] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0148] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the scope of protection of this invention. The scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for freezing endoscopic images, characterized in that, include: In super-resolution imaging mode, in response to an image freeze command, a first frozen image is acquired from the endoscopic video stream; The first frozen image is input into a generative network for image enhancement to obtain a first super-resolution image, wherein the optical resolution of the first super-resolution image is higher than that of the first frozen image. as well as A second frozen image is obtained based on the first super-resolution image, and the second frozen image is displayed in the user interface.

2. The endoscopic image freezing processing method as described in claim 1, characterized in that, The number of pixels in the first super-resolution image is the same as the number of pixels in the first frozen image.

3. The endoscopic image freezing processing method as described in claim 1, characterized in that, The generative network is trained by constructing a training dataset and performing a first operation multiple times based on the training dataset, wherein the training dataset includes multiple sets of corresponding high-resolution training images and low-resolution training images; The first operation includes: The low-resolution training image is input into the generator network to obtain a second super-resolution image; Based on the second super-resolution image and the high-resolution training image, the first loss function value of the generator network is calculated, and the generator network is trained based on at least the first loss function value.

4. The endoscopic image freezing processing method as described in claim 3, characterized in that, The first loss function value includes pixel loss function value and feature loss function value. The step of calculating the first loss function value of the generator network based on the second super-resolution image and the high-resolution training image includes: Calculate the pixel loss function value based on the pixel difference between the second super-resolution image and the high-resolution training image; The second super-resolution image and the high-resolution training image are respectively input into the trained feature extraction network to output their respective feature extraction results; Based on the feature difference between the feature extraction results of the second super-resolution image and the feature extraction results of the high-resolution training image, the feature loss function value is calculated. The first loss function value is calculated based on the pixel loss function value and the feature loss function value.

5. The endoscopic image freezing processing method as described in claim 3, characterized in that, The first operation is performed alternately with a second operation for training the discriminant network, wherein the second operation includes: The second super-resolution image is input into the discriminant network to output the true probability of the second super-resolution image, and the high-resolution training image is input into the discriminant network to output the true probability of the high-resolution training image. Based on the true probability of the second super-resolution image and the true probability of the high-resolution training image, the second loss function value of the discriminant network is calculated, and the discriminant network is trained based on the second loss function value; The first operation further includes: Based on the true probability of the second super-resolution image, the third loss function value of the generator network is calculated, wherein the training of the generator network is also based on the third loss function value.

6. The endoscopic image freezing processing method as described in claim 3, characterized in that, The construction of the training dataset includes: Obtain the original high-resolution training images; The original high-resolution training image is degraded to obtain the corresponding low-resolution training image, wherein the number of pixels in the original high-resolution training image and the number of pixels in the low-resolution training image are the same, and the degradation process includes blurring and / or a first noise-adding process.

7. The endoscopic image freezing processing method as described in claim 3, characterized in that, The construction of the training dataset includes: Obtain the original high-resolution training images; The original high-resolution training image is subjected to a second noise-adding process to obtain a noisy high-resolution training image, wherein the high-resolution training image in the training dataset is a noisy high-resolution training image.

8. The endoscopic image freezing processing method according to any one of claims 1-7, characterized in that, The step of obtaining the second frozen image based on the first super-resolution image includes: Based on the first super-resolution image and the first frozen image, a difference image is determined; The second frozen image is obtained based on the first frozen image and the difference image.

9. The endoscopic image freezing processing method as described in claim 8, characterized in that, Obtaining the second frozen image based on the first frozen image and the difference image includes: The differential image is subjected to parameter tuning, and a second frozen image is obtained based on the first frozen image and the parameter-tuned differential image. The parameter tuning process for the difference image includes: Gain adjustment is performed on the differential image based on the enhancement intensity coefficient; and / or, The differential image is then subjected to filtering processing.

10. The endoscopic image freezing processing method as described in claim 9, characterized in that, The user interface further includes a parameter setting area, which includes a first operable control and / or a second operable control. The first operable control is used to set the enhancement intensity coefficient in response to a user's first parameter setting operation, and the second operable control is used to set the processing parameters of the filtering operation in response to a user's second parameter setting operation.

11. An endoscopic device, comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the endoscopic image freezing processing method as described in any one of claims 1 to 10.

12. A storage medium storing program instructions that, when executed, perform the endoscopic image freezing processing method as described in any one of claims 1 to 10.

13. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the endoscopic image freezing processing method as described in any one of claims 1 to 10.

Images