Endoscope fluorescence image blood vessel enhancement method based on deep learning U-shaped network framework and self-group iteration upgrading device

Abstract

The invention discloses an endoscope fluorescence image blood vessel enhancement method based on a deep learning U-shaped frame and a self-group iteration upgrading device. The method comprises the following steps of: acquiring an original fluorescence image from a single machine or a plurality of networked imaging devices, performing blood vessel enhancement on the original fluorescence image by applying various detail enhancement algorithms such as an adaptive histogram equalization algorithm and the like to obtain a target image (label), and taking the original fluorescence image and the target image (label) as a training set of data, and constructing and training a U-shaped blood vessel enhanced convolutional neural network locally or at a cloud end, and inputting the original fluorescence image acquired by the camera into the trained U-shaped blood vessel enhanced convolutional neural network to obtain a blood vessel enhanced image. According to the blood vessel enhancement method of the endoscope fluorescence image based on the deep learning U-shaped frame, blood vessel details of the fluorescence image can be effectively enhanced.

Description

Technical Field

[0001] This invention relates to the field of medical imaging, and in particular to a method for enhancing blood vessels in endoscopic fluorescence images based on a deep learning U-shaped network framework and a self-group iterative upgrade device. Background Technology

[0002] The fluorescence image acquired by the endoscope is mainly obtained by superimposing the white light image acquired by the two cameras and the processed fluorescence image.

[0003] In the images we currently acquire, the information about blood vessels belongs to high-frequency signals, but they are often embedded in a large amount of low-frequency background signals, which greatly reduces the visual visibility of details. Therefore, it is necessary to enhance the high-frequency components of the image.

[0004] Existing advanced enhancement methods include: Adaptive Histogram Equalization (AHE); Adaptive Contrast Enhancement Algorithm (ACE); Histogram Equalization Algorithm (HE); and Adaptive Histogram Equalization Algorithm with Limited Contrast Based on the above algorithms (CLAHE).

[0005] The AHE algorithm maps data using relevant information from local histograms. This alters image contrast but requires significant computation. Later, bilinear interpolation was used to overcome this problem. The image is first divided into blocks, and then the mapping relationships within each block are calculated. To enhance the value of a particular pixel, the mapping relationship is obtained by subtracting the mapping relationships from the four adjacent blocks.

[0006] The ACE algorithm employs an unsharp mask technique. First, it divides the image into two parts: a low-frequency unsharp mask, primarily calculated by averaging the pixel values ​​in a local region centered on a given pixel (the low-frequency component), and a high-frequency component obtained by subtracting the unsharp mask from the original image. This high-frequency component is then amplified and added to the unsharp mask, resulting in the enhanced image.

[0007] Histogram equalization and its improved algorithms have shown good enhancement effects on blood vessels in images. However, since this algorithm is based on grayscale images, it requires multi-channel processing of color images based on different color spaces, resulting in severe color shifts in the processed images. Because medical images have high requirements for color fidelity, it seems that histogram equalization still has room for improvement before it can be applied to the field of medical imaging.

[0008] Furthermore, the aforementioned algorithms also exhibit disadvantages such as noticeable artifacts or amplification of noise, which to some extent affect the image clarity.

[0009] In recent decades, although numerous algorithms have been developed for enhancing angiographic images, they still cannot fully meet practical needs. The reasons are quite complex, including sensitivity to background noise, boundary shrinkage caused by Gaussian filtering, non-adaptive parameters, poor enhancement of small vessels, difficulty in identifying bifurcation points and nodes, slow computation speed, and rough vessel surfaces after enhancement. These factors make it impossible to achieve a perfect, universal enhancement method. Instead, reasonable choices must be made based on specific problems and needs, balancing or emphasizing key indicators such as accuracy, speed, and robustness. Summary of the Invention

[0010] To address the problems of non-adaptive parameters, poor enhancement of small blood vessels, difficulty in identifying bifurcation points and nodes, slow computation speed, and rough vessel surface after enhancement in existing traditional detail enhancement algorithms, this invention provides a method for enhancing blood vessels in endoscopic fluorescence images based on a deep learning U-shaped network framework. The method includes:

[0011] The raw fluorescence image output from the endoscope is acquired and input into a U-shaped convolutional neural network to obtain the endoscopic image with enhanced blood vessels.

[0012] The training method for the U-shaped blood vessel enhancement convolutional neural network is as follows:

[0013] Step S100: Obtain fluorescence images of biological tissues using an endoscopic fluorescence camera and segment them into 256*256 pixel sizes as the original images; apply various detail enhancement algorithms to process the original images to obtain fluorescence images with enhanced blood vessel details, which are used as the target images (labels).

[0014] In some preferred embodiments, the original image and the target image (label) are acquired using the following method:

[0015] Step S110: Use an endoscopic fluorescence camera to acquire raw fluorescence images of biological tissues and divide them into 256*256 pixel sizes as raw images;

[0016] In step S120, the original image described in step S110 is processed using various detail enhancement algorithms, such as the contrast-limited adaptive histogram equalization algorithm (CLAHE), to obtain a fluorescence image with enhanced blood vessel details, which serves as the target image (label).

[0017] Step S130: The original image and the target image (label) are randomly divided into a training set, a validation set and a test set according to a reasonable ratio (here it can be set to 7:2:1) for model training.

[0018] In step S200, the original fluorescence image from the endoscope and the enhanced fluorescence image of the blood vessels are used as a training set to construct and train a U-shaped blood vessel enhancement convolutional neural network. The training process utilizes a computing platform equipped with an NVIDIA graphics card (Tesla A800 with 80GB).

[0019] In some preferred embodiments, the training method for the U-shaped vessel enhancement convolutional neural network is as follows:

[0020] Step S210: Input the original image into the first encoder network and the first decoder network in sequence to obtain the first training image;

[0021] Step S220: Input the original image and the first training image into the medical segmentation large model encoder network to obtain the second training image;

[0022] Step S230: Input the original image into the second encoder network to obtain the third training image;

[0023] Step S240: Input the second training image and the third training image into the second decoder network to obtain the fourth training image;

[0024] In some preferred embodiments, the training method of the U-shaped vessel enhancement convolutional neural network further includes:

[0025] Step S250: Calculate the mixture loss function of the second training image and the target image (label). And according to the hybrid loss function The U-shaped vessel enhancement convolutional neural network is modified. The hybrid loss function... This can be expressed as:

[0026]

[0027] in, , , , , These are the weighting coefficients that balance the various loss functions.

[0028] Step S300: The original image obtained by the endoscopic fluorescence camera is input into the trained U-shaped blood vessel enhancement convolutional neural network, and the output is the blood vessel enhancement image.

[0029] Step S400: Input the original fluorescence image described in step S300 into the U-shaped vessel enhancement convolutional neural network self-population iterative upgrade device to correct the U-shaped vessel enhancement convolutional neural network.

[0030] The self-group iterative upgrade device includes a single-machine self-iterative upgrade module and a networked group iterative upgrade module. The difference between self-upgrade and group upgrade lies in whether the training information (dataset, dataset size, dataset quality, etc.) originates from a local single machine (self-upgrade) or a networked group (group upgrade). The group iterative upgrade module includes two functions: local group iterative upgrade and cloud group iterative upgrade. The difference between local group upgrade and cloud group upgrade lies in whether the training process is completed locally or in the cloud. The training device can be local, in the cloud, or both. The local group iterative upgrade includes two functions: local group single-machine training iterative upgrade and local group distributed training iterative upgrade. The local group single-machine training iterative upgrade includes two functions: local group single-machine training with shared dataset iterative upgrade (without sharing training weights) and local group single-machine training with shared dataset and weights iterative upgrade.

[0031] The deep learning-based self-iterative upgrade device includes:

[0032] A storage module is used to collect the raw fluorescence images input to the U-shaped blood vessel enhancement convolutional neural network;

[0033] The expert annotation module is used to annotate the original fluorescence images of the storage module and generate corresponding labels;

[0034] The neural network fine-tuning module is used to correct and fine-tune the U-shaped blood vessel enhancement convolutional neural network based on the images collected by the storage module.

[0035] The self-iterative upgrade step based on the self-iterative upgrade device is to input the original fluorescence image and corresponding label mentioned in step S300 into the U-shaped blood vessel enhancement convolutional neural network for training.

[0036] The deep learning-based group iterative upgrade device includes:

[0037] A storage module is used to collect the raw fluorescence images input to the U-shaped blood vessel enhancement convolutional neural network;

[0038] The expert annotation module is used to annotate the original fluorescence images of the storage module and generate corresponding labels;

[0039] The neural network fine-tuning module is used to correct and fine-tune the U-shaped blood vessel enhancement convolutional neural network based on the images collected by the storage module.

[0040] The network service module is used for communication and data transmission with the cloud control center.

[0041] The iterative upgrade of the shared local single-machine training dataset based on the group iterative upgrade module includes:

[0042] The training dataset comes from all networked devices;

[0043] The original fluorescence image and corresponding label described in step S300, which are shared by several other networked devices, are downloaded through the network module and input into the U-shaped blood vessel enhancement convolutional neural network of the current local device. The network is trained locally to obtain the corrected weights of the U-shaped blood vessel enhancement convolutional neural network for use by the current local device.

[0044] The group-based local single-machine training shared dataset and weight iterative upgrade based on the group iterative upgrade module includes:

[0045] The training dataset comes from all networked devices;

[0046] Prerequisite: All networked devices must use the same training model software;

[0047] The original fluorescence images and corresponding tags described in step S300 of several other networked devices are downloaded to the current local device through the network module and input into the U-shaped vessel enhancement convolutional neural network of the current local device. Then, the network device is trained to obtain the corrected weights of the U-shaped vessel enhancement convolutional neural network. The weights of the U-shaped vessel enhancement convolutional neural network of the current local device are updated and shared with all networked devices in real time for use by all networked devices.

[0048] The group-based local distributed training iterative upgrade based on the group iterative upgrade module includes:

[0049] Training information (dataset size, dataset quality, etc.) comes from all networked devices;

[0050] Prerequisites: Either all networked devices use the same training model software (which requires knowledge of the dataset size for each device), or the quantitative differences in the parameters of the training datasets generated by each device are known (which requires knowledge of the dataset quality for each device).

[0051] The original fluorescence images and corresponding labels described in step S300 of each current local device are input into the U-shaped vessel enhancement convolutional neural network of the current local device for training, and the corrected weights of the U-shaped vessel enhancement convolutional neural network are obtained. The corrected weights of the U-shaped vessel enhancement convolutional neural network of several networked devices are uploaded to the cloud control center for global reduction (AllReduce). If the training model software of all networked devices is the same, then global reduction is performed according to the size of the dataset of each device. If the training model software of the networked devices is different and the quantitative difference of the dataset parameters of each device is known, then global reduction is performed according to the quantitative difference of the dataset quality of each device. The updated weights of the U-shaped vessel enhancement convolutional neural network are obtained.

[0052] The group cloud-based iterative upgrade based on the group iterative upgrade module includes:

[0053] Prerequisite: The training model software must be identical across all networked devices and cloud-based devices;

[0054] The original fluorescence images and corresponding tags described in step S300 of several networked devices are transmitted to the cloud control center through the network service module; the cloud control center summarizes the received original fluorescence images and corresponding tags, and performs U-shaped vessel enhancement convolutional neural network training; the local network service module downloads and synchronously updates the weights of the current device's U-shaped vessel enhancement convolutional neural network.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) Near real-time enhancement: Due to the low time complexity of this algorithm, it can achieve near real-time enhancement of video.

[0057] (2) Color fidelity: Directly enhances color images, which can enhance image details and make the image more vivid and colorful without causing distortion or color deviation.

[0058] (3) Improve the accuracy of blood vessel recognition: Compared with traditional image endoscopic image blood vessel enhancement methods, deep learning algorithms not only have stronger generalization ability and can enhance images under different conditions, but also have higher accuracy in blood vessel recognition. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1This is a flowchart of an embodiment of an endoscopic image blood vessel enhancement method based on deep learning according to the present invention.

[0061] Figure 2 This is a diagram of the network structure of the U-shaped blood vessel enhancement convolutional neural network of the present invention.

[0062] Figure 3 This is a network structure diagram of the VM-UNet module of the U-shaped blood vessel enhancement convolutional neural network of the present invention.

[0063] Figure 4 This is a network structure diagram of the VSS basic module of the U-shaped blood vessel enhancement convolutional neural network of the present invention.

[0064] Figure 5 This is a schematic diagram of the self-group iterative upgrade device based on deep learning according to the present invention.

[0065] Figure 6 This is a schematic diagram of the self-iterative upgrade process based on deep learning in this invention.

[0066] Figure 7 This is a logical block diagram of the self-iterative upgrade module based on deep learning in this invention.

[0067] Figure 8 This is a schematic diagram of the process of iterative upgrading of a shared dataset based on local single-machine training of a group using deep learning, as described in this invention.

[0068] Figure 9 This is a logical block diagram of the deep learning-based group local single-machine training shared dataset iterative upgrade module of the present invention.

[0069] Figure 10 This is a schematic diagram of the process of group local single-machine training and weight iterative upgrade based on deep learning in this invention.

[0070] Figure 11 This is a logical block diagram of the deep learning-based group local single-machine training shared dataset and weight iterative upgrade module of the present invention.

[0071] Figure 12 This is a schematic diagram of the process of iterative upgrade of group local distributed training based on deep learning in this invention.

[0072] Figure 13 This is a logical block diagram of the group-local distributed iterative upgrade module based on deep learning in this invention.

[0073] Figure 14 This is a schematic diagram of the process of the cloud-based group iterative upgrade based on deep learning in this invention.

[0074] Figure 15This is a logical block diagram of the cloud-based group iterative upgrade module based on deep learning, which is the subject of this invention.

[0075] Figure 16 This is a schematic diagram of the network structure for group local single-machine training and iterative upgrading based on deep learning, as described in this invention.

[0076] Figure 17 This is a schematic diagram of the network structure of the present invention, which is based on deep learning for group local distributed and group cloud iterative upgrade.

[0077] Figure 18 This is a schematic diagram of the structure of a computer system used to implement the embodiments of the methods, systems, and apparatus of the present invention.

[0078] It should be noted that, Figure 16 and Figure 17 For illustration purposes only. The number of devices is not limited to 5 or more. The number of devices for group iterative upgrade can be 2, 3 or more. Detailed Implementation

[0079] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0080] See Figure 1 The first embodiment of the present invention provides a method for enhancing blood vessels in endoscopic fluorescence images based on a deep learning U-shaped network framework, comprising the following steps:

[0081] The raw fluorescence image output from the endoscope is acquired and input into a U-shaped convolutional neural network to obtain the endoscopic image with enhanced blood vessels.

[0082] The training method for the U-shaped blood vessel enhancement convolutional neural network is as follows:

[0083] Step S100: Obtain fluorescence images of biological tissues using an endoscopic fluorescence camera and segment them into 256*256 pixel sizes as the original images; apply various detail enhancement algorithms to process the original images to obtain fluorescence images with enhanced blood vessel details, which are used as the target images (labels).

[0084] In some preferred embodiments, the original image and the target image (label) are acquired using the following method:

[0085] Step S110: Use an endoscopic fluorescence camera to acquire raw fluorescence images of biological tissues and divide them into 256*256 pixel sizes as raw images;

[0086] In step S120, the original image described in step S110 is processed using various detail enhancement algorithms, such as the contrast-limited adaptive histogram equalization algorithm (CLAHE), to obtain a fluorescence image with enhanced blood vessel details, which serves as the target image (label).

[0087] Step S130: The original image and the target image (label) are randomly divided into a training set, a validation set and a test set according to a reasonable ratio (here it can be set to 7:2:1) for model training.

[0088] In step S200, the original endoscopic image and the target image (label) are used as a training set to construct and train a U-shaped blood vessel enhancement convolutional neural network. The training process uses a computing platform equipped with an NVIDIA graphics card (Tesla A800 with 80GB).

[0089] In some preferred embodiments, the U-shaped vessel-enhancing convolutional neural network is constructed as follows:

[0090] See Figure 2 The U-shaped vessel enhancement convolutional neural network is composed of two VM-UNet sets and a large medical segmentation model encoder cross-connected. The large medical segmentation model is an image segmentation model based on Vision Transformer (a large model based on the Transformer framework released by Google in 2020) for feature extraction, possessing strong versatility and generalization ability while adapting to the characteristics of medical images. During the training of the entire network, the encoder of the large medical segmentation model does not participate in parameter correction.

[0091] VM-UNet is composed of an encoder and a decoder connected sequentially. See [link / reference] Figure 3 The encoder is composed of two VSS basic modules and a batch fusion module connected in an alternating sequence; the decoder is composed of a batch extension module and two VSS basic modules connected in an alternating sequence.

[0092] The VSS basic module consists of a normalized layer, a linear layer, a depthwise separable convolutional layer, an SS2D layer, a normalized layer, and a linear layer connected in sequence. (See [link to VSS module]) Figure 4 The SS2D module consists of three main parts: the expanding operation, the S6 block operation, and the merging operation.

[0093] The scanning expanding operation flattens the input two-dimensional feature map into a one-dimensional vector along four different directions (top left to bottom right, bottom left to top right, bottom right to top left, and top right to bottom left). This operation ensures that each element in the feature map can integrate information from all other locations in different directions, thereby forming a global receptive field.

[0094] The S6 block operation involves independently feeding the four one-dimensional vectors obtained from the expanding operation into the S6 block for feature extraction. The S6 block selectively scans the spatial state sequence model, interacting with previously scanned information for each element, effectively reducing the quadratic complexity to linear. This step ensures that information in all directions is thoroughly scanned, thereby capturing diverse features.

[0095] The scanning merging operation involves adding and merging sequences from four directions to restore the output image to the same size as the input image. This step integrates global information and outputs a two-dimensional feature map containing rich feature information.

[0096] In some preferred embodiments, the training method for the U-shaped vessel enhancement convolutional neural network is as follows:

[0097] Step S210: Input the original image into the first encoder network and the first decoder network in sequence to obtain the first training image;

[0098] Step S220: Input the original image and the first training image into the medical segmentation large model encoder network to obtain the second training image;

[0099] Step S230: Input the original image into the second encoder network to obtain the third training image;

[0100] Step S240: Input the second training image and the third training image into the second decoder network to obtain the fourth training image;

[0101] In some preferred embodiments, the training method of the U-shaped vessel enhancement convolutional neural network further includes:

[0102] Step S250: Calculate the mixture loss function of the second training image and the target image (label). And according to the hybrid loss function The U-shaped vessel enhancement convolutional neural network is modified. The hybrid loss function... This can be expressed as:

[0103]

[0104] in, , , , , These are the weighting coefficients that balance the various loss functions.

[0105] definition For the target image, For the source image, For the predicted enhanced image, then

[0106] The loss function can be expressed as

[0107]

[0108] The loss function can be represented as

[0109]

[0110] To represent different scales; , represent the mean values ​​of the predicted enhanced image and the target image, respectively. , These represent the standard deviations of the predicted enhanced image and the target image, respectively. This represents the covariance between the predicted enhanced image and the target image; , This indicates the relative importance between two items. , This is a constant term.

[0111] The loss function can be expressed as

[0112]

[0113] This refers to VGG-19 (a classic convolutional neural network (CNN) model proposed by the Visual Geometry Group at Oxford University). Feature maps of each convolutional layer. Scalar , and These represent the number of channels, height, and width of the corresponding layer, respectively.

[0114] The loss function can be expressed as

[0115]

[0116] in This is a pre-trained discriminator network.

[0117] The loss function can be expressed as

[0118]

[0119] in , and These represent the number of channels, height, and width of the enhanced image, respectively.

[0120] In step S300, the original image obtained by the endoscope is input into the trained U-shaped blood vessel enhancement convolutional neural network, and the output is the blood vessel enhancement image.

[0121] Step S400: Input the original fluorescence image described in step S300 into the U-shaped vessel enhancement convolutional neural network self-population iterative upgrade device to correct the U-shaped vessel enhancement convolutional neural network.

[0122] See Figure 5 The self-group iterative upgrade device includes a single-machine self-iterative upgrade module and a networked group iterative upgrade module. The difference between self-upgrade and group upgrade lies in whether the training information (dataset, dataset size, dataset quality, etc.) originates from a local single machine (self-upgrade) or a networked group (group upgrade). The group iterative upgrade module includes two functions: local group iterative upgrade and cloud group iterative upgrade. The difference between local group upgrade and cloud group upgrade lies in whether the training process is completed locally or in the cloud. The training device can be local, in the cloud, or both. The local group iterative upgrade includes two functions: local group single-machine training iterative upgrade and local group distributed training iterative upgrade. The local group single-machine training iterative upgrade includes two functions: local group single-machine training with shared dataset iterative upgrade (without sharing training weights) and local group single-machine training with shared dataset and weights iterative upgrade.

[0123] The deep learning-based self-iterative upgrade device includes:

[0124] A storage module is used to collect the raw fluorescence images input to the U-shaped blood vessel enhancement convolutional neural network;

[0125] The expert annotation module is used to annotate the original fluorescence images of the storage module and generate corresponding labels;

[0126] The neural network fine-tuning module is used to correct and fine-tune the U-shaped blood vessel enhancement convolutional neural network based on the images collected by the storage module.

[0127] See Figure 6 , Figure 7The self-iterative upgrade step based on the self-iterative upgrade device is to input the original fluorescence image and corresponding label mentioned in step S300 into the U-shaped blood vessel enhancement convolutional neural network for training.

[0128] The deep learning-based group iterative upgrade device includes:

[0129] A storage module is used to collect the raw fluorescence images input to the U-shaped blood vessel enhancement convolutional neural network;

[0130] The expert annotation module is used to annotate the original fluorescence images of the storage module and generate corresponding labels;

[0131] The neural network fine-tuning module is used to correct and fine-tune the U-shaped blood vessel enhancement convolutional neural network based on the images collected by the storage module.

[0132] The network service module is used for communication and data transmission with the cloud control center.

[0133] See Figure 8 , Figure 9 , Figure 16 The group-based local single-machine training shared dataset iterative upgrade based on the group iterative upgrade module includes:

[0134] The training dataset comes from all networked devices;

[0135] The original fluorescence image and corresponding label described in step S300, which are shared by several other networked devices, are downloaded through the network module and input into the U-shaped blood vessel enhancement convolutional neural network of the current local device. The network is trained locally to obtain the corrected weights of the U-shaped blood vessel enhancement convolutional neural network for use by the current local device.

[0136] See Figure 10 , Figure 11 , Figure 16 The group-based local single-machine training shared dataset and weight iterative upgrade based on the group iterative upgrade module includes:

[0137] The training dataset comes from all networked devices;

[0138] Prerequisite: All networked devices must use the same training model software;

[0139] The original fluorescence images and corresponding tags described in step S300 of several other networked devices are downloaded to the current local device through the network module and input into the U-shaped vessel enhancement convolutional neural network of the current local device. Then, the network device is trained to obtain the corrected weights of the U-shaped vessel enhancement convolutional neural network. The weights of the U-shaped vessel enhancement convolutional neural network of the current local device are updated and shared with all networked devices in real time for use by all networked devices.

[0140] See Figure 12 , Figure 13 , Figure 17 The group-based local distributed training iterative upgrade based on the group iterative upgrade module includes:

[0141] Training information (dataset size, dataset quality, etc.) comes from all networked devices;

[0142] Prerequisites: Either all networked devices use the same training model software (which requires knowledge of the dataset size for each device), or the quantitative differences in the parameters of the training datasets generated by each device are known (which requires knowledge of the dataset quality for each device).

[0143] The original fluorescence images and corresponding labels described in step S300 of each current local device are input into the U-shaped vessel enhancement convolutional neural network of the current local device for training, and the corrected weights of the U-shaped vessel enhancement convolutional neural network are obtained. The corrected weights of the U-shaped vessel enhancement convolutional neural network of several networked devices are uploaded to the cloud control center for global reduction (AllReduce). If the training model software of all networked devices is the same, then global reduction is performed according to the size of the dataset of each device. If the training model software of the networked devices is different and the quantitative difference of the dataset parameters of each device is known, then global reduction is performed according to the quantitative difference of the dataset quality of each device. The updated weights of the U-shaped vessel enhancement convolutional neural network are obtained.

[0144] See Figure 14 , Figure 15 , Figure 17 The group cloud-based iterative upgrade based on the group iterative upgrade module includes:

[0145] Prerequisite: The training model software must be identical across all networked devices and cloud-based devices;

[0146] The original fluorescence images and corresponding tags described in step S300 of several networked devices are transmitted to the cloud control center through the network service module; the cloud control center summarizes the received original fluorescence images and corresponding tags, and performs U-shaped vessel enhancement convolutional neural network training; the local network service module downloads and synchronously updates the weights of the current device's U-shaped vessel enhancement convolutional neural network.

[0147] A second embodiment of the present invention provides a method for enhancing blood vessels in endoscopic fluorescence images based on a deep learning U-shaped network framework. The system includes: an acquisition module and a training module.

[0148] The acquisition module is configured to acquire the original endoscope image and input it into a trained U-shaped vessel enhancement convolutional neural network to obtain an output image with enhanced vessels.

[0149] The training method for the U-shaped neural network used for endoscopic fluorescence image vascular enhancement is as follows:

[0150] The training module is configured to acquire endoscopic fluorescence images after vascular enhancement is completed;

[0151] The original endoscopic image and target images (labels) processed by various detail enhancement algorithms, such as the contrast-limited adaptive histogram equalization (CLAHE), were used as the training set to construct and train a U-shaped vessel enhancement convolutional neural network.

[0152] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] It should be noted that the endoscopic fluorescence imaging system based on deep learning for endoscopic image vascular enhancement provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0154] An electronic device according to a third embodiment of the present invention includes: at least one processor and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described method for enhancing endoscopic fluorescence images of blood vessels based on a deep learning U-shaped network framework.

[0155] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer instructions, which are executed by the computer to implement the above-described method for enhancing endoscopic fluorescence images of blood vessels based on a deep learning U-shaped network framework.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0157] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0158] The following is for reference. Figure 18 It shows a schematic diagram of the structure of a computer system for implementing embodiments of the methods, systems, and apparatus of the present invention. Figure 18 The server shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0159] like Figure 18 As shown, the computer system includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in Read Only Memory (ROM) 502 or programs loaded from storage section 508 into Random Access Memory (RAM) 503. RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0160] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0161] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined in the methods of this invention. It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0162] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0165] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0166] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for enhancing blood vessels in endoscopic fluorescence images based on a deep learning U-shaped network framework, characterized in that, The method includes: The raw fluorescence image output from the endoscope is acquired and input into a U-shaped vascular enhancement convolutional neural network to obtain the endoscopic image after vascular enhancement. The training method for the U-shaped blood vessel enhancement convolutional neural network is as follows: Step S100: Acquire images of biological tissue using an endoscopic fluorescence camera and segment them into 256*256 pixel sizes as the original images; apply various detail enhancement algorithms to process the original images to obtain fluorescence images with enhanced blood vessel details, which are used as the target images (labels). Step S200: The original fluorescence image of the endoscope and the fluorescence image after blood vessel enhancement are used as training sets to construct and train a U-shaped blood vessel enhancement convolutional neural network; the training process uses a computing platform equipped with an NVIDIA graphics card (Tesla A800 with 80GB); Step S300: Input the original image obtained by the endoscopic fluorescence camera into the trained U-shaped blood vessel enhancement convolutional neural network, and output the enhanced blood vessel image. Step S400: Input the original fluorescence image of the endoscope into the U-shaped vessel enhancement convolutional neural network self-population iterative upgrade device to correct the U-shaped vessel enhancement convolutional neural network.

2. The endoscopic fluorescence image vessel enhancement method based on a deep learning U-shaped network framework according to claim 1, characterized in that, The original image and the target image (label) are acquired using the following method: Step S110: Use an endoscopic fluorescence camera to acquire raw fluorescence images of biological tissues and divide them into 256*256 pixel sizes as raw images; In step S120, the original image described in step S110 is processed using various detail enhancement algorithms, such as the contrast-limited adaptive histogram equalization algorithm (CLAHE), to obtain a fluorescence image with enhanced blood vessel details, which serves as the target image (label). Step S130: The original image and the target image (label) are randomly divided into a training set, a validation set and a test set according to a reasonable ratio (here it can be set to 7:2:1) for model training.

3. The endoscopic fluorescence image vessel enhancement method based on a deep learning U-shaped network framework according to claim 1, characterized in that, The U-shaped blood vessel enhancement convolutional neural network is constructed as follows: Referring to Figure 2, the U-shaped vessel enhancement convolutional neural network is composed of two sets of VM-UNet and a large medical segmentation model encoder connected in a cross-connection. The large medical segmentation model is an image segmentation model based on Vision Transformer (a large model based on the Transformer framework launched by Google in 2020) for feature extraction. It has strong versatility and generalization ability and is adapted to the characteristics of medical images. During the training of the entire network, the encoder of the large medical segmentation model does not participate in parameter correction. VM-UNet consists of an encoder and a decoder connected sequentially, as shown in Figure 3. The encoder is composed of two VSS basic modules and a batch fusion module connected sequentially in an alternating combination. The decoder is composed of a batch extension module and two VSS basic modules connected sequentially in an alternating combination. The VSS basic module consists of a normalized layer, a linear layer, a depthwise separable convolutional layer, an SS2D layer, a normalized layer, and a linear layer connected in sequence, as shown in Figure 4. The SS2D module consists of three main parts: the expanding operation, the S6 block operation, and the merging operation. The scanning expansion operation flattens the input two-dimensional feature map into a one-dimensional vector along four different directions (top left to bottom right, bottom left to top right, bottom right to top left, top right to bottom left); this operation ensures that each element in the feature map can integrate information from all other locations in different directions, thereby forming a global receptive field. The S6 block operation: The four one-dimensional vectors obtained in the previous step are independently fed into the S6 block for feature extraction; The S6 block selectively scans the spatial state sequence model, interacting with each element with the information scanned before, effectively reducing the quadratic complexity to linear; This step ensures that information in all directions is thoroughly scanned, thereby capturing different features; The scanning merging operation involves adding and merging sequences from four directions, and restoring the output image to the same size as the input two-dimensional feature map. This step integrates global information and outputs a two-dimensional feature map containing rich feature information.

4. The endoscopic fluorescence image vessel enhancement method based on a deep learning U-shaped network framework according to claim 1, characterized in that, The training method for the U-shaped blood vessel enhancement convolutional neural network is as follows: Step S210: Input the original image into the first encoder network and the first decoder network in sequence to obtain the first training image; Step S220: Input the original image and the first training image into the medical segmentation large model encoder network to obtain the second training image; Step S230: Input the original image into the second encoder network to obtain the third training image; Step S240: Input the second training image and the third training image into the second decoder network to obtain the fourth training image.

5. The U-shaped vessel enhancement convolutional neural network training method according to claim 4, characterized in that, The formula for the hybrid loss function L can be expressed as: in, , , , , These are the weighting coefficients that balance the various loss functions; definition For the target image, The source image, For the predicted enhanced image, then The loss function can be expressed as The loss function can be represented as To represent different scales; , represent the mean values ​​of the predicted enhanced image and the target image, respectively. , These represent the standard deviations of the predicted enhanced image and the target image, respectively. This represents the covariance between the predicted enhanced image and the target image; , This indicates the relative importance between two items. , For constant terms; The loss function can be expressed as This refers to VGG-19 (a classic convolutional neural network (CNN) model proposed by the Visual Geometry Group at Oxford University). Feature maps of each convolutional layer; scalar , and These represent the number of channels, height, and width of the corresponding layer, respectively. The loss function can be expressed as in For a pre-trained discriminator network; The loss function can be expressed as in , and These represent the number of channels, height, and width of the enhanced image, respectively.

6. A self-group iterative upgrade device based on deep learning, characterized in that, The self-group iterative upgrade device includes a single-machine self-iterative upgrade module and a networked group iterative upgrade module. The difference between self-upgrade and group upgrade lies in whether the training information (dataset, dataset size, dataset quality, etc.) originates from a local single machine (self-upgrade) or a networked group (group upgrade). The group iterative upgrade module includes two functions: local group iterative upgrade and cloud group iterative upgrade. The difference between local group upgrade and cloud group upgrade lies in whether the training process is completed locally or in the cloud. The training device can be local, in the cloud, or both. The local group iterative upgrade includes two functions: local group single-machine training iterative upgrade and local group distributed training iterative upgrade. The local group single-machine training iterative upgrade includes two functions: local group single-machine training with shared dataset iterative upgrade (without sharing training weights) and local group single-machine training with shared dataset and weights iterative upgrade.

7. The self-iterative upgrade module based on deep learning according to claim 6, characterized in that, The deep learning-based self-iterative upgrade module includes: Storage module for collecting the raw fluorescence images of the endoscope input to the U-shaped vessel enhancement convolutional neural network; The expert annotation module is used to annotate the original fluorescence images of the storage module and generate corresponding labels; The neural network fine-tuning module is used to correct and fine-tune the U-shaped blood vessel enhancement convolutional neural network based on the images collected by the storage module. The self-iterative upgrade step based on the self-iterative upgrade device is to input the original fluorescence image of the endoscope into the U-shaped vessel enhancement convolutional neural network and train the U-shaped vessel enhancement convolutional neural network.

8. The deep learning-based population iterative upgrade module according to claim 6, characterized in that, The deep learning-based group iterative upgrade module includes: Storage module for collecting the raw fluorescence images of the endoscope input to the U-shaped vessel enhancement convolutional neural network; The expert annotation module is used to annotate the original fluorescence images of the storage module and generate corresponding labels; The neural network fine-tuning module is used to correct and fine-tune the U-shaped blood vessel enhancement convolutional neural network based on the images collected by the storage module. The network service module is used for communication and data transmission with other devices or cloud control centers.

9. The group local single-machine training shared dataset iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 6, characterized in that, The iterative upgrade of the shared local single-machine training dataset based on the group iterative upgrade module includes: The training dataset comes from all networked devices; The original fluorescence images of the endoscope shared by several other networked devices are downloaded via a network module and input into the U-shaped vessel enhancement convolutional neural network of the current local device. The network is trained locally to obtain the corrected weights of the U-shaped vessel enhancement convolutional neural network for use by the current local device.

10. The group local single-machine training shared dataset and weight iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 6, characterized in that, The group-based local single-machine training shared dataset and weight iterative upgrade based on the group iterative upgrade module includes: The training dataset comes from all networked devices; Prerequisite: All networked devices must use the same training model software; The original fluorescence images of the endoscopes from several other networked devices are downloaded to the current local device via the network module and input into the U-shaped vessel enhancement convolutional neural network of the current local device. The network is then trained to obtain the corrected weights of the U-shaped vessel enhancement convolutional neural network. The weights of the U-shaped vessel enhancement convolutional neural network of the current local device are updated and shared with all networked devices in real time for their use.

11. The group-local distributed training iterative upgrade function of the deep learning-based group iterative upgrade module according to claim 6, characterized in that, The group-based local distributed training iterative upgrade based on the group iterative upgrade module includes: Training information (dataset size, dataset quality, etc.) comes from all networked devices; Prerequisites: Either all networked devices use the same training model software (which requires knowledge of the dataset size for each device), or the quantitative differences in the parameters of the training datasets generated by each device are known (which requires knowledge of the dataset quality for each device). The original fluorescence images of the endoscopes from each local device are input into the U-shaped vessel enhancement convolutional neural network of the local device for training, resulting in corrected weights for the U-shaped vessel enhancement convolutional neural network. The corrected weights of the U-shaped vessel enhancement convolutional neural networks from several networked devices are then uploaded to the cloud control center for global reduction (AllReduce). If the training model software of all networked devices is the same, global reduction is performed based on the size of each device's dataset. If the training model software of the networked devices is different and the quantitative differences in the parameters of each device's dataset are known, global reduction is performed based on the quantitative differences in the quality of each device's dataset. This yields updated weights for the U-shaped vessel enhancement convolutional neural network.

12. The group cloud-based iterative upgrade function of the group iterative upgrade module based on deep learning according to claim 6, characterized in that, The group cloud-based iterative upgrade based on the group iterative upgrade module includes: Prerequisite: The training model software must be identical across all networked devices and cloud-based devices; The raw fluorescence images of the endoscopes from several networked devices are transmitted to the cloud control center via the network service module; the cloud control center aggregates the received raw fluorescence images of the endoscopes and trains a U-shaped vessel enhancement convolutional neural network; the local network service module downloads and synchronously updates the weights of the current device's U-shaped vessel enhancement convolutional neural network.

Images