GPU-based medical endoscope image processing method, system and medium

Abstract

The present invention belongs to the field of medical image processing, and provides a method, system and medium for processing medical endoscopic images based on GPU. The present invention utilizes the powerful parallel processing capability of GPU to significantly improve the speed and accuracy of endoscopic image acquisition, preprocessing, feature point detection, image correction and pathological feature recognition. The image quality is enhanced by executing GPU-accelerated filtering algorithms and histogram equalization, while the GPU-optimized SIFT algorithm effectively improves the accuracy of feature point matching. The estimation of the affine transformation model and the transformation correction of the image further correct the distortion and ensure the accuracy of the image data. Finally, the pre-trained deep learning model is used to implement fast convolutional neural network forward propagation on the GPU, identify pathological features in real time, and assist clinical diagnosis. The efficiency and accuracy of clinical diagnosis are greatly improved, providing doctors with instant diagnostic assistance, thereby improving the quality of medical services.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a GPU-based medical endoscope image processing method, system and medium. Background Art

[0002] With the development of endoscopic technology, endoscopes have gradually become an important auxiliary tool for clinical diagnosis and treatment. They can assist doctors in directly observing the patient's internal conditions and are crucial for the diagnosis of diseases of the gastrointestinal tract, respiratory tract and other cavities. When using endoscopes, high requirements are placed on the quality of endoscopic images. However, due to optical limitations and equipment resolution constraints, the images obtained by traditional endoscopes in narrow body cavities may not be clear enough due to lighting and field of view limitations. The cameras equipped with most endoscopes do not meet HD or UHD standards, and their resolution limits the doctor's ability to diagnose lesions, resulting in the inability to provide doctors with endoscopic images with sufficient details for diagnosis. Therefore, real-time processing of such endoscopic images is crucial, especially in high-resolution reconstruction, real-time correction and feature recognition.

[0003] How to provide an endoscopic image processing method that can provide high resolution, correct distortion in real time, and extract lesion features is crucial to assisting doctors in making more accurate diagnoses, so as to improve the efficiency and effectiveness of endoscopic image processing. Summary of the invention

[0004] The purpose of the present invention is to provide a GPU-based medical endoscope image processing method, system and medium, aiming to accelerate the image registration and correction process through parallel computing, achieve high-precision image correction, make full use of the computing resources of the GPU, and be able to reconstruct the collected endoscopic images with high resolution and real-time correction, and provide real-time diagnostic assistance for abnormal lesions.

[0005] To achieve the above object, the present invention provides the following technical solutions.

[0006] In a first aspect, the present invention provides a method for processing medical endoscope images based on a GPU, comprising the following steps:

[0007] Loading the acquired image sequence into GPU memory, removing noise from the loaded image sequence and enhancing image contrast to obtain a preprocessed endoscopic image;

[0008] Perform feature point detection on the preprocessed endoscopic image and extract feature descriptors, and estimate the affine transformation model of the matched feature points on the GPU;

[0009] According to the estimated affine transformation model, image remapping is performed; image interpolation and transformation correction operations are performed on the GPU to correct the distortion of the endoscopic image and perform boundary processing on the corrected image;

[0010] The processed endoscopic images are used to identify pathological features of the endoscopic images by accelerating the forward propagation of the convolutional neural network on the GPU;

[0011] The extracted pathological features are compared with the disease database in real time to automatically identify and mark the diseased areas in the endoscopic images.

[0012] The method of the present invention makes full use of the parallel computing power of GPU, which not only significantly improves the speed of endoscopic image processing, but also, through the integrated deep learning model, can realize real-time and accurate recognition of pathological features, providing doctors with immediate diagnostic assistance, thereby improving the efficiency and accuracy of clinical diagnosis.

[0013] As a further solution of the present invention, synthesizing multiple frames of images with different exposure levels into a high dynamic range (HDR) image includes the following steps:

[0014] An endoscopic camera rotating around the center is used to capture multiple frames of images with different exposure times at different predetermined angles, and low dynamic range (LDR) images captured at the same position are aligned at the pixel level;

[0015] Exposure fusion is performed on the aligned image sequence, and the weight is calculated according to the exposure time of each frame of the image. All frames of the images with different exposures are merged into one HDR image using the calculated weight;

[0016] The obtained HDR image is subjected to tone mapping processing to obtain a high dynamic range (HDR) endoscopic image displayed on a standard display device.

[0017] As a further solution of the present invention, when calculating the weight according to the exposure time of each frame of image, different weights are assigned according to the exposure time and pixel brightness value of each frame of image, and the weight calculation function is:

[0018]

[0019] Where z is the pixel value, Z min and Z max are the minimum pixel value and the maximum pixel value respectively.

[0020] As a further solution of the present invention, when all frames of images with different exposures are merged into one HDR image, each frame of the LDR image is aligned, and the weight and pixel value corresponding to each pixel are used to calculate the value of the HDR pixel in combination with all frames, and the high dynamic range brightness value of each pixel is reconstructed by solving a least squares according to the weight and pixel value of each frame of the image; wherein the HDR synthesis function is:

[0021]

[0022] Where E(i,j) is the brightness value of pixel (i,j) in the final HDR image, N is the total number of frames of the image, Z(i,j,n) is the value of pixel (i,j) in the nth frame, W is the weight function, g is the camera response function, and t n is the exposure time of the nth frame image.

[0023] Through the above steps, multiple frames of images with different exposure levels can be synthesized into an endoscopic image with a high dynamic range. This image sequence will help improve the accuracy and reliability of lesion detection in subsequent processing steps.

[0024] As a further solution of the present invention, in the image sequence preprocessing, image noise is removed by executing a GPU-accelerated filtering algorithm, including the following steps:

[0025] Transfer the acquired image sequence from the host memory to the GPU memory, and allocate parallel processing threads for each pixel of the image on the GPU;

[0026] Use bilateral filtering algorithm to calculate the weighted average value in the neighborhood of each pixel; synchronize the processing threads and transfer the processed image data back to the host memory;

[0027] GPU memory access and thread allocation are optimized, and the parameters of bilateral filtering, including the neighborhood size and the standard deviation of the kernel function, are adjusted according to the image characteristics to obtain an endoscopic image with image noise removed.

[0028] As a further solution of the present invention, when using the bilateral filtering algorithm to calculate the weighted average value in the neighborhood of each pixel, the neighborhood size of the bilateral filtering is determined, and the spatial kernel function σ of the bilateral filtering is set. S and the intensity kernel function σ T The standard deviation of each pixel in the neighborhood of each pixel is calculated, and the bilateral weight of each pixel in the neighborhood is added up, and the weighted values ​​of all neighboring pixels are divided by the sum of all weights to obtain the average value; where each pixel I in the endoscopic image p The value i′ after bilateral filtering p The calculation formula is:

[0029]

[0030] Where S is the neighborhood window centered on pixel p; I q is the intensity value of the neighborhood pixel q, and ω(p,q) is based on the total weight of the spatial kernel and the intensity kernel, and the calculation formula is:

[0031]

[0032] In the formula, is a spatial Gaussian function that gives weights based on spatial proximity; where (|pq|) is the spatial distance between pixels p and q;

[0033] is an intensity Gaussian function that gives weights based on pixel intensity differences; where (|I p -I q |) is the difference in pixel intensity values;

[0034] The Gaussian function is:

[0035] As a further solution of the present invention, histogram equalization is applied to enhance image contrast to obtain a preprocessed endoscopic image, comprising the following steps:

[0036] The image data is transferred to the GPU memory, and a parallel processing thread is assigned to each pixel of the image on the GPU. The histogram of the entire image is calculated in parallel. Each thread calculates the number of occurrences of the corresponding pixel value and updates the global histogram data.

[0037] The cumulative distribution function (CDF) of the histogram is calculated by a parallel scanning algorithm, and the CDF is normalized to fit the target image pixel value range;

[0038] Again, a thread is assigned to each pixel on the GPU, the value of the pixel corresponding to each thread is read, and the image contrast is enhanced by mapping it to a new pixel value according to the CDF mapping function, and the enhanced image data is transferred from the GPU memory back to the host.

[0039] As a further solution of the present invention, feature point detection is performed on the preprocessed image, and feature descriptors are extracted from the feature points, including the following steps:

[0040] Step 1: Construct scale space

[0041] Use GPU to perform multi-scale Gaussian blur on the input image, generate scale space, resample the image at different scales, and construct a Gaussian pyramid;

[0042] Step 2: Detect scale space extreme points

[0043] By comparing a pixel with 26 pixels in the adjacent scale and spatial neighborhood, potential key points are obtained, and the extreme value comparison of each pixel is processed in parallel on the GPU;

[0044] Step 3: Key point positioning

[0045] Taylor expansion fitting is used to refine key point positioning and filter out stable key points with high contrast and not on the edge;

[0046] Step 4: Generation of feature descriptors

[0047] By parallel computing the gradient information (gradient direction and magnitude) of the area surrounding the key points, a feature descriptor that reflects the local image features is constructed. The feature descriptor consists of a histogram of the gradient direction around the key points. The parallel computing capability of the GPU is used to quickly compute the feature descriptor of each key point.

[0048] Step 5: Feature point matching

[0049] The nearest neighbor search algorithm is used to match feature points, and the matching point pairs are determined by calculating the Euclidean distance between descriptors to obtain the detected feature points and extracted feature descriptors.

[0050] The present invention not only significantly improves the processing speed by adopting the SIFT algorithm accelerated by GPU, but also ensures the accuracy of feature point detection in complex endoscopic image sequences, providing a reliable basis for subsequent image correction and pathological feature recognition.

[0051] As a further solution of the present invention, the nearest neighbor search algorithm performs feature point matching, including the following steps:

[0052] Extract feature points from the two images to be matched respectively, and generate a descriptor for each feature point;

[0053] The feature point descriptor set of one image is used as a training set to construct a nearest neighbor search structure; for each feature point descriptor of another image, the most similar descriptor is quickly found through the nearest neighbor search structure constructed previously, which is the nearest neighbor;

[0054] Perform a low ratio test to exclude unmatched nearest neighbor points and second nearest neighbor points, and output matching pairs based on the nearest neighbor search results, each pair containing corresponding feature points in the two images.

[0055] The present invention uses the nearest neighbor search algorithm to perform feature point matching, which improves the accuracy and efficiency of matching and provides a technical basis for applications in the fields of image processing, image recognition, computer vision, etc.

[0056] As a further solution of the present invention, an affine transformation model is estimated on a GPU by matching feature points, and robustly eliminating abnormal matches, including the following steps:

[0057] Select feature point pairs from the matching results obtained by the nearest neighbor search algorithm, and calculate the affine transformation parameters of the matching feature point pairs in parallel on the GPU;

[0058] RANSAC is used to identify and exclude abnormal matching point pairs, a robust affine transformation model is defined from the set of internal points identified from the matching point pairs, and abnormal matching point pairs that do not conform to the affine transformation model are excluded from the original matching point pairs.

[0059] As a further solution of the present invention, the distortion of the endoscopic image is corrected, comprising the following steps: performing calculation of a distortion correction mapping matrix on a GPU to map the original position of each pixel to a corrected position;

[0060] Calculate pixel values ​​at non-integer coordinate positions in the mapping matrix by image interpolation;

[0061] Each thread assigned by the GPU outputs the processed pixel value to the new image buffer to complete the image distortion correction.

[0062] As a further solution of the present invention, the pathological features of the endoscopic image after image transformation and correction are identified, including the following steps:

[0063] Perform image preprocessing tasks on the GPU to optimize image data and use the transformed and corrected endoscopic images as input to the CNN. When preprocessing the endoscopic images, they are scaled to the input size required by the CNN network.

[0064] The pre-trained CNN model is loaded on the GPU, the pre-processed endoscopic images are fed into the CNN, and the forward propagation of the convolutional neural network is performed on the GPU. The fully connected layer converts the features extracted by the previous convolutional layer into classification results and outputs different types of pathological features of the endoscopic images.

[0065] In a second aspect, the present invention further provides a GPU-based medical endoscope image processing system, comprising:

[0066] The image acquisition module is used to acquire multiple frames of images at different angles using an endoscopic camera rotating around the center, and synthesize the multiple frames of images with different exposure levels into a high dynamic range (HDR) image to form an image sequence acquired by the endoscopic camera.

[0067] The image preprocessing module is used to load the acquired image sequence into the GPU memory, remove image noise by executing the GPU-accelerated filtering algorithm, and apply histogram equalization to enhance the image contrast to obtain the preprocessed endoscopic image.

[0068] The feature matching module is used to detect feature points of the preprocessed image using a GPU-optimized version of the Scale-Invariant Feature Transform (SIFT) algorithm, extract feature descriptors from the feature points, match the feature points using a nearest neighbor search algorithm, estimate the affine transformation model on the GPU through the matched feature points, and use robustness to exclude abnormal matches.

[0069] The image transformation correction module is used to perform image remapping according to the estimated affine transformation model; perform image interpolation and transformation correction operations on the GPU to correct the distortion of the endoscopic image, and perform boundary processing on the corrected image to remove the irregular boundaries caused by the transformation.

[0070] The pathological feature recognition module is used to accelerate the forward propagation of the convolutional neural network (CNN) on the GPU, identify the pathological features of the endoscopic image after image transformation and correction, compare the extracted pathological features with the disease database in real time, and automatically identify and mark the lesion area in the endoscopic image.

[0071] In a third aspect, the present invention further provides a storage medium storing at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the GPU-based medical endoscopic image processing method as described in the first aspect above.

[0072] Compared with the prior art, the GPU-based medical endoscope image processing method, system and medium provided by the present invention have the following beneficial effects:

[0073] 1. Accelerates image processing speed. The present invention uses the parallel processing capability of the GPU to significantly reduce image processing time, making it possible to quickly process large amounts of data, which is particularly critical for real-time endoscopic image acquisition. When synthesizing HDR images from frame images, the GPU can process multiple image frames in parallel, increasing the synthesis speed.

[0074] 2. Improved image quality. The present invention uses a GPU-accelerated filtering algorithm to remove noise while maintaining edge information, resulting in a clearer image. Histogram equalization can enhance image contrast, making the details in the image clearer and facilitating subsequent feature extraction and analysis.

[0075] 3. Efficient feature point detection and matching. The present invention can quickly detect and describe image feature points through the GPU-optimized SIFT algorithm, improving the speed and accuracy of feature matching. Through robust feature point matching and affine transformation estimation, the image can be accurately corrected and distortion eliminated, providing a reliable image basis for pathological diagnosis.

[0076] 4. Improved efficiency of image transformation and correction. The present invention can quickly complete image remapping and interpolation operations on the GPU, which is essential for real-time correction of endoscopic image distortion. Boundary processing can also be used to remove irregular boundaries after transformation, maintain the neatness of the image, and provide more accurate image data for subsequent analysis.

[0077] 5. Real-time pathological feature recognition. The present invention uses GPU-accelerated CNN forward propagation to achieve rapid recognition of pathological features in images. Pre-trained deep learning models can provide real-time feedback when running on GPUs, which is crucial for medical situations that are urgent or require immediate decision-making.

[0078] 6. Assisted clinical diagnosis. The automated lesion area identification in the present invention not only reduces the workload of doctors, but also reduces the errors caused by human judgment. The real-time comparison of the disease database can provide auxiliary diagnosis information immediately and increase the accuracy of clinical diagnosis.

[0079] In summary, the GPU-based medical endoscope image processing method, system, and medium of the present invention can provide doctors with fast and accurate diagnostic tools, improve the quality of medical services, and may have a positive impact on the treatment effect of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0081] Figure 1 A flowchart of a method for processing medical endoscope images based on GPU provided by the present invention;

[0082] Figure 2 A flowchart of synthesizing a high dynamic range (HDR) image in a GPU-based medical endoscope image processing method provided in an embodiment of the present invention;

[0083] Figure 3 A flowchart of removing image noise in a GPU-based medical endoscope image processing method provided in an embodiment of the present invention;

[0084] Figure 4A flowchart of applying histogram equalization to enhance image contrast in a GPU-based medical endoscope image processing method provided in an embodiment of the present invention;

[0085] Figure 5 A flowchart of a method for processing medical endoscope images based on a GPU using a nearest neighbor search algorithm for matching feature points in an embodiment of the present invention;

[0086] Figure 6 A flowchart of correcting the distortion of an endoscopic image in a GPU-based medical endoscopic image processing method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] The following description and the accompanying drawings fully illustrate the specific embodiments of this paper so that those skilled in the art can practice them. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims, and all available equivalents of the claims.

[0088] Herein, the terms "first", "second", etc. are used only to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "comprise", "comprising", or any other variants thereof are intended to cover non-exclusive inclusion, so that a structure, system, or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such structure, system, or device.

[0089] Due to optical limitations and equipment resolution constraints, the images obtained by traditional endoscopes in narrow body cavities may not be clear enough due to lighting and field of view limitations. The cameras equipped with most endoscopes do not meet HD or UHD standards, and their resolution limits doctors' ability to diagnose lesions, resulting in the inability to provide doctors with endoscopic images with sufficient details for diagnosis. Therefore, real-time processing of such endoscopic images is crucial, especially in high-resolution reconstruction, real-time correction, and feature recognition.

[0090] Based on this, the present invention provides a GPU-based medical endoscope image processing method, system and medium, which aims to accelerate the image registration and correction process through parallel computing, achieve high-precision image correction, make full use of the GPU's computing resources, and be able to reconstruct the collected endoscopic images with high resolution and real-time correction, and provide real-time diagnostic assistance for abnormal lesions.

[0091] The technical solution of the present invention is further described below in conjunction with specific embodiments:

[0092] See also Figure 1 As shown, an embodiment of the present invention provides a method for processing medical endoscope images based on a GPU, comprising the following steps:

[0093] Step S10: Endoscopic image acquisition

[0094] An endoscopic camera rotating around the center is used to acquire multiple frames of images at different angles. The acquired multiple frames of images have different exposure levels and are used to synthesize high dynamic range (HDR) images to form an image sequence acquired by the endoscopic camera.

[0095] Step S20: Image sequence preprocessing

[0096] The acquired image sequence is loaded into the GPU memory, the image noise is removed by executing the GPU-accelerated filtering algorithm, and the image contrast is enhanced by applying histogram equalization to obtain the preprocessed endoscopic image.

[0097] Step S30: Feature point detection and matching

[0098] The GPU-optimized version of the Scale-Invariant Feature Transform (SIFT) algorithm is used to detect feature points in the preprocessed image and extract feature descriptors from these feature points. The nearest neighbor search algorithm is used for feature point matching. The affine transformation model is estimated on the GPU through the matched feature points, and a robust method is used to eliminate abnormal matches.

[0099] Step S40: Image transformation and correction

[0100] Image remapping is performed based on the estimated affine transformation model. Image interpolation and transformation correction operations are performed on the GPU to correct the distortion of the endoscopic image. Boundary processing is performed on the corrected image to remove irregular boundaries caused by the transformation.

[0101] Step S50: Pathological feature recognition and real-time diagnosis assistance

[0102] The forward propagation of the convolutional neural network (CNN) is accelerated on the GPU to identify the pathological features of the endoscopic image after image transformation and correction. This step uses a pre-trained deep learning model to classify the features in the image to achieve real-time diagnostic assistance. By comparing the extracted pathological features with the disease database in real time, the lesion area in the endoscopic image is automatically identified and marked.

[0103] The method of the present invention makes full use of the parallel computing power of GPU, which not only significantly improves the speed of endoscopic image processing, but also, through the integrated deep learning model, can realize real-time and accurate recognition of pathological features, providing doctors with immediate diagnostic assistance, thereby improving the efficiency and accuracy of clinical diagnosis.

[0104] In the embodiments of the present invention, see Figure 2 As shown, synthesizing multiple frames of images with different exposure levels into a high dynamic range (HDR) image includes the following steps:

[0105] Step S101, using an endoscope camera rotating around a center to capture multiple frames of images with different exposure times at different predetermined angles, and aligning low dynamic range (LDR) images captured at the same position at a pixel level;

[0106] Step S102: performing exposure fusion on the aligned image sequence, calculating a weight according to the exposure time of each frame of the image, and merging all frames of the images with different exposures into one HDR image using the calculated weight;

[0107] Step S103: Perform tone mapping processing on the obtained HDR image to obtain a high dynamic range (HDR) endoscopic image displayed on a standard display device.

[0108] In this embodiment, when calculating the weight according to the exposure time of each frame of image, different weights are assigned according to the exposure time and pixel brightness value of each frame of image, and the weight calculation function is:

[0109]

[0110] Where z is the pixel value, Z min and Z max are the minimum pixel value and the maximum pixel value respectively.

[0111] When synthesizing multiple frames of images with different exposure levels into a high dynamic range (HDR) image, first, a series of multiple frames of images with different exposure times are captured at different preset angles using an endoscope camera that rotates around the center. These multiple frames represent low dynamic range (LDR) images at the same observation position and need to be accurately aligned at the pixel level to ensure the accuracy of image synthesis. During the alignment process, the GPU is used to process a large amount of data to achieve fast and accurate pixel matching. Then, the GPU continues to play its parallel processing advantages and performs exposure fusion on the aligned LDR image sequence. Each frame of the image is assigned a different weight according to its exposure time. These weights are used to guide the fusion of different exposure image frames to synthesize an HDR image containing richer lighting information. Finally, the synthesized HDR image is tone mapped. This step adjusts the dynamic range of the image to adapt it to the display capabilities of standard display devices while retaining image details and contrast as much as possible. The tone mapping process is also accelerated by the GPU to ensure real-time and efficient processing.

[0112] In this embodiment, when all frames of images with different exposures are merged into one HDR image, each frame of the LDR image is aligned, and the weight and pixel value corresponding to each pixel are used to calculate the value of the HDR pixel in combination with all frames. According to the weight and pixel value of each frame of the image, the high dynamic range brightness value of each pixel is reconstructed by solving a least squares method; wherein the HDR synthesis function is:

[0113]

[0114] Where E(i,j) is the brightness value of pixel (i,j) in the final HDR image, N is the total number of frames of the image, Z(i,j,n) is the value of pixel (i,j) in the nth frame, W is the weight function, g is the camera response function, and t n is the exposure time of the nth frame image.

[0115] Through the above steps, multiple frames of images with different exposure levels can be synthesized into an endoscopic image with a high dynamic range. This image sequence will help improve the accuracy and reliability of lesion detection in subsequent processing steps.

[0116] The HDR image synthesis method of this embodiment significantly improves the dynamic range and image quality of endoscopic imaging. Through efficient parallel processing of the GPU, fast image alignment, exposure fusion and tone mapping are achieved, and high-quality HDR images are synthesized, which plays a vital role in the identification and evaluation of details in medical diagnosis. This method not only improves the speed of image processing, but also improves the visual effect of the image through accurate dynamic range expansion, providing doctors with more accurate and detailed endoscopic views.

[0117] In the embodiments of the present invention, see Figure 3 As shown, in the image sequence preprocessing, the image noise is removed by executing the GPU accelerated filtering algorithm, including the following steps:

[0118] Step S201: transfer the acquired image sequence from the host memory to the GPU memory, and allocate a parallel processing thread to each pixel of the image on the GPU.

[0119] When performing image transmission and thread allocation in this step, the acquired endoscopic image sequence is transferred from the host's CPU memory to the GPU memory. Based on the GPU's unique parallel processing capabilities, one or more processing threads are allocated to each pixel point. The GPU is used for large-scale parallel computing, which can ensure the high efficiency of subsequent processing steps.

[0120] Step S202, using a bilateral filtering algorithm to calculate the weighted average value in the neighborhood of each pixel; synchronizing the processing threads, and transmitting the processed image data back to the host memory;

[0121] Among them, when using the bilateral filtering algorithm to calculate the weighted average value in the neighborhood of each pixel, the neighborhood size of the bilateral filter is determined, and the spatial kernel function σ of the bilateral filter is set S and the intensity kernel function σ T The standard deviation of each pixel in the neighborhood of each pixel is calculated, and the bilateral weight of each pixel in the neighborhood is added up, and the weighted values ​​of all neighboring pixels are divided by the sum of all weights to obtain the average value; where each pixel I in the endoscopic image p The value I′ after bilateral filtering p The calculation formula is:

[0122]

[0123] Where S is the neighborhood window centered on pixel p; I q is the intensity value of the neighborhood pixel q, and ω(p,q) is based on the total weight of the spatial kernel and the intensity kernel, and the calculation formula is:

[0124]

[0125] In the formula, is a spatial Gaussian function that gives weights based on spatial proximity; where (|pq|) is the spatial distance between pixels p and q;

[0126] is an intensity Gaussian function that gives weights based on pixel intensity differences; where (|I p -I q |) is the difference in pixel intensity values;

[0127] The Gaussian function is:

[0128] In the above steps, the bilateral filtering algorithm is used to process each pixel, and a weight can be assigned to each neighboring pixel of the target pixel. The weight is determined by the product of two Gaussian functions: one is based on spatial proximity, and the other is based on pixel value similarity. This can effectively remove noise while maintaining edge information.

[0129] Step S203, optimizing GPU memory access and thread allocation, adjusting the parameters of the bilateral filter according to the image characteristics, including the neighborhood size and the standard deviation of the kernel function, to obtain an endoscopic image with image noise removed.

[0130] In this step, all threads on the GPU synchronize after completing their respective filtering calculations to ensure that all pixels are processed correctly. The processed image data is then transferred back to the host memory. In order to better perform bilateral filtering, the parameter settings are optimized according to the characteristics of the endoscopic image, including determining the optimal neighborhood size and the standard deviation of the Gaussian kernel function. The neighborhood size determines the pixel range considered during filtering, and the standard deviation affects the width of the weight distribution. The above steps can quickly and effectively remove noise from endoscopic images, providing a clearer and more accurate image basis for subsequent image analysis and diagnosis.

[0131] In the embodiments of the present invention, see Figure 4 As shown, histogram equalization is applied to enhance the image contrast to obtain a preprocessed endoscopic image, including the following steps:

[0132] Step S301, transfer the image data to the GPU memory, assign parallel processing threads to each pixel of the image on the GPU, calculate the histogram of the entire image in parallel, each thread calculates the number of occurrences of the corresponding pixel value, and updates the global histogram data.

[0133] Step S302, calculating the cumulative distribution function (CDF) of the histogram by a parallel scanning algorithm, and normalizing the CDF to fit the target image pixel value range;

[0134] Step S303, again assigning a thread to each pixel point on the GPU, reading the value of the pixel corresponding to each thread, mapping it to a new pixel value according to the CDF mapping function to enhance the image contrast, and transmitting the enhanced image data from the GPU memory back to the host.

[0135] In the above steps, by transferring the image data from the host memory to the GPU memory, a parallel processing thread is assigned to each pixel of the image on the GPU. Each thread is responsible for calculating the number of times a pixel value appears in the image, and all threads jointly update a global histogram data structure. The cumulative distribution function (CDF) of the histogram is calculated using a parallel scanning algorithm (Blelloch scan), and the CDF value is scaled to the target image pixel value range. Exemplarily, for an 8-bit image, the CDF value is normalized to 0 to 255. Then a thread is assigned to each pixel point, and the CDF is used to map it to a new pixel value based on its original pixel value. After applying the CDF mapping, the image contrast is enhanced, and the processed image data is transferred from the GPU memory back to the host memory.

[0136] The present invention accelerates the entire process of histogram equalization through the GPU. Using the GPU for image preprocessing not only greatly improves the processing speed, but also makes it an ideal choice for GPU acceleration due to the parallel nature of the histogram equalization process. Thousands of threads on the GPU can process different parts of the image at the same time, greatly reducing the calculation time. After the calculation is completed, the image after equalization has better contrast, which is crucial for subsequent image analysis and medical diagnosis.

[0137] In an embodiment of the present invention, feature point detection is performed on the preprocessed image, and feature descriptors are extracted from the feature points, including the following steps:

[0138] Step 1: Construct scale space

[0139] Use GPU to perform multi-scale Gaussian blur on the input image, generate scale space, resample the image at different scales, and construct a Gaussian pyramid;

[0140] Step 2: Detect scale space extreme points

[0141] By comparing a pixel with 26 pixels in the adjacent scale and spatial neighborhood, potential key points are obtained, and the extreme value comparison of each pixel is processed in parallel on the GPU;

[0142] Step 3: Key point positioning

[0143] Taylor expansion fitting is used to refine key point positioning and filter out stable key points with high contrast and not on the edge;

[0144] Step 4: Generation of feature descriptors

[0145] By parallel computing the gradient information (gradient direction and magnitude) of the area surrounding the key points, a feature descriptor that reflects the local image features is constructed. The feature descriptor consists of a histogram of the gradient direction around the key points. The parallel computing capability of the GPU is used to quickly compute the feature descriptor of each key point.

[0146] Step 5: Feature point matching

[0147] The nearest neighbor search algorithm is used to match feature points, and the matching point pairs are determined by calculating the Euclidean distance between descriptors to obtain the detected feature points and extracted feature descriptors.

[0148] The present invention not only significantly improves the processing speed by adopting the SIFT algorithm accelerated by GPU, but also ensures the accuracy of feature point detection in complex endoscopic image sequences, providing a reliable basis for subsequent image correction and pathological feature recognition.

[0149] In the embodiments of the present invention, see Figure 5 As shown, the nearest neighbor search algorithm performs feature point matching, including the following steps:

[0150] Step S401, extracting feature points from two images to be matched respectively, and generating a descriptor for each feature point;

[0151] Step S402: Taking the feature point descriptor set of one image as a training set, constructing a nearest neighbor search structure; for each feature point descriptor of another image, quickly find the most similar descriptor through the nearest neighbor search structure constructed previously, that is, the nearest neighbor;

[0152] Step S403: perform a low ratio test to exclude unmatched nearest neighbor points and second nearest neighbor points, and output matching pairs based on the nearest neighbor search results, each pair containing corresponding feature points in the two images.

[0153] The present invention uses the nearest neighbor search algorithm to perform feature point matching, which improves the accuracy and efficiency of matching and provides a technical basis for applications in the fields of image processing, image recognition, computer vision, etc.

[0154] In an embodiment of the present invention, an affine transformation model is estimated on a GPU by matching feature points, and robustly eliminating abnormal matches, including the following steps:

[0155] Select feature point pairs from the matching results obtained by the nearest neighbor search algorithm, and calculate the affine transformation parameters of the matching feature point pairs in parallel on the GPU;

[0156] RANSAC is used to identify and exclude abnormal matching point pairs, a robust affine transformation model is defined from the set of internal points identified from the matching point pairs, and abnormal matching point pairs that do not conform to the affine transformation model are excluded from the original matching point pairs.

[0157] In the embodiments of the present invention, see Figure 6 As shown, correcting the distortion of the endoscopic image includes the following steps:

[0158] Step S501: Calculate the distortion correction mapping matrix on the GPU to map the original position of each pixel to the corrected position;

[0159] Step S502, calculating pixel values ​​at non-integer coordinate positions in the mapping matrix by image interpolation;

[0160] Step S503: Each thread assigned by the GPU outputs the processed pixel value to a new image buffer, thereby completing the distortion correction of the image.

[0161] In this embodiment, the distortion parameters of the endoscope camera obtained in advance are used to calculate the distortion correction mapping matrix, and for non-integer coordinates in the mapping matrix, bilinear interpolation or other interpolation algorithms are used to calculate the values ​​of the corrected pixels. The GPU thread is used to write the corrected pixel values ​​into the new image buffer to complete the distortion correction.

[0162] In an embodiment of the present invention, identifying the pathological features of the endoscopic image after image transformation and correction includes the following steps:

[0163] Perform image preprocessing tasks on the GPU to optimize image data and use the transformed and corrected endoscopic images as input to the CNN. When preprocessing the endoscopic images, they are scaled to the input size required by the CNN network.

[0164] The pre-trained CNN model is loaded on the GPU, the pre-processed endoscopic images are fed into the CNN, and the forward propagation of the convolutional neural network is performed on the GPU. The fully connected layer converts the features extracted by the previous convolutional layer into classification results and outputs different types of pathological features of the endoscopic images.

[0165] The embodiments of the present invention utilize the powerful parallel processing capability of GPU to significantly improve the speed and efficiency of endoscopic image processing and analysis. Through the nearest neighbor search algorithm for feature point matching, GPU parallel computing affine transformation estimation, distortion correction and CNN application, rapid enhancement of endoscopic images and accurate identification of pathological features are achieved, which is of great significance for improving the diagnostic value of endoscopic images and assisting medical decision-making.

[0166] In summary, the GPU-based medical endoscopic image processing method of the present invention can improve the quality of endoscopic images and increase the accuracy of pathological feature recognition. It not only improves the efficiency and accuracy of endoscopic image processing, but also has important value in improving clinical diagnosis efficiency and helping doctors make more accurate medical decisions. It brings innovative technological improvements to the field of medical image analysis, helps to improve the diagnostic capabilities of endoscopic examinations, and has a positive impact on patients' health and treatment outcomes.

[0167] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0168] In one embodiment, the present invention further provides a GPU-based medical endoscope image processing system, comprising the following modules:

[0169] The image acquisition module is used to acquire multiple frames of images at different angles using an endoscopic camera rotating around the center, and synthesize the multiple frames of images with different exposure levels into a high dynamic range (HDR) image to form an image sequence acquired by the endoscopic camera.

[0170] The image preprocessing module is used to load the acquired image sequence into the GPU memory, remove image noise by executing the GPU-accelerated filtering algorithm, and apply histogram equalization to enhance the image contrast to obtain the preprocessed endoscopic image.

[0171] The feature matching module is used to detect feature points of the preprocessed image using a GPU-optimized version of the Scale-Invariant Feature Transform (SIFT) algorithm, extract feature descriptors from the feature points, match the feature points using a nearest neighbor search algorithm, estimate the affine transformation model on the GPU through the matched feature points, and use robustness to exclude abnormal matches.

[0172] The image transformation correction module is used to perform image remapping according to the estimated affine transformation model; perform image interpolation and transformation correction operations on the GPU to correct the distortion of the endoscopic image, and perform boundary processing on the corrected image to remove the irregular boundaries caused by the transformation.

[0173] The pathological feature recognition module is used to accelerate the forward propagation of the convolutional neural network (CNN) on the GPU, identify the pathological features of the endoscopic image after image transformation and correction, compare the extracted pathological features with the disease database in real time, and automatically identify and mark the lesion area in the endoscopic image.

[0174] In this embodiment, the GPU-based medical endoscope image processing system of the present invention acquires images at different angles through an endoscope camera rotating around the center and synthesizes HDR images, which can significantly improve the dynamic range of the image, thereby enhancing the details of the image, especially in high-contrast visual scenes. This is crucial for identifying and diagnosing lesion areas because it can reveal details that may not be observed in conventional dynamic range images.

[0175] Moreover, by using GPU-accelerated filtering algorithms to remove noise and applying histogram equalization to enhance contrast, image preprocessing can be completed in a very short time, thus providing clearer and higher-quality images, laying a solid foundation for subsequent image analysis. By using the GPU-optimized SIFT algorithm to increase the speed of feature point detection and description, combined with the nearest neighbor search algorithm and a robust abnormal matching exclusion mechanism, this system can quickly and accurately match image feature points, providing a reliable foundation for image registration.

[0176] The GPU-based medical endoscope image processing system of the present invention can effectively correct the distortion of the endoscope image and process irregular boundaries based on the estimated affine transformation model, and the image remapping, interpolation and transformation correction operations on the GPU can ensure the image quality and provide more accurate information for the identification of pathological features. The pathological feature recognition module of the present system performs CNN forward propagation on the GPU. Such parallel computing greatly improves the computing speed, enabling the system to identify and mark the lesion area in the image in real time, and perform instant comparison with the disease database, greatly improving the accuracy and speed of diagnosis.

[0177] The present invention uses automated image processing and pathological feature recognition to enable doctors to obtain diagnosis results more quickly, reducing dependence on professionals and improving the efficiency of clinical operations. The automated processing of the system of the present invention reduces the possibility of human error, improves the consistency and repeatability of diagnosis, and thus improves the accuracy of diagnosis.

[0178] In summary, the GPU-based medical endoscopic image processing system of the present invention integrates modern image processing technology and deep learning algorithms, fully utilizes the powerful computing power of GPU, and provides clinicians with a fast and accurate tool to improve the quality of endoscopic examinations and the accuracy of diagnosis, ultimately promoting the health and treatment of patients. The GPU-based medical endoscopic image processing system adopts the steps of a GPU-based medical endoscopic image processing method as described above during execution. Therefore, the operation process of the GPU-based medical endoscopic image processing system in this embodiment will not be described in detail.

[0179] In one embodiment, a computer device is provided in an embodiment of the present invention, comprising at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes a method for processing medical endoscopic images based on a GPU, and the processor implements the steps of the method for processing medical endoscopic images based on a GPU when executing the instructions.

[0180] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions for causing a computer to execute the steps of a method for processing a medical endoscopic image based on a GPU.

[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program represented by computer instructions, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0182] Non-volatile memory may include read-only memory, magnetic tape, floppy disk, flash memory or optical storage, etc. Volatile memory may include random access memory or external cache memory. As an illustration and not limitation, RAM may be in various forms, such as static random access memory or dynamic random access memory, etc.

[0183] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing medical endoscope images based on GPU, characterized in that: The following steps are involved: An endoscopic camera rotating around the center is used to obtain multiple frames of images at different angles, and multiple frames of images with different exposure levels are synthesized into a high dynamic range image to form an image sequence collected by the endoscopic camera; Loading the acquired image sequence into GPU memory, removing noise from the loaded image sequence and enhancing image contrast to obtain a preprocessed endoscopic image; Perform feature point detection on the preprocessed endoscopic image and extract feature descriptors, and estimate the affine transformation model of the matched feature points on the GPU; According to the estimated affine transformation model, image remapping is performed; image interpolation and transformation correction operations are performed on the GPU to correct the distortion of the endoscopic image and perform boundary processing on the corrected image; The processed endoscopic images are used to identify pathological features of the endoscopic images by accelerating the forward propagation of the convolutional neural network on the GPU; The extracted pathological features are compared with the disease database in real time to automatically identify and mark the lesion areas in the endoscopic images; The process of synthesizing multiple frames of images with different exposure levels into a high dynamic range image includes the following steps: An endoscopic camera rotating around the center is used to capture multiple frames of images with different exposure times at different predetermined angles, and low dynamic range images captured at the same position are aligned at the pixel level; Exposure fusion is performed on the aligned image sequence, and the weight is calculated according to the exposure time of each frame of the image. All frames of the images with different exposures are merged into one HDR image using the calculated weight; Performing tone mapping on the obtained HDR image to obtain a high dynamic range endoscopic image displayed on a standard display device; Among them, feature point detection is performed on the preprocessed endoscopic image, and feature descriptors are extracted, including the following steps: Use GPU to perform multi-scale Gaussian blur on the input image, generate scale space, resample the image at different scales, and construct a Gaussian pyramid; By comparing a pixel with 26 pixels in the adjacent scale and spatial neighborhood, potential key points are obtained, and the extreme value comparison of each pixel is processed in parallel on the GPU; Taylor expansion fitting is used to refine key point positioning and filter out stable key points with high contrast and not on the edge; By parallel computing the gradient information of the area around the key point, a feature descriptor reflecting the local image features is constructed. The feature descriptor consists of a histogram of the gradient direction around the key point. The parallel computing capability of the GPU is used to quickly calculate the feature descriptor of each key point. Use the nearest neighbor search algorithm to match feature points, determine matching point pairs by calculating the Euclidean distance between descriptors, and obtain detected feature points and extracted feature descriptors; Among them, when matching feature points, a nearest neighbor search algorithm is used to match feature points, including the following steps: Extract feature points from the two images to be matched respectively, and generate a descriptor for each feature point; The feature point descriptor set of one image is used as a training set to construct a nearest neighbor search structure; for each feature point descriptor of another image, the most similar descriptor is quickly found through the nearest neighbor search structure constructed previously, which is the nearest neighbor; Perform a low ratio test to exclude unmatched nearest neighbor points and next nearest neighbor points, and output matching pairs based on the nearest neighbor search results, each pair contains corresponding feature points in the two images; Among them, estimating the affine transformation model on the GPU through the matched feature points includes the following steps: Select feature point pairs from the matching results obtained by the nearest neighbor search algorithm, and calculate the affine transformation parameters of the matching feature point pairs in parallel on the GPU; RANSAC is used to identify and exclude abnormal matching point pairs, a robust affine transformation model is defined from the set of internal points identified from the matching point pairs, and abnormal matching point pairs that do not conform to the affine transformation model are excluded from the original matching point pairs.

2. The method for processing medical endoscope images based on GPU according to claim 1, characterized in that: In the image sequence preprocessing, the image noise is removed by executing the GPU-accelerated filtering algorithm, which includes the following steps: Transfer the acquired image sequence from the host memory to the GPU memory, and allocate parallel processing threads for each pixel of the image on the GPU; Use bilateral filtering algorithm to calculate the weighted average value in the neighborhood of each pixel; synchronize the processing threads and transfer the processed image data back to the host memory; GPU memory access and thread allocation are optimized, and the parameters of bilateral filtering, including the neighborhood size and the standard deviation of the kernel function, are adjusted according to the image characteristics to obtain an endoscopic image with image noise removed.

3. The method for processing medical endoscope images based on GPU according to claim 2, characterized in that: When using the bilateral filtering algorithm to calculate the weighted average value in the neighborhood of each pixel, determine the neighborhood size of the bilateral filter and set the spatial kernel function σ of the bilateral filter. S and the intensity kernel function σ T The standard deviation of , calculate the bilateral weight of each pixel in the neighborhood of each pixel, add the weighted values ​​of all neighborhood pixels, and divide by the sum of all weights to get the average.

4. The method for processing medical endoscope images based on GPU according to claim 3, characterized in that: When enhancing the image contrast, histogram equalization is applied to enhance the image contrast, and a preprocessed endoscopic image is obtained, which includes the following steps: The image data is transferred to the GPU memory, and a parallel processing thread is assigned to each pixel of the image on the GPU. The histogram of the entire image is calculated in parallel. Each thread calculates the number of occurrences of the corresponding pixel value and updates the global histogram data. The cumulative distribution function of the histogram is calculated by a parallel scanning algorithm, and the cumulative distribution function is normalized to fit the target image pixel value range; Again, a thread is assigned to each pixel on the GPU, the value of the pixel corresponding to each thread is read, and the image contrast is enhanced by mapping it to a new pixel value according to the CDF mapping function, and the enhanced image data is transferred from the GPU memory back to the host.

5. A GPU-based medical endoscopic image processing system, used to execute the GPU-based medical endoscopic image processing method according to any one of claims 1 to 4, characterized in that: The system includes: An image acquisition module is used to acquire multiple frames of images at different angles using an endoscope camera rotating around a center, and synthesize the multiple frames of images at different exposure levels into a high dynamic range image to form an image sequence acquired by the endoscope camera; An image preprocessing module is used to load the acquired image sequence into the GPU memory, remove image noise by executing a GPU-accelerated filtering algorithm, and apply histogram equalization to enhance image contrast, thereby obtaining a preprocessed endoscopic image; A feature matching module is used to detect feature points of the preprocessed image using a GPU-optimized version of a scale-invariant feature transformation algorithm, extract feature descriptors from the feature points, match the feature points using a nearest neighbor search algorithm, estimate an affine transformation model on the GPU through the matched feature points, and use robustness to exclude abnormal matches; An image transformation correction module is used to perform image remapping according to the estimated affine transformation model; perform image interpolation and transformation correction operations on the GPU to correct the distortion of the endoscopic image, and perform boundary processing on the corrected image to remove irregular boundaries caused by the transformation; The pathological feature recognition module is used to accelerate the forward propagation of the convolutional neural network on the GPU, identify the pathological features of the endoscopic image after image transformation and correction, compare the extracted pathological features with the disease database in real time, and automatically identify and mark the lesion area in the endoscopic image.

6. A storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the steps of the GPU-based medical endoscopic image processing method described in any one of claims 1-4.

Images