Image processing system for fluorescent endoscope

Abstract

The invention relates to the technical field of medical image processing, in particular to an image processing system for a fluorescent endoscope, the system comprises a control layer, an equipment layer and a database layer, the equipment layer comprises a camera group, a display and an illumination light source, the camera group is used for collecting visible light images and fluorescent light images, and the display is used for displaying the visible light images and the fluorescent light images; the illumination light source comprises a white light source, a fluorescence excitation light source and a light splitting filtering assembly, the database layer is used for storing data collected by the system, the control layer comprises a CPU, an image processing module, a calculation module and an image output module, the CPU is used for managing and controlling operation of the system, and the image output module is used for outputting the image. The image processing module is used for fusing the visible light image and the fluorescence image to generate a visible light fusion image which only emits fluorescence by the diseased tissue; and the image output module is used for connecting the host with a display to output the visible light fusion image which only emits fluorescence by the diseased tissue in real time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and particularly relates to an image processing system for a fluorescence endoscope. BACKGROUND

[0002] At present, surgical resection as the main scheme for treating cancer is developing towards minimally invasive and precise direction. Electronic endoscope is one of the important observation means of minimally invasive surgery, which can help doctors to observe deep lesions and organ surface characteristics more easily compared with traditional fiber endoscope. However, in minimally invasive tumor resection surgery, doctors only rely on the visual information provided by the endoscope to identify tumors, which leads to the omission of small tumor lesions similar to normal tissues in color and shape. Fluorescence molecular imaging is a new imaging method, which increases the contrast between diseased tissues and normal tissues by labeling tumor cells with fluorescent agents. Since the observation effect of fluorescent agents under visible light is not obvious, the labeling effect is significant under lightless conditions. Therefore, in clinical practice, doctors manually turn on and off the illumination light source and observe the image difference information between normal tissues and tumor tissues through electronic endoscope. Due to benign lesions such as liver cirrhosis nodules and liver tissue inflammatory changes, the structure of small bile ducts in the liver is disordered or missing, causing indocyanine green molecule excretion disorder, which increases the false positive rate, so that doctors cannot accurately judge the lesion area.

[0003] A patent with publication number CN115644772A discloses an endoscope imaging system, which includes an endoscope subsystem, a multi-modal imaging subsystem connected with the optical fiber bundle thereof, a data acquisition subsystem, a control subsystem connected with the endoscope subsystem and the data acquisition subsystem respectively, and an image reconstruction subsystem. The control subsystem triggers the multi-modal imaging subsystem to sequentially generate OCT excitation light, photoacoustic excitation light, fluorescence excitation light and Raman excitation light in order based on a preset light emission timing pulse, which enters the endoscope subsystem through the optical fiber bundle; and based on the timing pulse, the data acquisition subsystem is synchronously triggered to collect electrical signals corresponding to each excitation light, so that the image reconstruction subsystem reconstructs OCT images, photoacoustic images, fluorescence images and Raman images based on the electrical signals respectively, which is beneficial to accurately obtaining effective biological feature information of the imaged tissue and meets the high-resolution imaging needs of users for the endoscopic imaging system.

[0004] A patent with publication number CN107744382A discloses an optical molecular image navigation system, which comprises: a visible light source provides visible light for collecting a background image; a near-infrared light source provides near-infrared light, which is an excitation light for generating fluorescence; an endoscope device, comprising: an objective lens converges and receives a mixed light beam of visible light and near-infrared light, the mixed light beam is a mixed reflected light after the visible light and the near-infrared light irradiate a to-be-imaged object; an image conversion structure converts the inverted image mixed light beam received by the objective lens into a positive image mixed light beam, and transmits the positive image mixed light beam to a light splitting device; the light splitting device splits the positive image mixed light beam, the split visible light is a light beam of the background image of the background environment, and the split near-infrared light is a light beam of the object fluorescence image of the to-be-imaged object; a camera device receives the background image and the object fluorescence image from the light splitting device; and an image processing device fuses and processes the background image and the object fluorescence image, and outputs a fused image.

[0005] The above application can detect the lesion tissue emitting ICG fluorescence, but when ICG excretion is blocked, both normal tissue and lesion tissue emit ICG fluorescence, and the position of the lesion tissue cannot be accurately provided. SUMMARY

[0006] The application aims to provide an image processing system for a fluorescence endoscope, which extracts normal tissue dyed by ICG by distinguishing the differences in fluorescence dyeing intensity, texture features and shape features of normal tissue and lesion tissue, improves the accuracy of a lesion area under ICG excretion blockage, and solves the problem of inaccurate judgment of a lesion area by a doctor due to indocyanine green excretion blockage.

[0007] The specific technical scheme of the application is as follows:

[0008] An image processing system for a fluorescence endoscope, which comprises a control layer, a device layer and a database layer, the device layer comprises a camera group, a display and an illumination light source, the camera group is used for collecting a visible light image and a fluorescence image, the illumination light source comprises a white light source, a fluorescence excitation light source and a light splitting filter assembly, the database layer is used for storing data collected by the system, the control layer comprises a CPU, an image processing module, a calculation module and an image output module, the CPU is used for managing and controlling the operation of the system, the image processing module is used for fusing the visible light image and the fluorescence image to generate a visible light fused image in which only lesion tissue emits fluorescence, and the image output module is used for connecting the display of a host computer to output images in real time.

[0009] Further, the light splitting filter assembly comprises a dichroic mirror, a first filter and a second filter, the dichroic mirror splits the incident light into a visible light branch and a fluorescent light branch, the first filter only passes visible light in the wavelength range of 400nm-700nm, filters excitation light and fluorescent light, and the second filter only passes fluorescent light in the wavelength range of 810nm-850nm, filters visible light and excitation light.

[0010] Further, the image processing module comprises an image denoising unit, which is configured to perform denoising processing on the collected visible light image and fluorescent image, and enhance the signal-to-noise ratio of the image signal.

[0011] Further, the calculation module comprises a threshold calculation unit and a feature extraction calculation unit, the threshold calculation unit calculates the specific position of the lesion tissue in the image by setting a threshold, and the feature extraction calculation unit extracts the position of the normal tissue dyed by ICG in the image and filters the fluorescent light at the normal tissue in the image.

[0012] Further, the calculation module comprises an ICG fluorescent dyeing lesion tissue extraction calculation strategy in the image, which comprises the following specific steps:

[0013] S101, calculating the fluorescent intensity threshold of the lesion tissue and the normal tissue according to the ICG dyeing fluorescent intensity;

[0014] S102, calculating the texture features and shape features different from the normal tissue and the lesion tissue, and extracting the normal tissue dyed by ICG fluorescent in the image;

[0015] Further, the threshold calculation unit comprises a threshold calculation strategy for ICG fluorescent dyeing lesion tissue in the image, which is:

[0016] S201, setting the mean value of the pixel values of all pixel points in the fluorescent image as , setting the segmentation threshold as , which is used to segment the pixel points with pixel values less than in the fluorescent image into a fluorescent background image , and the pixel points with pixel values greater than or equal to in the fluorescent image into a fluorescent target image , wherein the mean value of the pixel values of all pixel points in the fluorescent background image is , the mean value of the pixel values of all pixel points in the fluorescent target image is , and the probabilities of the pixel points in the fluorescent image being in are ,

[0017] S202, Based on the fluorescence background image The average pixel value of all pixels in Fluorescent target images The average pixel value of all pixels in And the pixels in the fluorescence image are divided into probability , Calculate the inter-class variance of gray values ​​in fluorescence images. ;

[0018] S203, when When the fluorescence image shows a pixel, it represents a lesion with high ICG staining intensity. When the fluorescence image is in the specified state, the pixels represent normal tissue with low ICG staining intensity or no ICG staining.

[0019] Furthermore, the feature extraction calculation unit includes the extraction of texture and shape features of the lesion tissue, and its calculation strategy is as follows:

[0020] S301. Calculation of texture features extracted from ICG fluorescent staining. Methods for extracting texture features include:

[0021] Using any pixel in the fluorescence image as a reference point, record the brightness value of the reference point;

[0022] Within a preset direction and a preset distance from the reference point, multiple similar points corresponding to the reference point are obtained by traversing through corner point features, and the brightness values ​​of the multiple similar points are recorded.

[0023] By statistically analyzing the frequencies corresponding to different brightness combinations among all similar points and reference points, the distribution pattern of neighborhood brightness in the fluorescence image can be obtained.

[0024] The texture energy feature value of the fluorescence image is calculated based on the distribution pattern, and the average energy value is calculated based on the texture energy feature value to obtain the texture feature.

[0025] S302. Extract and calculate the shape features of ICG fluorescence staining, and binarize the lesion tissue with high ICG staining intensity in the fluorescence image calculated by the threshold calculation unit to obtain a binary matrix image. The binary matrix image is converted into a matrix representation, with the geometric center as the starting point. The discretized geometric distribution of the starting point in the horizontal direction is calculated to obtain the variance index of the binary matrix image in the horizontal axis direction. Based on the variance index, the mean horizontal axis variance is calculated to obtain the shape features.

[0026] S303. When the texture energy of an image region in the fluorescence image is greater than or equal to the average energy value of the fluorescence image, and the corresponding horizontal axis variance is greater than or equal to the average horizontal axis variance, the corresponding image region is determined to belong to the lesion tissue region. The determined lesion tissue region is compared with the fluorescence target image obtained by the threshold calculation unit, and the normal tissue region is obtained by difference. The normal tissue region is removed from the fluorescence image to obtain a fluorescence image containing only the lesion tissue.

[0027] Furthermore, normal tissue areas were removed from the fluorescence images.

[0028] S401. Let the set of coordinates of all pixels in the lesion tissue region of the image extracted by the feature extraction calculation unit be... The set of coordinates of all pixels in the fluorescent target image is The logical judgment function determines whether the pixel coordinates of the lesion tissue region in the image extracted by the feature extraction calculation unit are the same as the pixel coordinates of the fluorescent target image.

[0029] S402. When the logical judgment function is true, that is, the pixel coordinates of the lesion tissue region extracted by the feature extraction calculation unit are the same as the pixel coordinates in the fluorescent target image, the gray value of the pixel coordinates is set to 255; when the logical judgment function is false, that is, the pixel coordinates of the lesion tissue region extracted by the feature extraction calculation unit are different from the pixel coordinates in the fluorescent target image, the gray value of the pixel coordinates is set to 0, and a fluorescent image of only the lesion tissue is obtained.

[0030] Furthermore, the image processing module also includes an image fusion unit, which is used to calculate the fusion of the visible light image acquired by the camera group and the fluorescence image of only the lesion tissue extracted by the feature extraction calculation unit to form a visible light fused image in which only the lesion tissue is labeled with ICG fluorescence.

[0031] Furthermore, the image fusion unit decomposes the visible light image and the fluorescence image of the lesion tissue extracted by the feature extraction calculation unit into a base layer image and a detail layer image, and then fuses the base layer image and the detail layer image to obtain a visible light fused image of the lesion tissue labeled with ICG fluorescence. The specific fusion steps are as follows:

[0032] S501. Let the visible light image and the fluorescence image be represented as follows: , The base image of a visible light image is The base image of the fluorescence image is Fluorescence images , the base layer image of fluorescence images The visible light detail layer image is obtained by subtracting the visible light image from the visible light base layer image. The fluorescence detail layer image is obtained by subtracting the fluorescence image from the fluorescence base layer image. ;

[0033] S502. The base layer images of the visible light image and the fluorescence image are fused using Gaussian filtering to obtain a base layer weight map. The fused base layer image is obtained by weighting the base layer image of the green channel of the visible light image and the fluorescence image through the base layer weight map.

[0034] S503. Perform mean filtering and median filtering on the visible light image and the fluorescence image to obtain the visual saliency features of the visible light image and the fluorescence image. Enhance the detail layer image to obtain the detail layer weight map of the visible light image and the detail layer weight map of the fluorescence image. Obtain the fused detail layer image by weighted averaging.

[0035] S504. A fused image of lesion tissue labeled with ICG fluorescence is obtained through the fused base layer image and the fused detail layer image.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. This invention can filter out normal tissue in ICG fluorescently stained tissue by distinguishing the texture and shape characteristics of normal tissue and lesion tissue when false positive fluorescence staining is caused by ICG excretion disorders, so that doctors can better determine the location of lesion tissue.

[0038] 2. By filtering normal tissue stained with ICG fluorescence, the monitor outputs a visible light image showing only the fluorescence emitted by the lesion tissue, enabling more precise excision of the lesion area. This further improves the accuracy of lesion area excision based on existing technology, while reducing the amount of normal tissue removed. Attached Figure Description

[0039] Figure 1 This is a system overall framework diagram of Embodiment 1 of the present invention;

[0040] Figure 2 This is an optical path diagram of the fluorescence collection system in Embodiment 1 of the present invention;

[0041] Figure 3 This is a flowchart of the overall calculation strategy in Embodiment 2 of the present invention;

[0042] Figure 4 This is a statistical module diagram for the maximum inter-class variance threshold calculation in Embodiment 2 of the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0046] Example 1

[0047] This embodiment utilizes the image processing system of a fluorescence endoscope to compare and distinguish the differences in fluorescence staining intensity, texture features, and shape features between normal tissue and lesion tissue in ICG false-positive fluorescently stained areas, thereby extracting normal tissue stained with ICG and improving the accuracy of lesion areas under ICG excretion disorders. The specific scheme is as follows: Figure 1 As shown, an image processing system for a fluorescence endoscope has a framework comprising a control layer, a device layer, and a database layer. The device layer includes a camera group, a display, and an illumination source. The camera group is used to acquire visible light images and fluorescence images. The illumination source includes a white light source, a fluorescence excitation source, and a spectral filtering component. The database layer is used to store the data acquired by the image system. The control layer includes a CPU, an image processing module, a computing module, and an image output module. The CPU is used to manage and control the operation of the system. The image processing module is used to fuse visible light images and fluorescence images to generate a visible light fused image in which only lesion tissue emits fluorescence. The image output module is used to connect the host to the display and output images in real time.

[0048] In this embodiment of the invention, the excitation source employs a 785nm multimode fiber-coupled infrared semiconductor laser, which can excite the dye ICG, whose near-infrared fluorescence peak is 830nm, thereby activating its fluorescence emission. The beam-splitting filter assembly includes a dichroic mirror, a first filter, and a second filter, such as... Figure 2As shown, the dichroic mirror is placed with a 45° incident angle, which divides the incident light into a visible light branch and a fluorescence branch. The first filter only allows visible light in the wavelength range of 400nm to 700nm, filtering out excitation light and fluorescence. The second filter only allows fluorescence in the wavelength range of 810nm to 850nm, filtering out visible light and excitation light.

[0049] Example 2

[0050] This embodiment calculates the specific location of the lesion tissue using a computing module and fuses the results to generate a visible light fused image where only the lesion tissue emits fluorescence. The computing module includes an overall computing strategy, such as… Figure 3 As shown, the overall calculation strategy includes the following specific steps:

[0051] a1. Calculate the fluorescence intensity thresholds of diseased and normal tissues based on ICG staining fluorescence intensity;

[0052] a2. Calculate the different texture and shape features between normal and diseased tissues, and extract the normal tissue stained with ICG fluorescence;

[0053] a3. Calculate the fusion of the visible light image acquired by the camera group with the fluorescent image of only the lesion tissue to form a visible light fused image of the lesion tissue labeled with ICG fluorescence.

[0054] In this embodiment of the invention, the threshold calculation unit includes a threshold calculation strategy for ICG fluorescently stained lesions in the image as follows:

[0055] b1. Set the mean pixel value of all pixels in the fluorescence image to be... Let the segmentation threshold be... Used to filter pixels with values ​​smaller than 1 in a fluorescence image. The pixels are segmented into fluorescent background images. In the fluorescence image, the pixel value is greater than or equal to The pixels are segmented into fluorescent target images. The fluorescent background image The mean pixel value of all pixels in the image is Fluorescent target image The mean pixel value of all pixels in the image is The pixels in the fluorescence image are divided into The probabilities are respectively , ,but ; ;

[0056] b2. Based on the fluorescent background image The average pixel value of all pixels in Fluorescent target images The average pixel value of all pixels in And the pixels in the fluorescence image are divided into probability , Calculate the inter-class variance of gray values ​​in fluorescence images. Among them, the inter-class variance of the gray values ​​in the fluorescence image is ;

[0057] b3. When When the fluorescence image shows a pixel, it represents a lesion with high ICG staining intensity. When the fluorescence image is in the specified state, the pixels represent normal tissue with low ICG staining intensity or no ICG staining.

[0058] In this embodiment of the invention, the threshold calculation of the maximum inter-class variance method consists of three FPGA modules: a statistics module, a calculation module, and a maximum value selection module.

[0059] c1. The statistics module consists of a histogram statistics module, a cumulative histogram statistics module, and a cumulative grayscale statistics module. For example... Figure 4 As shown, each statistical submodule in the histogram statistics module consists of a dual-port RAM with a depth of 256. Histogram statistics are completed by reading, incrementing, and writing to the RAM using grayscale levels as addresses. Since counting each pixel consumes 3 clock cycles, this invention uses three RAMs in a time-sharing manner to count pixels, trading area for speed. A simple three-state machine is defined, which sequentially activates the three statistical submodules during the pixel's validity period. When the first pixel arrives, submodule 1 initiates a read operation; after one clock cycle, the second pixel arrives, submodule 1 increments, and submodule 2 performs a read operation; when the third pixel arrives, submodule 1 writes, submodule 2 increments, and submodule 3 performs a read operation; when the fourth pixel arrives, submodule 1 initiates a read operation again, and so on, until the Nth pixel is counted.

[0060] Once the histogram statistics for one frame of image are complete, the cumulative histogram statistics module and the cumulative grayscale statistics module are started. Similar to the statistics submodule, both accumulation modules utilize a dual-port RAM with a depth of 256. The pipelined design allows for the completion of cumulative histogram and cumulative grayscale statistics within 256+2 clock cycles.

[0061] c2. After both cumulative modules are completed, the variance calculation module adopts a 5-stage pipeline structure. t is used as the cumulative histogram and cumulative grayscale address, and it cycles from 0 to 255 to calculate the intermediate variance corresponding to each grayscale level.

[0062] c3. Unlike the previous two modules, the maximum value selection module does not require the calculation module to complete all grayscale calculations before starting. After the first variance is calculated, this module starts every 255 clock cycles, compares the new variance value with the previous variance, updates the maximum variance value, and saves its grayscale level.

[0063] In this embodiment of the invention, the feature extraction calculation unit includes the extraction of texture features and shape features of normal tissue, and its calculation strategy is as follows:

[0064] d1. Calculation of texture features extracted from ICG fluorescent staining, where the extraction methods include:

[0065] Using any pixel in the fluorescence image as a reference point, record the brightness value of the reference point;

[0066] Within a preset direction and a preset distance from the reference point, multiple similar points corresponding to the reference point are obtained by traversing through corner point features, and the brightness values ​​of the multiple similar points are recorded.

[0067] By statistically analyzing the frequencies corresponding to different brightness combinations among all similar points and reference points, the distribution pattern of neighborhood brightness in the fluorescence image can be obtained.

[0068] The texture energy feature value of the fluorescence image is calculated based on the distribution pattern, and the average energy value is calculated based on the texture energy feature value to obtain the texture feature.

[0069] In some optional implementations, the texture feature extraction calculation for ICG fluorescent staining is denoted as f. For an image of size f, let L be the gray level of the image, and i be the gray value of a reference pixel coordinate point in the image f. j is the reference pixel coordinate point, the distance is d, and the direction is The grayscale value of the pixel coordinates. A two-dimensional Cartesian coordinate system is established with the reference pixel coordinate point as the origin. The x-axis is a horizontal line passing through the origin, and the y-axis is a line perpendicular to the x-axis passing through the origin. Let the origin be the initial point of the azimuth angle, with the positive x-axis at 0° and the negative x-axis at 180°. Let A be the grayscale matrix of image f, where grayscale matrix A represents all directions reachable from the reference pixel coordinate point with grayscale value i. Let the ratio of the total number of pixels with a distance of d and a gray value of j to the total number of pixels in image f be given. , Let f be the reference pixel coordinates. In the direction of The coordinates of a pixel with a distance of d and a gray value of j, and... Forming pixel coordinate pairs, direction , express The corresponding pixel grayscale value is i. express The corresponding pixel grayscale value is j; the energy value N is selected as the feature quantity of the texture, and the energy value... Let the average energy of the fluorescence image be ; ;

[0070] d2. Extract and calculate the shape features of ICG fluorescence staining, and binarize the lesion tissue with high ICG staining intensity in the fluorescence image calculated by the threshold calculation unit to obtain a binary matrix image. The binary matrix image is converted into a matrix representation, with the geometric center as the starting point; the discretized geometric distribution of the starting point in the horizontal direction is calculated to obtain the variance index of the binary matrix image in the horizontal axis direction; the average horizontal axis variance is calculated based on the variance index to obtain the shape features.

[0071] In some optional embodiments, the shape features of ICG fluorescence staining are extracted and calculated, and the lesions with high ICG staining intensity in the fluorescence image calculated by the threshold calculation unit are binarized to obtain a binary matrix image. Then the binary matrix image In the (p+q) order matrix representation, p and q are integers greater than or equal to zero. A (p+q)-order matrix represents a geometric matrix containing (p+q) orders. , The center point of the binary matrix image is represented by the discretized second-order geometric center distance variance along the horizontal axis, which is a feature quantity representing the shape. Let the mean variance of the horizontal axis of the fluorescence image be . ;

[0072] d3. When the texture energy of an image region in the fluorescence image is greater than or equal to the average energy value of the fluorescence image, and the corresponding horizontal axis variance is greater than or equal to the average horizontal axis variance, the corresponding image region is determined to belong to the lesion tissue region. The determined lesion tissue region is compared with the fluorescence target image obtained by the threshold calculation unit. The normal tissue region is obtained by difference. The normal tissue region is removed from the fluorescence image to obtain a fluorescence image containing only the lesion tissue.

[0073] In some optional implementations, when and In this case, the portion of the image stained with ICG fluorescence represents the lesion tissue extracted by the feature extraction calculation unit. The lesion tissue extracted by the feature extraction calculation unit With the fluorescent target image Normal tissue obtained by differential diagnosis ,but The normal tissue Filtering the image yields the diseased tissue extracted by the feature extraction calculation unit. Fluorescence images.

[0074] In this embodiment of the invention, the normal tissue is... The specific steps for filtering in an image are as follows:

[0075] e1, Suppose that the diseased tissue region in the image extracted by the feature extraction calculation unit is... The set of all pixel coordinates is The fluorescent target image The set of all pixel coordinates is Determine the diseased tissue region in the image extracted by the feature extraction calculation unit. Pixel coordinates and fluorescent target image The formula for determining whether the coordinates of the pixels are the same is... , Conditional function;

[0076] e2. When the logical judgment function is true. That is, the lesion tissue region extracted by the feature extraction calculation unit. Pixel coordinates and fluorescent target image If the pixel coordinates are the same, set the grayscale value of that pixel coordinate to 255; if the logical judgment function is false, That is, the lesion tissue region extracted by the feature extraction calculation unit. Pixel coordinates and fluorescent target image If the coordinates of a pixel are different, set the grayscale value of that pixel's coordinates to 0 to obtain only the diseased tissue area. Fluorescence images.

[0077] In this embodiment of the invention, the image fusion unit decomposes the visible light image and the fluorescence image of the lesion tissue extracted by the feature extraction calculation unit into a base layer image and a detail layer image, and then fuses the base layer image and the detail layer image to obtain a visible light fused image of the lesion tissue labeled with ICG fluorescence. The specific fusion steps are as follows:

[0078] f1. Let the visible light image and the fluorescence image be represented as follows: , The base image of a visible light image is The base image of the fluorescence image is Fluorescence images , the base layer image of fluorescence images The visible light detail layer image is obtained by subtracting the visible light image from the visible light base layer image. The fluorescence detail layer image is obtained by subtracting the fluorescence image from the fluorescence base layer image. ;

[0079] f2. The base layer image of the visible light image and the fluorescence image are fused using Gaussian filtering to obtain the base layer weight map. ,in Gaussian filtering, For the location of the lesion tissue information, when , indicating in The area contains diseased tissue. As a nonlinear function, the fused base image is obtained by weighting the base image and fluorescence image from the green channel of the visible light image using a base weight map. ;

[0080] f3. Apply mean filtering and median filtering to the visible light image and the fluorescence image to obtain the visual saliency features of the visible light image and the fluorescence image, respectively. , ,in , These are the visible light image and fluorescence image after mean filtering. , For the median-filtered visible light image and fluorescence image, detail layer image enhancement is performed to obtain the detail layer weight map of the visible light image. Detail layer weight map of fluorescence image K is the fluorescence enhancement coefficient, and the fused detail layer image is obtained by weighted averaging. ;

[0081] f4. Obtain a fused image of lesion tissue labeled with ICG fluorescence through the fused base layer image and the fused detail layer image. .

[0082] Example 3

[0083] In this embodiment, the image output module converts the YUV mode of the image synthesis into RGB mode and outputs it to the LCD. The image data from the DVP output is cached in the database layer and fused into images, maintaining the YUV format throughout, and is transmitted in 16-bit YUYV format each time. Since the input format of VGA data is set to RGB888, the YUYV is first converted into a 24-bit YUV transmission format through simple pixel copying, and then input to the image output module at the VGA pixel clock speed. The image output module is divided into two sub-modules: a VGA timing module and a YUV to RGB conversion module.

[0084] The g1 VGA timing module scans from left to right and from top to bottom. After scanning each line, it is synchronized with the horizontal sync signal, and the electron beam returns to the starting position of the next line on the left side of the screen. When all lines have been scanned, it is synchronized with the vertical sync signal, and the electron beam returns to the upper left of the screen. Simultaneously, vertical blanking is performed, and the next frame begins. Both timing and vertical timing require a sync pulse (Synca), a back porch (b), a display interval (c), and a front porch (d).

[0085] In VGA display mode, the pixel clock and the number of clock cycles for the four segments at different resolutions are represented by e / j, which indicates the total number of clock cycles for the row / column display, i.e., e = a + b + c + d. The logic design uses two 12-bit counters, incrementing by one on each dp_clk (74.25MHz) clock edge. a~d and f~i are represented by eight constants. The counters are compared with these constants to determine which segment the current moment falls into. The output signals vs and hs are then toggled accordingly within that segment. When both counter values ​​are within the row / column display timing segment (c / h), the data valid signal de is set to 1, and data is retrieved from the asynchronous FIFO of the read / write sequence control module, as shown in Table 1 for each resolution.

[0086] Table 1 Timing Tables for Each Resolution

[0087]

[0088] The g2 YUV to RGB module calculates by multiplying all coefficients by 256, taking the integer part as the multiplication / addition coefficient, and outputting the high 8 bits of the result. The conversion module has a two-stage pipeline: the first stage performs multiplication, the second stage performs addition, and the third stage performs subtraction. To handle potential negative numbers during subtraction, the minuend and subtrahend are compared before the final subtraction operation; if a negative number is generated, the result is set to zero.

[0089] ;

[0090] ;

[0091] ;

[0092] The layer0_rdreq signal (connected to the data enable signal de in the VGA timing module) is input to the read / write sequence control module for one clock cycle. After another clock cycle of read FIFO operation, asynchronous FIFO2 transmits pixel data layer0_ycbcr to the image output module. layer0_ycbcr undergoes format conversion over three clock cycles to obtain RGB data. For data alignment, the VGA timing module...

[0093] The VS, HS, and DE signals are delayed by two clock cycles before entering the YUV to RGB converter module. In this module, they are delayed by another three clock cycles before being finally aligned with the RGB data and output to the LCD in real time.

[0094] In summary, the purpose of this invention is to provide an image processing system for fluorescence endoscopy that extracts ICG-stained normal tissue by distinguishing the differences in fluorescence staining intensity, texture features, and shape features between normal and diseased tissues. This improves the accuracy of lesion areas under ICG excretion disorders and solves the problem of inaccurate judgment of lesion areas by doctors due to ICG molecule excretion disorders.

[0095] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image processing system for a fluorescence endoscope, characterized in that, Its framework includes a control layer, a device layer, and a database layer. The device layer includes a camera group, a display, and an illumination source. The camera group is used to acquire visible light images and fluorescence images. The illumination source includes a white light source, a fluorescence excitation source, and a spectral filtering component. The database layer is used to store the data acquired by the system. The control layer includes a CPU, an image processing module, a computing module, and an image output module. The CPU is used to manage and control the operation of the system. The image processing module is used to fuse visible light images and fluorescence images to generate a visible light fused image in which only lesion tissue emits fluorescence. The image output module is used to connect the host to the display and output images in real time.

2. The image processing system for a fluorescence endoscope according to claim 1, characterized in that, The beam splitting and filtering component includes a dichroic mirror, a first filter, and a second filter. The dichroic mirror splits the incident light into a visible light branch and a fluorescence branch. The first filter allows only visible light in the wavelength range of 400nm to 700nm to pass through, filtering out excitation light and fluorescence. The second filter allows only fluorescence in the wavelength range of 810nm to 850nm to pass through, filtering out visible light and excitation light.

3. The image processing system for a fluorescence endoscope according to claim 1, characterized in that, The image processing module includes an image denoising unit, which is used to denoise the visible light images and fluorescence images acquired by the camera group.

4. The image processing system for a fluorescence endoscope according to claim 1, characterized in that, The calculation module includes a threshold calculation unit and a feature extraction calculation unit. The threshold calculation unit calculates the specific location of the lesion tissue in the image by setting a threshold, and the feature extraction calculation unit extracts the location of normal tissue stained by ICG in the image by feature calculation and filters the fluorescence in the normal tissue in the image.

5. An image processing system for a fluorescence endoscope according to claim 4, characterized in that, The calculation module includes a calculation strategy for extracting lesions stained with ICG fluorescence from images. The calculation strategy for extracting lesions stained with ICG fluorescence from images includes the following specific steps: S101. Calculate the fluorescence intensity thresholds of diseased and normal tissues based on ICG staining fluorescence intensity. S102. Calculate the different texture and shape features of normal tissue and diseased tissue, and extract the diseased tissue stained with ICG fluorescence in the image.

6. The image processing system for a fluorescence endoscope according to claim 5, characterized in that, The threshold calculation unit includes a threshold calculation strategy for ICG fluorescently stained lesions in the image, which is as follows: S201. Set the mean pixel value of all pixels in the fluorescence image to be... Let the segmentation threshold be... Used to filter pixels with values ​​smaller than 1 in a fluorescence image. The pixels are segmented into fluorescent background images. In the fluorescence image, the pixel value is greater than or equal to The pixels are segmented into fluorescent target images. The fluorescent background image The mean pixel value of all pixels in the image is Fluorescent target image The mean pixel value of all pixels in the image is The pixels in the fluorescence image are divided into The probabilities are respectively , ; S202, Based on the fluorescence background image The average pixel value of all pixels in Fluorescent target images The average pixel value of all pixels in And the pixels in the fluorescence image are divided into probability , Calculate the inter-class variance of gray values ​​in fluorescence images. ; S203, when When the fluorescence image shows a pixel, it represents a lesion with high ICG staining intensity. When the fluorescence image is in the specified state, the pixels represent normal tissue with low ICG staining intensity or no ICG staining.

7. An image processing system for a fluorescence endoscope according to claim 6, characterized in that, The feature extraction calculation unit includes the extraction of texture and shape features of the diseased tissue, and its calculation strategy is as follows: S301. Calculation of texture features extracted from ICG fluorescent staining, wherein the extraction methods for texture features include: Using any pixel in the fluorescence image as a reference point, record the brightness value of the reference point; Within a preset direction and a preset distance from the reference point, multiple similar points corresponding to the reference point are obtained by traversing through corner point features, and the brightness values ​​of the multiple similar points are recorded. By statistically analyzing the frequencies corresponding to different brightness combinations among all similar points and reference points, the distribution pattern of neighborhood brightness in the fluorescence image can be obtained. The texture energy feature value of the fluorescence image is calculated based on the distribution pattern, and the average energy value is calculated based on the texture energy feature value to obtain the texture feature. S302. Extraction and calculation of shape features of ICG fluorescence staining: Binarize the lesion tissue with high ICG staining intensity in the fluorescence image calculated by the threshold calculation unit to obtain a binary matrix image; convert the binary matrix image into a matrix representation, and take the geometric center as the starting point; calculate the discretized geometric distribution of the starting point in the horizontal direction to obtain the variance index of the binary matrix image in the horizontal axis direction; calculate the average horizontal axis variance based on the variance index to obtain the shape features; S303. When the texture energy of an image region in the fluorescence image is greater than or equal to the average energy value of the fluorescence image, and the corresponding horizontal axis variance is greater than or equal to the average horizontal axis variance, the corresponding image region is determined to belong to the lesion tissue region. The determined lesion tissue region is compared with the fluorescence target image obtained by the threshold calculation unit, and the normal tissue region is obtained by difference. The normal tissue region is removed from the fluorescence image to obtain a fluorescence image containing only the lesion tissue.

8. The image processing system for a fluorescence endoscope according to claim 7, characterized in that, The specific steps for removing normal tissue areas from fluorescence images are as follows: S401. Let the set of coordinates of all pixels in the lesion tissue region of the image extracted by the feature extraction calculation unit be... The set of coordinates of all pixels in the fluorescent target image is The logical judgment function determines whether the pixel coordinates of the lesion tissue region in the image extracted by the feature extraction calculation unit are the same as the pixel coordinates of the fluorescent target image. S402. When the logical judgment function is true, that is, the pixel coordinates of the lesion tissue region extracted by the feature extraction calculation unit are the same as the pixel coordinates in the fluorescent target image, the gray value of the pixel coordinates is set to 255; when the logical judgment function is false, that is, the pixel coordinates of the lesion tissue region extracted by the feature extraction calculation unit are different from the pixel coordinates in the fluorescent target image, the gray value of the pixel coordinates is set to 0, and a fluorescent image of only the lesion tissue is obtained.

9. An image processing system for a fluorescence endoscope according to claim 8, characterized in that, The image processing module further includes an image fusion unit, which is used to calculate the fusion of the visible light image acquired by the camera group and the fluorescence image of only the lesion tissue extracted by the feature extraction calculation unit to form a visible light fused image in which only the lesion tissue is labeled with ICG fluorescence.

10. An image processing system for a fluorescence endoscope according to claim 9, characterized in that, The image fusion unit decomposes the visible light image and the fluorescence image of the lesion tissue extracted by the feature extraction calculation unit into a base layer image and a detail layer image, and then fuses the base layer image and the detail layer image to obtain a visible light fused image of the lesion tissue labeled with ICG fluorescence. The specific fusion steps are as follows: S501. Let the visible light image and the fluorescence image be represented as follows: , The base image of a visible light image is The base image of the fluorescence image is Fluorescence images , the base layer image of fluorescence images The visible light detail layer image is obtained by subtracting the visible light image from the visible light base layer image. The fluorescence detail layer image is obtained by subtracting the fluorescence image from the fluorescence base layer image. ; S502. The base layer images of the visible light image and the fluorescence image are fused using Gaussian filtering to obtain a base layer weight map. The fused base layer image is obtained by weighting the base layer image of the green channel of the visible light image and the fluorescence image through the base layer weight map. S503. Perform mean filtering and median filtering on the visible light image and the fluorescence image to obtain the visual saliency features of the visible light image and the fluorescence image. Enhance the detail layer image to obtain the detail layer weight map of the visible light image and the detail layer weight map of the fluorescence image. Obtain the fused detail layer image by weighted averaging. S504. A fused image of lesion tissue labeled with ICG fluorescence is obtained through the fused base layer image and the fused detail layer image.

Images