A method for fusion of endoscope fluorescence and visible light images

Abstract

The present invention relates to the field of endoscope imaging technology, and discloses a method for fusion of fluorescence visible light images in an endoscope scene, the method comprising the following steps: obtaining a registered visible light image and a fluorescence image; extracting a fluorescence area by threshold segmentation; calculating a correction coefficient; generating a normalized corrected fluorescence image; calculating a fusion coefficient; generating a fused image. The present invention can clearly and strikingly fuse fluorescence information into a visible light image, thereby facilitating doctors to accurately determine the location of the lesion by combining visible light and fluorescence information.

Description

Technical Field

[0001] The invention belongs to the field of medical image processing, and in particular relates to a method for fusing endoscopic fluorescence and visible light images. Background Art

[0002] In the current medical system, fluorescent endoscopy imaging plays an increasingly important role. Fluorescent dyes such as ICG (indocyanine green) can be injected and automatically gather at lesions such as tumors. During the use of fluorescent endoscopy, visible light images and fluorescent images are obtained separately. Doctors often need to combine the information of the two to make a more accurate diagnosis. Therefore, it is particularly important to fuse visible light images and fluorescent images.

[0003] The current mainstream fusion algorithms all present the information in the fluorescence image in green in the visible light image, but the implementation methods are different. The simplest method is to directly superimpose the fluorescence image signal value on the green channel of the visible light image, but the visual effect is poor, and in some cases the fused image cannot reflect the strength of the fluorescence signal. In addition, the fluorescence image signal value is often lower because the fluorescence intensity is weaker than that of visible light, and it will cause certain changes due to different observation angles and distances, so it cannot fully reflect the actual fluorescence intensity. Summary of the invention

[0004] The purpose of the present invention is to provide a method for fusing endoscopic fluorescence and visible light images, which can make the fluorescence image clearly visible in the fused image, with a better overall appearance, and can eliminate the influence of uneven brightness caused by different observation angles and distances, thereby better reflecting the relative size of the actual fluorescence intensity.

[0005] In order to solve the above technical problems, the present invention proposes the following technical solutions:

[0006] S1: Obtain a visible light image VL = (R1, G1, B1) and a fluorescence image FL registered with the visible light image, wherein the two images have the same bit depth. If the bit depths are different, first convert the high bit depth image to the color depth of the low bit depth image;

[0007] S2: Use the threshold calculation method to calculate the gray threshold A1 of the fluorescence image FL, and then obtain the binary image BW according to formula (1), using 1 and 0 to represent the fluorescence area and non-fluorescence area respectively:

[0008]

[0009] In formula (1), FL represents the gray value of each pixel in the fluorescence image;

[0010] S3: Calculate the average gray value of the fluorescent area in the fluorescent image FL The average red channel signal of the fluorescent area in the visible light image VL

[0011] S4: Calculate the normalized corrected fluorescence image FL_nor of each image point according to formula (2):

[0012]

[0013] In formula (2), R1 is the red channel signal value of each pixel in the visible light image;

[0014] S5: Calculate the fusion coefficients x and y according to formulas (3) and (4), where x is a fixed coefficient and y is a coefficient that changes with the changes of R1, B1, and FL_nor at each point:

[0015] x=(kr+kb)*d (3)

[0016]

[0017] Among them, B1 is the blue channel signal value of each pixel in the visible light image, kr, kg, kb are the brightness weighting factors in the brightness calculation formula Y = kr*R+kg*G+kb*B, and d is the superposition coefficient, which represents the degree of fusion of the fluorescence image and has a value range of [0,1].

[0018] S6: According to formula (5), the visible light image VL = (R1, G1, B1) is fused with the normalized corrected fluorescence image FL_nor to obtain the final fused image FUS = (R2, G2, B2):

[0019]

[0020] In the present invention, the threshold calculation method in S2 may be determined manually, or an adaptive threshold method such as the maximum inter-class variance method may be used to calculate the threshold.

[0021] In the present invention, the brightness weighting factors kr, kg, and kb in S5 are generally set according to kr=0.299, kg=0.587, and kb=0.114 defined in ITU-R BT.601, and can also be customized according to requirements, but generally kr+kg+kb=1 should be ensured.

[0022] The beneficial effects of the present invention are:

[0023] The present invention normalizes the fluorescence image based on the signal average values ​​of the red channels of the fluorescence image and the visible light image in the fluorescence area respectively, and designs a specific fusion coefficient and fusion method, which can effectively, clearly and strikingly fuse the fluorescence information into the visible light image, and is not affected by the shooting angle or distance. Compared with the existing methods, this can greatly help doctors to accurately determine the location of the lesion by combining visible light and fluorescence information. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the method for fusing endoscope fluorescence and visible light images proposed in the present invention.

[0025] Figure 2A It is an illustration of the R channel of the visible light image acquired in a specific embodiment of the present invention.

[0026] Figure 2B It is an illustration of the G channel of the visible light image acquired in a specific embodiment of the present invention.

[0027] Figure 2C It is an illustration of the B channel of the visible light image acquired in a specific embodiment of the present invention.

[0028] Figure 2D is an illustration of a registered fluorescence image acquired in a specific embodiment of the present invention.

[0029] Figure 3A It is an illustration of the fluorescent area in the fluorescent image according to a specific embodiment of the present invention.

[0030] Figure 3B It is an illustration of the R channel of the fluorescent area in the visible light image in a specific implementation manner of the present invention.

[0031] Figure 3C It is an illustration of a normalized and corrected fluorescence image in a specific embodiment of the present invention.

[0032] Figure 3D It is an illustration of the fluorescence image after correction using the correction method of the present invention and other correction methods (not using the average value).

[0033] Figure 4A It is an illustration of the R channel of the fused image in a specific implementation manner of the present invention.

[0034] Figure 4B It is an illustration of the G channel of the fused image in a specific embodiment of the present invention.

[0035] Figure 4C It is an illustration of the B channel of the fused image in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0036] In order to achieve clearer imaging, the present application relates to a method for fusing endoscopic fluorescence and visible light images, which can clearly and strikingly superimpose and fuse fluorescence information on visible light images, is not affected by the shooting distance and angle, and has efficient processing and real-time display. The technical solution of the present invention is further explained below in conjunction with the accompanying drawings and examples.

[0037] Combination Figure 1 4 , the process of the endoscopic fluorescence and visible light image fusion method is specifically described.

[0038] S1: Obtain visible light image VL = (R1, G1, B1) Figure 2A to Figure 2C ) and the fluorescence image FL after registration with the visible light image ( Figure 2D ), where the two images have the same bit depth. If the bit depths are different, the high bit depth image is first converted to the color depth of the low bit depth image, where the two images have the same bit depth.

[0039] Among them, the visible light image VL is a color image, and the fluorescence image FL is a grayscale image.

[0040] Since fluorescent agents are not only present in the lesion site, but also distributed in the normal site to a certain extent, they can also emit fluorescence after being irradiated with excitation light. Figure 2D ), the normal part will be the low brightness area ( Figure 2D Therefore, the fluorescence image includes the area with relatively high brightness due to the fluorescence emitted by the lesion ( Figure 2D D-1), the normal part becomes a relatively high brightness area because it is closer to the endoscope ( Figure 2D D-2) and low brightness areas representing normal parts ( Figure 2D D-3).

[0041] Similarly, the visible light image includes the normal part which has a relatively high brightness due to its close distance to the endoscope (such as Figure 2A A-1), and low-brightness areas as background (such as Figure 2A A-2).

[0042] S2: Use the threshold calculation method to calculate the threshold A1 on the fluorescence image FL, and then obtain the binary image BW according to formula (1), using 1 and 0 to represent the fluorescence area and the non-fluorescence area respectively.

[0043]

[0044] Extract the fluorescent area in the fluorescent image FL ( Figure 3A A-1, A-2) and the red channel of the fluorescent area in the visible light image VL ( Figure 3B B-1, B-2). Among them, Figure 3AA-1 is a relatively bright area due to the fluorescence emitted by the lesion. Figure 3B B-1 is the lesion that cannot be identified in visible light. Figure 3A A-2 with Figure 3B B-2 is a normal part that is close to the endoscope and has a relatively high brightness.

[0045] S3: Calculate the average gray value of the fluorescent area in the fluorescent image FL The average value of the red channel of the fluorescent area in the visible light image VL

[0046] Since fluorescence is mostly near-infrared light, which is closer to red light in various spectral characteristics, the red channel of visible light is selected as the reference for normalization correction.

[0047] S4: Calculate the normalized corrected fluorescence image FL_nor according to formula (2) Figure 3C ).

[0048]

[0049] Among them, the fluorescence image FL has a relatively high brightness area due to the fluorescence emitted by the lesion ( Figure 2D D-1) After the above processing, the signal value will increase ( Figure 3C C-1), while the normal part in the fluorescent image FL becomes a relatively high brightness area because it is closer to the endoscope ( Figure 2D D-2) After the above processing, the signal value will decrease ( Figure 3C C-2). Therefore, after the above processing, the lesion in the fluorescence image is highlighted, while the normal part with relatively high brightness due to its close distance to the endoscope is not easily diagnosed as the lesion due to the reduced signal value, thereby reducing the influence of the distance between the endoscope and the photographed part on the doctor. If the normalization method not based on the average value is used to correct the fluorescence image, as shown in the following formula, the effect is as follows: Figure 3D , it can be seen that because Fluorescence correction image gray value ( Figure 3D D-1 and D-2) are compared with the grayscale value of the corrected image obtained by the method of the present invention ( Figure 3C The C-1 and C-2 of In the case of , the fluorescence signal will be too strong, which will also affect the fusion effect. On the one hand, the signal value may overflow and lose the texture information of the light and dark changes in the fluorescence image. On the other hand, too high a signal value will cause the RGB value of the visible light to change too much during image fusion, resulting in a certain degree of loss of texture information in the visible light image. If formula (2) is used, this problem can be solved well, so that the fluorescence signal intensity is at a more reasonable level.

[0050]

[0051] S5: Calculate the fusion coefficients x and y according to formulas (3) and (4), where x is a fixed coefficient and y is a coefficient that changes with the changes of R1, B1, and FL_nor at each point:

[0052] x=(kr+kb)*d (3)

[0053]

[0054] Among them, kr, kg, and kb are the brightness weighting factors in the brightness calculation formula Y = kr*R + kg*G + kb*B. Here, according to the definition in ITU-R BT.601, kr = 0.299, kg = 0.587, and kb = 0.114. d is the superposition coefficient, which represents the degree of fusion of the fluorescence image and has a value range of [0,1]. When it is 0, the fused image is a visible light image, and when it is 1, the fluorescence information in the fused image is the most obvious. Here, 1 is taken to make the fusion effect most obvious.

[0055] S6: According to formula (5), the visible light image VL = (R1, G1, B1) is fused with the normalized corrected fluorescence image FL_nor to obtain the final fused image FUS = (R2, G2, B2). Figure 4A , Figure 4B , Figure 4C They are the fused image RGB channel maps respectively.

[0056]

[0057] According to the definition of ITU-R BT.601, kr = 0.299, kg = 0.587, kb = 0.114, then x = (kr + kb) * d = 0.413, which will not cause G channel overflow in most cases, while reducing the signal values ​​of the R and B channels. On the one hand, the fluorescent area can be made closer to green, thereby improving the contrast between the high fluorescence area and the low fluorescence area. On the other hand, the brightness of the fused image and the visible light image remains unchanged, thereby preserving the texture information in the visible light image to a large extent (the final fusion effect can be clearly seen in the color display that the fluorescent area of ​​the lesion area presents a clear and obvious green, forming a striking contrast with other areas (normal areas including normal areas closer to the shooting lens)).

Claims

1. A method for fusing endoscope fluorescence and visible light images, characterized in that: The steps include: S1: Obtain a visible light image VL = (R1, G1, B1) and a fluorescence image FL registered with the visible light image, wherein the two images have the same bit depth. If the bit depths are different, first convert the high bit depth image to the color depth of the low bit depth image; S2: Use the threshold calculation method to calculate the gray threshold A1 of the fluorescence image FL, and then obtain the binary image BW according to formula (1), using 1 and 0 to represent the fluorescence area and non-fluorescence area respectively: S3: Calculate the average gray value of the fluorescent area in the fluorescent image FL The average red channel signal of the fluorescent area in the visible light image VL S4: Calculate the normalized corrected fluorescence image FL_nor according to formula (2): S5: Calculate the fusion coefficients x and y according to formulas (3) and (4), where x is a fixed coefficient and y is a coefficient that changes with the changes of R1, B1, and FL_nor at each point: x=(kr+kb)*d (3) Among them, kr, kg, and kb are the brightness weighting factors in the brightness calculation formula Y=kr*R+kg*G+kb*B, and d is the superposition coefficient, which represents the degree of fusion of the fluorescence image and has a value range of [0,1]; S6: According to formula (5), the visible light image VL = (R1, G1, B1) is fused with the normalized corrected fluorescence image FL_nor to obtain the final fused image FUS = (R2, G2, B2):

2. The method for fusion of endoscope fluorescence and visible light images according to claim 1, characterized in that: The threshold calculation method used in S2 may be determined manually, or an adaptive threshold method such as the maximum inter-class variance method may be used to calculate the threshold.

3. The method for fusion of endoscope fluorescence and visible light images according to claim 1, characterized in that: The brightness weighting factors Kr, kg, and kb in S5 can be set according to kr=0.299, kg=0.587, and kb=0.114 defined in ITU-R BT.601, or can be customized according to needs, but kr+kg+kb=1 should be ensured.

4. The method for fusion of endoscope fluorescence and visible light images according to claim 1, characterized in that: The method is provided with a d adjustment unit, and the user adjusts d according to the required display effect to adjust the degree of fusion of the fluorescent image into the visible light image. If d is 0, it is displayed as a visible light image, and if d is 1, the fluorescent image is completely fused with the visible light image.

Images