Endoscope fluorescence image extraction and fusion method

Abstract

The application relates to the technical field of endoscope imaging, and discloses a method for extracting and fusing a fluorescent image under an endoscope scene, which comprises the following steps: adjusting the brightness of a light source, acquiring a visible light image and a double-waveband image; according to a brightness response test result, realizing extraction of a fluorescent image in a brightness domain; according to the intensity of the fluorescent image, determining a pseudo-color range and a mapping relationship, and realizing pseudo-color image fusion. The application can extract a fluorescent image from a double-waveband image and a visible light image, and clearly and prominently fuse the fluorescent information in pseudo-color in the visible light image, can reflect the absolute intensity and the relative intensity of the fluorescent information, and thus facilitates a doctor to accurately judge a lesion position by combining the visible light and the fluorescent information.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image processing, and particularly relates to an endoscope fluorescence image extraction and fusion method. BACKGROUND

[0002] In the current medical system, fluorescence endoscopy imaging plays an increasingly important role. Fluorescent dyes such as ICG (indocyanine green) can automatically gather in tumor and other lesion sites through injection. The common fluorescence endoscope currently uses a multi-sensor scheme, that is, two or four sensors are used to correspond to visible light / fluorescence or R / G / B / fluorescence. The visible light and fluorescence are irradiated onto different sensors after passing through a dichroic prism to complete imaging. This scheme can obtain visible light images and fluorescence images at the same time, without misalignment, and the image quality is theoretically good. However, it has a large volume and weight, requires high-precision assembly, and has a greater heat dissipation pressure. The single-sensor scheme can better solve the above shortcomings. This scheme uses a special filter that can pass white light and fluorescence, and blocks the infrared light emitted by the infrared light source. When white light is illuminated, a visible light image is obtained. When infrared light is illuminated, a fluorescence image is obtained. When white light and infrared light are illuminated at the same time, the obtained image contains both visible light and fluorescence information, so it can be called a dual-band image. However, this scheme also faces the difficulty of obtaining dual-band signals at the same time.

[0003] A common solution is to alternate white light and infrared light illumination while controlling the sensor imaging to obtain fluorescence images and visible light images. However, this method will make the frame rate of visible light images and fluorescence images only half of the total frame rate of the sensor, affecting the viewing experience. The method in Stryker patent US-11025867-B2 is to keep the infrared light source on and illuminate the white light source at intervals. Because part of the frames only contain fluorescence information and do not contain visible light information, the final visible light image frame rate is 2 / 3 of the total frame rate. In order to obtain fluorescence images and highlight the fluorescence signal, the obtained visible light images and dual-band images need to be subjected to fluorescence image extraction.

[0004] After obtaining the fluorescence image, it is usually necessary to fuse the fluorescence image with the visible light image. A common way is to fuse the fluorescence information in the visible light image in the form of pseudo-color, so that the relative intensity of the fluorescence can be directly observed. However, red and yellow are close to the color of human tissues, and confusion is easy to occur, so the color range of the pseudo-color needs to be limited. In addition, in addition to presenting the relative intensity of the fluorescence in the same time image, if the pseudo-color fusion image can also reflect the absolute intensity of the fluorescence, it will help doctors to make better diagnoses or complete surgeries. SUMMARY

[0005] The present application aims to provide an endoscope fluorescence image extraction method and fusion method, which extracts fluorescence images from dual-band images and visible light images, presents fluorescence information in the form of pseudo-color in the fusion image, and can distinguish the relative intensity of fluorescence in the image according to the color difference, and can determine the absolute intensity of the overall fluorescence to some extent.

[0006] To solve the above technical problems, the present application proposes the following technical solutions:

[0007] S1: Adjust the brightness of the white light source and the infrared light source, control the white light source to be always on, control the infrared light source to be illuminated at intervals, and synchronously control the camera exposure so that the visible light image VL=(R1, G1, B1) and the dual-band image DB=(R2, G2, B2) are alternately acquired at the same exposure time and gain, wherein the bit depth of the two images is the same;

[0008] S2: Calculate the fluorescence image RGB value in the brightness domain based on the visible light image and the dual-band image through brightness response testing, and calculate the fluorescence grayscale image according to the sensor spectral response and the white balance parameter, and perform brightness enhancement on the visible light image and the fluorescence grayscale image;

[0009] S3: Determine the pseudo-color range according to the absolute intensity of the fluorescence image, determine the mapping relationship according to the relative intensity of the fluorescence image, calculate the mapping image, and finally realize pseudo-color image fusion by weighted combination of the mapping image and the visible light image.

[0010] In the present application, the adjustment of the light source brightness in S1 should respectively make the brightness of the visible light image and the fluorescence image both lower than the normal perception at the same exposure time and gain, such as controlling the brightness of the two images to be a% and b% (a<100, b<100, the specific values can be set as needed) of the normal exposure state. In the alternating imaging process of the visible light image and the dual-band image, the exposure time and the gain should be determined only by reference to the dual-band image.

[0011] In the present application, the brightness response test in S2 should be: under the conditions of exposure time x, gain y, and default white balance, imaging the light source box with adjustable brightness, recording multiple groups of light source box brightness and R, G, B three channel averages in the light source box area of the image, covering the complete dynamic range of the camera, and fitting to obtain three brightness response curves representing the brightness-channel value relationship. In order to speed up the subsequent calculation, the brightness response table can be obtained by first calculating the brightness value corresponding to all values of R, G, and B three channels.

[0012] In the present application, the luminance response curve in S2 is obtained under certain exposure time and gain conditions. In actual use, the values of each channel in the curve should be linearly related to the exposure time and gain, and change with the change of the exposure time and gain. Moreover, the values of each channel in the curve should also be related to the white balance parameter, and change with the change of the white balance parameter, so that the channel intensity values obtained by imaging the same intensity light before and after white balance remain consistent.

[0013] In the present application, the method for calculating the fluorescent RGB image in the luminance domain in S2 should be: according to the R, G, B three-channel values of each point in the visible light image VL=(R1, G1, B1) and the dual-band image DB=(R2, G2, B2), the luminance values of each point corresponding to the three channels are calculated through the luminance response curve IVL=(IR1, IG1, IB1), IDB=(IR2, IG2, IB2). The luminance of the dual-band image IDB is subtracted from the luminance of the visible light image IVL to obtain the corresponding luminance value of the fluorescent image IFL=(IR3, IG3, IB3), which is substituted into the luminance response curve to obtain the RGB value of each point of the fluorescent image FL=(R3, G3, B3).

[0014] In the present application, the calculation method of the fluorescent gray scale image in S2 is shown in formula (1).

[0015] FL'=kr*R3+kg*G3+kb*B3 (1)

[0016] Wherein, the luminance weighting factors kr, kg, kb should be set according to the spectral response of the used sensor to the fluorescent band, and can also be set according to the requirements. However, after the white balance of the camera system, the luminance weighting factors should be changed correspondingly, so that the fluorescent gray scale image intensity values obtained by imaging the same intensity fluorescent before and after white balance remain consistent.

[0017] In the present application, the luminance enhancement of the visible light image and the fluorescent gray scale image in S2 should be combined with the image luminance percentage in S1, such as dividing the visible light image and the fluorescent gray scale image by a% and b% respectively to obtain the brightened visible light image and the fluorescent gray scale image.

[0018] In the present application, in S3, the actual hue range of the pseudo color needs to set the maximum hue range of the fluorescent pseudo color [minH1, maxH1] first, and then set the threshold A1 and the threshold A3, the average of which is A2. The threshold value is calculated by using the threshold value calculation method to calculate the threshold value B of the fluorescent gray scale image FL', and the maximum value maxFL of the fluorescent gray scale image is calculated. The actual hue range [minH2, maxH2] is determined according to the maximum value maxFL of the fluorescent gray scale image and the threshold value B, as shown in formula (2) and (3).

[0019]

[0020]

[0021] The threshold value B can be set artificially or calculated by using a self-adaptive threshold method such as the maximum inter-class variance method. In order to prevent the maximum value from being too high due to the high brightness of individual points in the image, the maximum brightness of the fluorescent grayscale image can be replaced by the brightness value of the pixel points whose brightness reaches the top c% (set artificially) among the pixel points in the fluorescent grayscale image that exceed the threshold value, and the pixel points in the fluorescent grayscale image that exceed the value are all changed to the value.

[0022] In the present application, the mapping relationship of the pseudo color in S3 is that the maximum value maxFL in the fluorescent grayscale image corresponds to the actual minimum hue minH2, and the threshold value B and below correspond to the actual maximum hue maxH2. The mapping method is to convert the visible light image VL to the HSV space, perform transformation on the H channel as shown in equation (4), and then convert the transformed image back to the RGB space to obtain the mapping image VL'.

[0023]

[0024] In the present application, the weighted combination method in S3 is to calculate the weights of the mapping image VL' and the visible light image VL according to the fluorescent intensity, and to obtain the fusion image FUS. In the fusion image FUS, the region with the fluorescent intensity higher than B uses the value of the mapping image VL', the region with the fluorescent intensity lower than B uses the value of the visible light image VL, and the region with the fluorescent intensity between B and B uses the weighted value of the visible light image VL and the mapping image VL'.

[0025]

[0026] FUS = VL * (1-BW) + VL' * BW (6)

[0027] The present application has the following beneficial effects:

[0028] ​​The method of the present application keeps the white light source constant, the infrared light source interval illuminates, and controls the sensor imaging to obtain the visible light image and the dual-band image; for the visible light picture and the dual-band picture, each frame contains visible light information, so that the frame rate of the visible light picture can be doubled by calculation, and the total frame rate of the sensor is the same, compared with the method of keeping the red light source constant and the white light source interval illuminating, the visible light frame rate can be greatly improved. And the method of the present application can extract the fluorescent image from the dual-band image and the visible light image, and clearly and prominently fuse the fluorescent information in pseudo color in the visible light image, which can simultaneously reflect the absolute intensity and the relative intensity of the fluorescent information, so as to facilitate the doctor to accurately judge the lesion site combined with the visible light and the fluorescent information. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flow chart of the endoscope fluorescent image extraction and fusion method of the present application.

[0030] Figure 2 is a flow chart of image shooting, processing and display in the specific embodiment of the present application.

[0031] Figure 3 is a luminance response characteristic diagram of the camera system obtained by testing in the specific embodiment of the present application.

[0032] Figure 4A is a B channel diagram of the visible light image obtained in the specific embodiment of the present application.

[0033] Figure 4B is a B channel diagram of the dual-band image obtained in the specific embodiment of the present application.

[0034] Figure 4C is a B channel diagram of the fluorescent image calculated in the specific embodiment of the present application.

[0035] Figure 4D is a fluorescent gray scale image diagram calculated in the specific embodiment of the present application.

[0036] Figure 5A is a fusion image gray scale value diagram when the fluorescent gray scale image is low in luminance in the specific embodiment of the present application.

[0037] Figure 5B is a fusion image gray scale value diagram when the fluorescent gray scale image is medium in luminance in the specific embodiment of the present application.

[0038] Figure 5C is a fusion image gray scale value diagram when the fluorescent gray scale image is high in luminance in the specific embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to better clarify the imaging, the application relates to an endoscope fluorescence image extraction method and a fusion display method. The technical solutions of the application are further described in detail below in combination with the drawings and specific embodiments.

[0040] As Figure 1 To Fig. 5, the process of the extraction and fusion display of the endoscope fluorescence image of the embodiment of the application includes the following:

[0041] S1: First, adjust the brightness of the white light source and the infrared light source, then control the white light source to be always on, control the infrared light source to be illuminated at intervals, and synchronously control the camera exposure, so that the camera alternately acquires the visible light image VL=(R1, G1, B1) and the dual-band image DB=(R2, G2, B2) under the same exposure time and gain, wherein the bit depth of the two images is the same, as shown in the camera shooting sequence in Figure 4A Figure 4B Figure 2

[0042] In the embodiment, the sensor B channel has a strong fluorescence response, so the difference between the dual-band image and the visible light image is more obvious in the B channel, and thus the B channel is used for display. Figure 4B In the embodiment, the fluorescence region is at B-1.

[0043] In the embodiment, the adjustment of the brightness of the light source should make the brightness of the visible light image and the fluorescence image both be lower than the normal perception under the same exposure time and gain, for example, the brightness of the two images is controlled to be a% and b% (a<100, b<100) of the normal exposure state. In the alternating imaging process of the visible light image and the dual-band image, the exposure time and the gain should be determined only by reference to the dual-band image. In the embodiment, a=66.67 and b=50.

[0044] S2: Convert the calculation of the fluorescence RGB image to the luminance domain through a luminance response test. According to the spectral response of the sensor and the white balance parameter, a fluorescence grayscale image is calculated, and the luminance of the visible light image and the fluorescence grayscale image is improved.

[0045] In the embodiment, the luminance response test should be: under the conditions of the exposure time x, the gain y, and the default white balance, image the light source box with adjustable brightness, record the R, G, and B three-channel average values of the light source box region in the image corresponding to multiple groups of light source box brightness, cover the complete dynamic range of the camera, and fit three luminance response curves representing the luminance-channel value relationship. In order to speed up the subsequent calculation, the luminance values corresponding to all values of the R, G, and B three channels can be calculated first to obtain a luminance response table; in the embodiment, under a certain exposure time and gain, the luminance response curves of the RGB three channels are the same, as shown in Figure 3 .

[0046] ​​​The method for calculating the fluorescent RGB image in the luminance domain should be: according to the R, G, B three channel values of each point in the visible light image VL=(R1, G1, B1) and the dual-band image DB=(R2, G2, B2), the luminance values IVL=(IR1, IG1, IB1) and IDB=(IR2, IG2, IB2) corresponding to the three channels of each point are calculated through the luminance response curve. The difference between the luminance IDB of the dual-band image and the luminance IVL of the visible light image is obtained to obtain the corresponding luminance value IFL=(IR3, IG3, IB3) of the fluorescent image, which is substituted into the luminance response curve to obtain the RGB value FL=(R3, G3, B3) of each point of the fluorescent image, as shown in Figure 4C , wherein the fluorescent region is at C-1.

[0047] In the present application, the calculation method of the fluorescent gray scale image in S2 is shown in formula (1);

[0048] FL'=kr*R3+kg*G3+kb*B3 (1)

[0049] The luminance weighting factors kr, kg, and kb should be set according to the spectral response of the used sensor to the fluorescent band, and can also be set as needed. However, after the white balance of the camera system, the luminance weighting factors should be changed correspondingly, so that the intensity values of the fluorescent gray scale image obtained by imaging the same intensity of fluorescent light before and after white balance remain consistent. In the present embodiment, kr=1.01, kg=1.11, and kb=0.41. The calculated fluorescent gray scale image is shown in Figure 4D , wherein the fluorescent region is at D-1. During shooting, the adjacent dual-band image (i) and the visible light image (i) are extracted in real time to obtain the fluorescent image (i), thereby obtaining a sequence of fluorescent gray scale images, as shown in the fluorescent image extraction sequence in Figure 2 . After shooting the dual-band image (i), the RGB channel values of the dual-band image (i) and the fluorescent color image (i-1) calculated in the last step are converted into luminance values, and the difference between the two luminance values is converted back into RGB values to calculate the calculated visible light image VL_cal(i).

[0050] According to the settings in S1, the visible light image (including the actually shot visible light image and the calculated visible light image) and the fluorescent gray scale image are divided by 66.67% and 50% respectively, to realize the luminance enhancement of the visible light image and the fluorescent image.

[0051] In the display window of the visible light image, the luminance-enhanced visible light image and the calculated visible light image are displayed alternately, as shown in the visible light image display sequence in Figure 2 , wherein the calculated visible light image (i) is represented by the dual-band image (i+1)-fluorescent image (i). Thus, high frame rate display of the visible light image can be realized.

[0052] S3: According to the brightness of the fluorescence gray image, the actual hue range and mapping relationship of the pseudo-color are determined and the mapping image is calculated. Finally, the mapping image and the visible light image are weighted and combined to realize the pseudo-color image fusion.

[0053] First, the maximum hue range [minH1, maxH1] of the fluorescence pseudo-color is set, and then the threshold A1 and the threshold A3 are artificially set, and the average of the two is A2. The threshold value B is calculated by using the threshold calculation method to calculate the fluorescence gray image FL', and the maximum value maxFL of the fluorescence gray image is calculated. According to the maximum value maxFL of the fluorescence gray image and the threshold value B, the actual hue range [minH2, maxH2] is determined, as shown in equations (2) and (3). In this embodiment, minH1=0.25, maxH1=0.75, A1=60, A2=100, and A3=140.

[0054]

[0055]

[0056] In the fluorescence gray image, the maximum value maxFL corresponds to the actual minimum hue minH2, and the threshold value B and below correspond to the actual maximum hue maxH2. The mapping method is: converting the visible light image VL to the HSV space, transforming the H channel as shown in equation (4), and then converting the transformed image back to the RGB space to obtain the mapping image VL'.

[0057]

[0058] The weighted combination method of the mapping image and the visible light image is to calculate the weight of the mapping image VL' and the visible light image VL according to the fluorescence intensity, and to calculate the fusion image FUS. In the fusion image FUS, the area with fluorescence intensity higher than B uses the value of the mapping image VL', the area with fluorescence intensity lower than uses the value of the visible light image VL, and the area between and B is weighted by the visible light image VL and the mapping image VL' as shown in equations (5) and (6).

[0059]

[0060] FUS=VL*(1-BW)+VL′*BW (6)

[0061] The fluorescence gray scale images obtained by using 0.5 times, 1 times and 2 times of the fluorescence intensity low, medium and high of the fluorescence gray scale image FL' obtained in S2 are respectively simulated, and the threshold B and the maximum value maxFL are respectively [27, 78], [55, 157] and [105, 313]. And the visible light image VL is fused and displayed according to the above method, and the corresponding actual hue range is respectively [0.50, 0.75], [0.25, 0.75] and [0.25, 0.50], so as to obtain the fusion image Figures 5A-5C . Figure 5A In the formula, A-1, A-2 and A-3 respectively correspond to cyan, blue and purple; Figure 5B In the formula, B-1, B-2, B-3, B-4 and B-5 respectively correspond to light green, green, cyan, blue and purple; Figure 5C In the formula, C-1, C-2 and C-3 respectively correspond to light green, green and cyan. Figures 5A-5C In the formula, the edges of the fluorescence region are all gradually transitioned. In the same image, the relative strength of the fluorescence at this moment can be distinguished according to the different hues, and in the process of moving the endoscope, the change of the hue can also judge the change of the absolute strength of the fluorescence, and can prevent the fusion result of the region with lower fluorescence intensity and certain change being consistent with the fusion result of the strong fluorescence region, so as to produce misjudgment.

Claims

1. A method for extracting and fusing endoscopic fluorescence images, characterized in that, Includes the following steps: S1: Adjust the brightness of the white light source and the infrared light source, and control the white light source to be constantly lit, while the infrared light source is illuminated intermittently. Simultaneously control the camera exposure so that it alternately acquires the visible light image VL=(R1,G1,B1) and the dual-band image DB=(R2,G2,B2) under the same exposure time and gain, where the two images have the same bit depth. S2: Through brightness response testing, based on visible light images and dual-band images, the RGB values ​​of the fluorescence image are calculated in the brightness domain. Furthermore, based on the sensor's spectral response and white balance parameters, a fluorescence grayscale image is calculated, and the brightness of both the visible light image and the fluorescence grayscale image is enhanced. The method for calculating the fluorescence RGB image in the brightness domain is as follows: Based on the R, G, and B channel values ​​of each point in the visible light image VL=(R1,G1,B1) and the dual-band image DB=(R2,G2,B2), the corresponding brightness values ​​IVL=(IR1,IG1,IB1) and IDB=(IR2,IG2,IB2) for each of the three channels are calculated using the brightness response curve. The brightness IDB of the dual-band image is subtracted from the brightness IVL of the visible light image to obtain the corresponding brightness value IFL=(IR3,IG3,IB3) of the fluorescence image. This value is then substituted into the brightness response curve to calculate the RGB value FL=(R3,G3,B3) for each point of the fluorescence image. The calculation method for fluorescence grayscale images is shown in equation (1): （1）， The brightness weighting factors kr, kg, and kb should be set according to the spectral response of the sensor used to the fluorescence band, or they can be customized as needed. However, after the camera system performs white balance, the brightness weighting factors should be changed accordingly so that the intensity values ​​of the fluorescence grayscale images obtained before and after white balance for the same intensity of fluorescence remain consistent. S3: Determine the pseudo-color range based on the absolute intensity of the fluorescence image, determine the mapping relationship based on the relative intensity of the fluorescence image, calculate the mapping image, and finally combine the mapping image with the visible light image in a weighted manner to achieve pseudo-color image fusion.

2. The method for extracting and fusing endoscopic fluorescence images according to claim 1, characterized in that, In S1, the brightness adjustment of the light source should ensure that, under the same exposure time and gain, the brightness of both the visible light image and the fluorescence image is lower than the normal visual perception. The brightness of the two images should be controlled to be a% and b% of the normal exposure state, respectively, where a < 100 and b < 100. During the alternating imaging process of the visible light image and the dual-band image, the exposure time and gain should be determined only with reference to the dual-band image.

3. The method for extracting and fusing endoscopic fluorescence images according to claim 1, characterized in that, The brightness response test in S2 should be as follows: Under the conditions of exposure time x, gain y, and default white balance, image an adjustable light source box, record multiple sets of light source box brightness and the average values ​​of the R, G, and B channels of the light source box area in the image, covering the complete dynamic range of the camera, and fit three brightness response curves representing the relationship between brightness and channel values.

4. The method for extracting and fusing endoscopic fluorescence images according to claim 1, characterized in that, The brightness response curve in S2 is obtained under certain exposure time and gain conditions. In actual use, the channel values ​​in the curve should have a linear relationship with the exposure time and gain, changing with the exposure time and gain; and The channel values ​​in the curve should also be related to the white balance parameters and change with the white balance parameters, so that the channel intensity values ​​obtained by imaging the same intensity light before and after white balance remain consistent.

5. The method for extracting and fusing endoscopic fluorescence images according to claim 1, characterized in that, In S2, the brightness enhancement of the visible light image and the fluorescent grayscale image should be combined with the image brightness percentage in S1. The visible light image and the fluorescent grayscale image should be divided by a% and b% respectively to obtain the brightened visible light image and the fluorescent grayscale image.

6. The method for extracting and fusing endoscopic fluorescence images according to claim 1, characterized in that, In S3, to determine the actual hue range of the pseudo-color, it is necessary to first set the maximum hue range of the fluorescent pseudo-color [minH1, maxH1], then manually set the thresholds A1 and A3, the average of which is A2. The threshold calculation method is used to calculate the threshold B of the fluorescent grayscale image FL', and the maximum value of the fluorescent grayscale image maxFL is calculated. The actual hue range [minH2, maxH2] is determined based on the maximum value of the fluorescent grayscale image maxFL and the threshold B, as shown in equations (2) and (3). （2）， （3）。 7. The method for extracting and fusing endoscopic fluorescence images according to claim 6, characterized in that, The mapping relationship of pseudo-color in S3 is as follows: In the fluorescent grayscale image, the maximum value maxFL corresponds to the actual minimum hue minH2, and the threshold B and below correspond to the actual maximum hue maxH2; The mapping method is: convert the visible light image VL to the HSV space, transform the H channel as shown in Equation (4), and then convert the transformed image back to the RGB space to obtain the mapped image VL'; （4）。 8. The method for extracting and fusing endoscopic fluorescence images according to claim 7, characterized in that, The weighted combination method in S3 is to calculate the weights of the mapped image VL' and the visible light image VL based on the fluorescence intensity, and then calculate the fused image FUS. In the fused image FUS, regions with fluorescence intensity higher than B use the value of the mapped image VL', and regions with fluorescence intensity lower than B use the value of the mapped image VL'. The portion uses the visible light image's VL value, which is between The portion between B and B is obtained by weighting the visible light image VL and the mapped image VL', as shown in equations (5) and (6); （5）， （6）。

Images