Endoscope system and image display method of endoscope

Abstract

According to the endoscope system and the image display method of the endoscope, different color characterization values are given to all the fluorescence sub-images in the original fluorescence image, and the fluorescence color image is obtained. The fluorescent color image comprises a first fluorescent color image and / or a second fluorescent color image; at least one fluorescent sub-image in the first fluorescent color image and / or the second fluorescent color image is a multi-color fluorescent sub-image, and color characterization values of all pixels of the multi-color fluorescent sub-image cover a color characterization value interval. And further displaying the fluorescent color image and / or a corresponding fusion image, wherein the fusion image is obtained by fusing the fluorescent color image and the white light image of the visual field area. Therefore, no matter the fluorescent color image or the fusion image thereof, the image of the tissue structure marked by different fluoresces can present different colors, the number of the types of the fluoresces is large, user distinguishing is not affected, and the image effect of the visual field area of the endoscope is good.

Description

Technical Field

[0001] The present invention relates to the field of medical devices, and particularly to an endoscope system and an image display method for an endoscope. Background Art

[0002] Traditional white light endoscopy imaging has been widely used in minimally invasive surgery. In addition, with the rapid development of various different fluorescent labels, fluorescence-based endoscopy navigation technology has gradually been applied clinically, bringing great convenience to vascular imaging, lymphatic tracing, and lesion localization in surgical operations. However, the display effects of fluorescent-labeled images and traditional white light images will have a major impact on the doctor's perception and judgment during the operation.

[0003] In the prior art, there is already a technology for fusing fluorescent images and white light images, that is, different fluorescent labels are used to label different tissue structures, and after various fluorescences are imaged, they are fused and displayed or fused with white light images and then displayed. The prior art is usually the fusion of dual-fluorescent images, such as using a simple red-blue channel ratio as the distinction of fluorescent images and attaching a single color for fusion. After the types of fluorescences increase, the images of different tissue structures will be difficult to distinguish in a fused single picture, and it is not suitable for the fusion display of more than two channels of fluorescences.

[0004] Therefore, the fusion effect of fluorescent images in existing endoscope imaging still needs to be improved. Summary of the Invention

[0005] The present invention mainly provides an endoscope system and an image display method for an endoscope, aiming to improve the image fusion effect.

[0006] An embodiment provides an image display method for an endoscope, including:

[0007] Exciting at least two fluorescent markers in the field of view of the endoscope to emit fluorescence, and the fluorescence bands of the fluorescence emitted by different types of fluorescent markers are different, and the fluorescence of one fluorescence band is used to label one tissue structure;

[0008] Collecting the original fluorescent image of the field of view, where the original fluorescent image includes sub-fluorescent images formed by the fluorescence emitted by the at least two fluorescent markers respectively;

[0009] Assigning different color characterization values to each of the sub-fluorescent images in the original fluorescent image to generate a fluorescent color image; wherein, the fluorescent color image includes a first fluorescent color image and / or a second fluorescent color image; the first fluorescent color image and / or the second fluorescent color image include multi-color sub-fluorescent images generated after the sub-fluorescent images are assigned color characterization values, and the color characterization values of all pixels of the multi-color sub-fluorescent images cover a color characterization value range;

[0010] Display the fluorescent color image, and / or capture a white light image of the field of view and display a fused image corresponding to the fluorescent color image, the fused image corresponding to the fluorescent color image being obtained by fusing the fluorescent color image with the white light image of the field of view.

[0011] In the method provided by an embodiment,

[0012] Each fluorescent sub-image in the first fluorescent color image is a multi-color fluorescent sub-image, and the color characterization value intervals corresponding to each multi-color fluorescent sub-image do not overlap;

[0013] The second fluorescent color image further includes a single-color fluorescent sub-image generated after the fluorescent sub-image is assigned a color characterization value; the color characterization values of all pixels of the single-color fluorescent sub-image are the same fixed value, the fixed values corresponding to each single-color fluorescent sub-image in the second fluorescent color image are different, and the color characterization value intervals corresponding to each multi-color fluorescent sub-image in the second fluorescent color image do not overlap.

[0014] In the method provided by an embodiment, the fluorescent color image further includes a third fluorescent color image, the third fluorescent color image includes a single-color fluorescent sub-image generated after the fluorescent sub-image is assigned a color characterization value, and each fluorescent sub-image in the third fluorescent color image is the single-color fluorescent sub-image; the fixed values corresponding to each single-color fluorescent sub-image in the third fluorescent color image are different.

[0015] In the method provided by an embodiment, the method further includes:

[0016] Display a setting window for the user to set the types of the respective fluorescent sub-images in the fluorescent color image, the types of the fluorescent sub-images being divided into two types: single-color fluorescent sub-images and multi-color fluorescent sub-images.

[0017] In the method provided by an embodiment, it further includes:

[0018] Receive an instruction from the user to select one or more fluorescent sub-images for deletion;

[0019] In response to the instruction, remove the selected fluorescent sub-images from the fluorescent color image and / or the corresponding fused image.

[0020] In the method provided by an embodiment, the endoscope has three display modes: color mode, hybrid mode, and monochrome mode; the method further includes: determining the current display mode based on the user's setting operation.

[0021] Displaying the fluorescent color image, and / or, acquiring a white light image of the field of view and displaying a fused image corresponding to the fluorescent color image, includes:

[0022] When the current display mode is the color mode, display the first fluorescent color image and / or acquire a white light image of the field of view and display a fused image corresponding to the first fluorescent color image;

[0023] When the current display mode is the hybrid mode, display the second fluorescent color image and / or acquire a white light image of the field of view and display a fused image corresponding to the second fluorescent color image;

[0024] When the current display mode is the monochrome mode, display the third fluorescent color image and / or acquire a white light image of the field of view and display a fused image corresponding to the third fluorescent color image.

[0025] In the method provided by an embodiment, the displaying the fluorescent color image, and / or, acquiring a white light image of the field of view and displaying a fused image corresponding to the fluorescent color image, includes:

[0026] Display multiple image combinations for the user to select; each image combination is associated with multiple target images, and the multiple target images associated with each image combination are not completely the same; the multiple target images include: multiple of the first fluorescent color image, the fused image corresponding to the first fluorescent color image, the second fluorescent color image, the fused image corresponding to the second fluorescent color image, the third fluorescent color image, and the fused image corresponding to the third fluorescent color image;

[0027] According to the image combination selected by the user, display the multiple target images associated with the selected image combination.

[0028] In the method provided by an embodiment, one of the multiple target images associated with each image combination is the main target image, and the displaying the multiple target images associated with the selected image combination includes:

[0029] On the display interface, display the multiple target images associated with the selected image combination, wherein, the display space occupied by the target images other than the main target image is less than half of the display space occupied by the main target image.

[0030] In the method provided by an embodiment, the color characterization value includes a hue value, the color characterization value range corresponding to the multi-color fluorescent sub-image includes the hue value range corresponding to the multi-color fluorescent sub-image; the hues between the respective fluorescent sub-images are complementary.

[0031] In the method provided by one embodiment, the color representation value includes a hue value, and the color representation value range corresponding to the multi-color fluorescent sub-image includes the hue value range corresponding to the multi-color fluorescent sub-image; endowing each fluorescent sub-image in the original fluorescent image with different color representation values includes:

[0032] According to the number of the fluorescent sub-images, equally divide the hue range to obtain a plurality of equally divided hue value ranges;

[0033] Assign the median of each equally divided hue value range to each fluorescent sub-image in the original fluorescent image respectively, where the median assigned to the monochromatic fluorescent sub-image in the original fluorescent image is used as the hue value of all pixels of the monochromatic fluorescent sub-image, and the median of an equally divided hue value range assigned to the multi-color fluorescent sub-image in the original fluorescent image is used as the median of the hue value range corresponding to the multi-color fluorescent sub-image.

[0034] In the method provided by one embodiment, endowing each fluorescent sub-image in the original fluorescent image with different color representation values to generate a fluorescent color image includes:

[0035] Endow each fluorescent sub-image in the original fluorescent image with different color representation values; according to the first mapping relationship between the preset fluorescence intensity and the brightness representation value of the fluorescent sub-image, map the fluorescence intensity corresponding to each pixel in the fluorescent sub-image into the brightness representation value of the pixel; thereby generating a fluorescent color image; wherein, the brightness representation value of the pixel is positively or negatively correlated with the fluorescence intensity of the pixel.

[0036] In the method provided by one embodiment, endowing each fluorescent sub-image in the original fluorescent image with different color representation values to generate a fluorescent color image includes:

[0037] Endow each fluorescent sub-image in the original fluorescent image with different color representation values; according to the second mapping relationship between the preset fluorescence intensity and the saturation representation value of the fluorescent sub-image, map the fluorescence intensity corresponding to each pixel in the fluorescent sub-image into the saturation representation value of the pixel; thereby generating a fluorescent color image; wherein, the saturation representation value of the pixel is positively or negatively correlated with the fluorescence intensity of the pixel.

[0038] In the method provided by one embodiment, endowing each fluorescent sub-image in the original fluorescent image with different color representation values to generate a first fluorescent color image includes:

[0039] Endow each fluorescent sub-image in the original fluorescent image with different color representation value ranges;

[0040] According to the color characterization value range of the fluorescent sub-image and the third mapping relationship between the preset fluorescent intensity and the color characterization value of the fluorescent sub-image, map the fluorescent intensity corresponding to each pixel in the fluorescent sub-image into the color characterization value of the pixel; thereby generating a first fluorescent color image.

[0041] In the method provided by an embodiment, the first mapping relationship includes a first mapping curve representing the mapping relationship between the fluorescent intensity and the brightness characterization value. The first mapping curve includes a background intensity threshold reflecting the fluorescent background. The curve segment on the first mapping curve where the fluorescent intensity is less than the background intensity threshold is the first curve segment, and the curve segment on the first mapping curve where the fluorescent intensity is greater than the background intensity threshold is the second curve segment; the greater the fluorescent intensity, the slower the growth of the brightness characterization value corresponding to the points on the second curve segment.

[0042] In the method provided by an embodiment, the greater the fluorescent intensity, the faster the growth of the brightness characterization value corresponding to the points on the first curve segment, or the brightness characterization values corresponding to the points on the first curve segment are all preset background brightness.

[0043] In the method provided by an embodiment, it further includes:

[0044] Receiving an instruction for increasing the brightness of a target fluorescent sub-image in the fluorescent color image, and in response to this instruction, increasing the brightness characterization value corresponding to the points on the first mapping curve of the target fluorescent sub-image, where the increase amount of the brightness characterization value corresponding to the background intensity threshold is the largest, and the increase amount of the brightness characterization value corresponding to the points on the first mapping curve that are farther away from the background intensity threshold is smaller; and / or,

[0045] Receiving an instruction for decreasing the brightness of a target fluorescent sub-image in the fluorescent color image, and in response to this instruction, decreasing the brightness characterization value corresponding to the points on the first mapping curve of the target fluorescent sub-image, where the decrease amount of the brightness characterization value corresponding to the background intensity threshold is the largest, and the decrease amount of the brightness characterization value corresponding to the points on the first mapping curve that are farther away from the background intensity threshold is smaller.

[0046] In the method provided by an embodiment,

[0047] Both the fluorescent original image and the fluorescent color image adopt the HSV color model or the HSL color model, and the color characterization value includes: hue value; or,

[0048] Both the fluorescent original image and the fluorescent color image adopt the YUV color model, and the color characterization value includes: chroma value.

[0049] An embodiment provides an image display method for an endoscope, including:

[0050] Using at least two imaging channels to image the field of view of the endoscope to obtain at least two channel images; the optical signal characteristics of each channel image are different, and one optical signal characteristic is used to mark a tissue structure;

[0051] Assigning a different color representation value to each channel image; wherein the color representation values ​​of all pixels of at least one channel image cover a color representation value range;

[0052] Each channel image is superimposed and displayed.

[0053] In the method provided by an embodiment, the channel image is obtained by imaging using narrow-band light imaging technology and / or laser speckle imaging technology.

[0054] One embodiment provides an endoscope system, comprising:

[0055] A light source is used to excite at least two fluorescent markers in the field of view of the endoscope to emit fluorescence, wherein different types of fluorescent markers emit fluorescence in different fluorescence bands, and fluorescence in one fluorescence band is used to mark one tissue structure;

[0056] An image acquisition device, used for acquiring a fluorescent original image of the visual field area, wherein the fluorescent original image includes fluorescent sub-images formed by the fluorescence emitted by the at least two fluorescent markers;

[0057] Memory, used to store programs;

[0058] A processor is used to execute the program to implement the method as described above.

[0059] An embodiment provides a computer-readable storage medium, on which a program is stored. The program can be executed by a processor to implement the method described above.

[0060] According to the endoscope system and the image display method of the endoscope in the above-mentioned embodiment, a different color representation value is assigned to each fluorescent sub-image in the fluorescent original image to obtain a fluorescent color image; wherein the fluorescent color image includes a first fluorescent color image and / or a second fluorescent color image; at least one fluorescent sub-image in the first fluorescent color image and / or the second fluorescent color image is a multi-color fluorescent sub-image, and the color representation values ​​of all pixels of the multi-color fluorescent sub-image cover a color representation value interval. Then, the fluorescent color image can be displayed, and the white light image of the field of view area can also be collected and the fused image corresponding to the fluorescent color image can be displayed. The fused image corresponding to the fluorescent color image is obtained by fusion of the fluorescent color image and the white light image of the field of view area. It can be seen that whether it is a fluorescent color image or its fused image, the image of the tissue structure marked by different fluorescent markers will present different colors, and the large number of fluorescent types does not affect the user's distinction, and the image effect of the field of view area of ​​the endoscope is good. Brief Description of the Drawings

[0061] Figure 1 Flow chart of an embodiment of the image display method of the endoscope provided by the present invention;

[0062] Figure 2 Schematic diagram of a black-and-white fluorescence original image;

[0063] Figure 3 Schematic diagram of a first fluorescence color image;

[0064] Figure 4 Schematic diagram of a second fluorescence color image;

[0065] Figure 5 Schematic diagram of a third fluorescence color image;

[0066] Figure 6 Schematic diagram of a white light image;

[0067] Figure 7 Schematic diagram of a fused image of a first fluorescence color image;

[0068] Figure 8 Schematic diagram of a fused image of a second fluorescence color image;

[0069] Figure 9 Schematic diagram of a fused image of a third fluorescence color image;

[0070] Figure 10 Block diagram of the structure of an embodiment of the endoscope system provided by the present invention;

[0071] Figure 11 For Figure 1 Flow chart of an embodiment of step 3 in;

[0072] Figure 12 Schematic diagram of a mapping curve;

[0073] Figure 13 Another schematic diagram of a mapping curve;

[0074] Figure 14 Schematic diagram of a display interface displaying multiple target images associated with an image combination;

[0075] Figure 15 Flow chart of another embodiment of the image display method of the endoscope provided by the present invention. Detailed Description of the Invention

[0076] The present invention will be further described in detail below in conjunction with the specific embodiments and the accompanying drawings. Similar elements in different embodiments are denoted by related similar element numbers. In the following embodiments, many detailed descriptions are provided to enable a better understanding of the present application. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification to avoid overwhelming the core part of the present application with excessive descriptions. For those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations based on the descriptions in the specification and the general technical knowledge in the art.

[0077] In addition, the features, operations, or characteristics described in the specification can be combined in any appropriate manner to form various embodiments. At the same time, the steps or actions in the method description can also be reordered or adjusted in a manner obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for clearly describing a certain embodiment and do not mean a necessary sequence, unless it is stated that a certain sequence must be followed.

[0078] The serial numbers assigned to the components in this article, such as "first", "second", etc., are only used to distinguish the described objects and do not have any sequential or technical meaning. The terms "connected" and "coupled" used in the present application, unless otherwise specified, both include direct and indirect connections (couplings).

[0079] In the present invention, the hue is innovatively adopted as the main criterion for distinguishing multi-channel fluorescence, and mappings are made through parameters such as saturation and brightness to reflect the fluorescence intensity, so that various tissue structures labeled with fluorescence in the image of the endoscopic viewing area are easily distinguishable. In addition, in order to balance the fusion effect of different channels, the fusion intensity of each channel can be subjected to multi-channel fusion using a mapping curve related to the signal and noise levels, thereby further improving the fusion effect of the image. The following will be described in detail through some embodiments.

[0080] As Figure 1 shown, the image display method of the endoscope provided by the present invention may include the following steps:

[0081] Step 1: Excite at least two fluorescent markers within the field of view of the endoscope to emit fluorescence. Different types of tissue structures can be pre-labeled with different types of fluorescent markers. When the endoscope is imaging, these fluorescent markers are excited to emit fluorescence by the light source of the endoscope or an external light source. The fluorescence here is a general concept, and actually refers to the light with a certain wavelength range emitted by the fluorescent marker after being excited by the light source. Different types of fluorescent markers emit fluorescence with different fluorescence wavelength ranges. The fluorescence of one fluorescence wavelength range is used to label one type of tissue structure, that is, one fluorescent marker can label one type of tissue structure.

[0082] Step 2: Collect the original fluorescence image of the field of view. The original fluorescence image contains sub-fluorescence images formed by the fluorescence emitted by the at least two fluorescent markers respectively. There are various ways to collect the original fluorescence image of the field of view. For example, various fluorescence imaging of the field of view is performed to obtain an original fluorescence image. The endoscope can detect the wavelength range of each fluorescence, such as analyzing the spectrum of the fluorescence to obtain the wavelength range of each fluorescence, so as to distinguish the fluorescence emitted by different types of fluorescent markers according to the different fluorescence wavelength ranges, and image the fluorescence emitted by each fluorescent marker separately, then the sub-fluorescence image of each fluorescent marker is obtained. If the sub-fluorescence image is displayed, it can present the picture of the tissue structure labeled by the fluorescent marker. Another example is that the endoscope has multiple imaging channels, and one imaging channel is used to image the fluorescence of one fluorescence wavelength range, so as to obtain the sub-fluorescence images formed by the fluorescence emitted by each fluorescent marker respectively. The sub-fluorescence images are fused or superimposed to obtain the original fluorescence image. In short, in this step, not only an original fluorescence image reflecting the overall field of view can be obtained, but also the sub-fluorescence images formed by the fluorescence emitted by each fluorescent marker separately can be obtained. The sub-fluorescence image is actually the sum of the fluorescence data corresponding to its fluorescence wavelength range, which not only covers the distribution of the fluorescence of this fluorescence wavelength range in the field of view, but also covers the fluorescence intensity of the fluorescence of this fluorescence wavelength range at each position in the field of view. It is only necessary to obtain these data, and the specific methods for obtaining these data are not limited in the present invention.

[0083] In the prior art, although the imaging device of some endoscopes can detect the fluorescence wavelength range, the image after fluorescence imaging is black and white, as Figure 2 shown, that is, the original fluorescence image obtained is black and white. If it is directly displayed, it will be impossible to distinguish each tissue structure. Figure 2G1, G2, and G3 will be difficult to distinguish if they are adjacent). Some imaging devices of endoscopes can detect the fluorescence band and directly present the color of the fluorescence on the image. The original fluorescence image obtained in this way is in color. However, for the colored fluorescence, their colors are usually quite similar, and it is still difficult to distinguish various tissue structures from the original fluorescence image. Through subsequent steps, the present invention processes the original fluorescence image, enabling users to clearly distinguish different tissue structures on the displayed image.

[0084] Step 3: Assign different color representation values to each fluorescence sub-image in the original fluorescence image to generate a fluorescence color image. For example, after each fluorescence sub-image in the original fluorescence image is assigned a different color representation value, the original fluorescence image becomes a fluorescence color image. Therefore, the fluorescence color image also has multiple fluorescence sub-images. That is, regardless of whether the fluorescence sub-images obtained in the previous step are black and white or colored, in this step, the fluorescence sub-images will be re-assigned colors to turn the original fluorescence image into a fluorescence color image. The fluorescence color image can include a first fluorescence color image and / or a second fluorescence color image. That is, this step can obtain one of the first and second fluorescence color images, or both the first and second fluorescence color images. Whether it is the first fluorescence color image or the second fluorescence color image, it includes the multi-color fluorescence sub-images generated after the fluorescence sub-images in the original fluorescence image are assigned color representation values. That is, among the fluorescence sub-images included in the first fluorescence color image and the second fluorescence color image, there is a type of multi-color fluorescence sub-images. The color representation values of all pixels of the multi-color fluorescence sub-images cover a color representation value range. That is, if the multi-color fluorescence sub-images are displayed, it can be seen that the colors at different positions on their tissue structures are different; it is equivalent to assigning a color representation value range to the fluorescence sub-images in the original fluorescence image to obtain the multi-color fluorescence sub-images in the fluorescence color image.

[0085] The difference between the first fluorescence color image and the second fluorescence color image lies in the presence or absence of monochromatic fluorescence sub-images. As Figure 3 shown, each fluorescence sub-image in the first fluorescence color image is a multi-color fluorescence sub-image. That is, each fluorescence sub-image (multi-color fluorescence sub-image) in the first fluorescence color image is obtained by assigning a color representation value range to each fluorescence sub-image in the original fluorescence image. Figure 3 is a simple schematic diagram of a first fluorescence color image. C1, C2, and C3 represent three multi-color fluorescence sub-images. It can be seen from the figure that in each multi-color fluorescence sub-image, the color representation values of the pixel points at different positions are not exactly the same, but there are multiple color representation values. The color representation value ranges corresponding to each multi-color fluorescence sub-image do not overlap. For example, C1 is in the yellow and green range, C2 is in the blue range, and C3 is in the red range. These three color ranges do not overlap, so that users can easily distinguish them when displayed on the display interface.

[0086] As Figure 4 shown, the second fluorescence color image further includes a monochromatic fluorescence sub-image generated after the fluorescence sub-images in the fluorescence original image are assigned color representation values. That is, a part of the fluorescence sub-images in the second fluorescence color image are monochromatic fluorescence sub-images, and the other part of the fluorescence sub-images are multi-color fluorescence sub-images. The color representation values of all pixels of the monochromatic fluorescence sub-image are the same fixed value, Figure 4 and S1 in Figure 4 is the monochromatic fluorescence sub-image, and the colors of all its pixels are yellowish green. The fixed value corresponding to each monochromatic fluorescence sub-image is different, that is, the single color corresponding to each monochromatic fluorescence sub-image is different, so that users can distinguish well when displayed on the display interface.

[0087] In some embodiments, the fluorescence color image may further include a third fluorescence color image. That is to say, the fluorescence color image may have three types: the first, the second, and the third fluorescence color images. As Figure 5 shown, the third fluorescence color image includes a monochromatic fluorescence sub-image generated after the fluorescence sub-images in the fluorescence original image are assigned color representation values, and each fluorescence sub-image in the third fluorescence color image is a monochromatic fluorescence sub-image. That is, each fluorescence sub-image in the third fluorescence color image is a monochromatic fluorescence sub-image. The color representation values of all pixels of the monochromatic fluorescence sub-image are the same fixed value, and the fixed value corresponding to each monochromatic fluorescence sub-image is different, that is, Figure 5 among the three monochromatic fluorescence sub-images S1, S2, and S3 in

[0088] Step 4: Display the fluorescence color image obtained in the previous step. Whether it is to display any one or several of the first, second, or third fluorescence color images, users can well distinguish the tissue structures marked by the fluorescence of each fluorescence band, which is convenient for performing relevant examinations and diagnoses on patients.

[0089] It is also possible to display the fusion image corresponding to the fluorescence color image obtained in the previous step. The fusion image corresponding to the fluorescence color image is obtained by fusing the fluorescence color image with the white light image of the field of view. For example, in the previous step, the white light image is also acquired at the same time. For example, the built-in or external light source of the endoscope emits white light to irradiate the field of view, and then images the field of view to obtain the white light image of the field of view, such as Figure 6As shown. The white light image and the fluorescence color image have the same size. After fusing the two, a fused image corresponding to the fluorescence color image is obtained (generated), such as Figures 7-9 shown. In the fused image, the part of the white light image serves as the background, enabling the doctor to comprehensively grasp the situation of the entire visual field area. Figures 6-9 The white light image in is colored in actual application. For the need of patent application, the color of the white light image in the attached drawings is removed.

[0090] The present invention also provides an endoscope system, such as Figure 10 shown. The system includes: a processor 10, a light source 20, an image acquisition device 30, and a memory 40.

[0091] The light source 20 is used to excite at least two fluorescent markers in the visual field area of the endoscope to emit fluorescence. Different types of fluorescent markers emit fluorescence in different fluorescence bands, and the fluorescence of one fluorescence band is used to label one tissue structure.

[0092] The image acquisition device 30 is used to acquire the original fluorescence image of the visual field area. The original fluorescence image contains fluorescence sub-images formed by the fluorescence emitted by at least two fluorescent markers respectively.

[0093] In this embodiment, the endoscope system may further include a white light source for emitting white light to irradiate the visual field area. The image acquisition device 30 is also used to acquire the white light image of the visual field area.

[0094] The memory 40 is used to store programs.

[0095] The processor 10 is used to execute the program to implement the method as Figure 1 shown. That is, each step in the above method embodiment can be completed by the processor 10 controlling the endoscope system. Further explanation will be given based on the above method embodiment.

[0096] The processor 10 excites at least two fluorescent markers in the visual field area of the endoscope to emit fluorescence through the light source 20. The more types of fluorescent markers in the visual field area, the more the advantages of the present invention can be reflected. For example, there are three, four, five,..., n or more fluorescent markers in the visual field area. The specific process is as described in step 1 above and will not be elaborated here.

[0097] The processor 10 acquires the original fluorescence image of the visual field area through the image acquisition device 30. The original fluorescence image contains fluorescence sub-images formed by the fluorescence emitted by at least two fluorescent markers respectively. There are various ways to specifically acquire the original fluorescence image of the visual field area, as described in step 2 above and will not be elaborated here.

[0098] The processor 10 can also emit white light through a white light source to illuminate the field of view area, and collect a white light image of the field of view area through the image acquisition device 30.

[0099] The processor 10 assigns different color representation values to each fluorescent sub-image in the fluorescent original image to generate a fluorescent color image. The specific process is as described in step 3 above and will not be elaborated here.

[0100] The endoscope system may further include a human-computer interaction device 50. The human-computer interaction device 50 is used for human-computer interaction, such as outputting visual information and receiving user input. The human-computer interaction device 50 includes an input device and at least one display. The input device is used to receive user input, which can be a keyboard, operation buttons, mouse, trackball, touchpad, etc., or a touch screen integrated with the display.

[0101] The processor 10 can display the obtained fluorescent color image on the display interface of the human-computer interaction device 50; it can also display a fusion image corresponding to the obtained fluorescent color image on the display interface of the human-computer interaction device 50. The specific process is as described in step 4 above and will not be elaborated here.

[0102] The endoscope system processes images, which can be based on any existing color model, such as the HSV color model, HSL color model, and YUV (YCbCr) color model, etc. For example, both the fluorescent original image and the fluorescent color image can adopt the HSV color model, or both can adopt the HSL color model, and the color representation value includes: hue value. Both the fluorescent original image and the fluorescent color image can also adopt the YUV color model, and the color representation value includes: chroma value.

[0103] Whether it is step 3 in the foregoing method embodiments or the processor 10, further limitations can be imposed on assigning different color representation values to each fluorescent sub-image to make it easier to distinguish between each fluorescent sub-image. Since the human eye has different sensitivities to different colors, how to map the fluorescent signals of different channels (fluorescent bands) to color (pseudo-color) is the key to effectively distinguishing each fluorescent sub-image and the white light image. For this reason, the color representation value of each fluorescent sub-image can be assigned based on the human eye color difference discrimination ability model, and the saturation, hue, etc. can also be mapped based on the fluorescence intensity. The principle is as follows: 1) The human eye has a weaker sensitivity to colors with high saturation and a stronger sensitivity to colors with low saturation. 2) The human eye has different sensitivities to different hues. The fundamental reason is determined by the different spectral response curves of the three types of cone cells (s, m, l) in the retina. The human eye is relatively more sensitive to green and yellow. 3) Different combinations of colors can produce different degrees of strong contrast. Generally speaking, the contrast produced by the combination of two or more colors with complementary hues is the strongest. In this embodiment, an example is given where the color representation value includes a hue value. The color representation value interval corresponding to the multi-color fluorescent sub-image includes the hue value interval corresponding to the multi-color fluorescent sub-image. The hues of each fluorescent sub-image are complementary to each other, and the difference between two complementary colors is more prominent.

[0104] Specifically, as Figure 11 shown, step 3 may include the following steps:

[0105] Step 31: The processor 10 equally divides the hue range according to the number of fluorescent sub-images to obtain a plurality of equally divided hue value intervals. Here, the hue range is usually the entire hue range defined by the color model, usually 0 - 360. If the subsequent display is a fluorescent color image and the fused image is not to be displayed, the processor 10 can equally divide the hue range according to the number n of fluorescent sub-images to obtain n equally divided hue value intervals. If the fused image needs to be displayed subsequently, in order to make it easier to distinguish between each fluorescent sub-image and the white light image serving as the background, the white light image is also taken into account during the equal division. The processor 10 can equally divide the hue range according to the number n of fluorescent sub-images and the white light image to obtain n + 1 equally divided hue value intervals. Since the greater the difference between different hues, the stronger the color contrast, the equal division method can make the color contrast between all images the strongest.

[0106] Step 32: The processor 10 assigns the median of each equal - divided hue - value interval to each fluorescent sub - image in the original fluorescent image. Among them, the median assigned to the single - color fluorescent sub - image in the original fluorescent image is used as the hue value of all pixels in the single - color fluorescent sub - image. That is to say, the hue values of all pixels in the single - color fluorescent sub - image are the median of the assigned hue - value interval. And for the multi - color fluorescent sub - image in the original fluorescent image, the median of an equal - divided hue - value interval assigned is used as the median of the corresponding hue - value interval of the multi - color fluorescent sub - image. The equal - divided hue - value interval can be used as the corresponding hue - value interval of the multi - color fluorescent sub - image. However, since the equal - divided hue - value intervals are relatively close to each other, the median of the equal - divided hue - value interval can remain unchanged, and the two endpoints of the interval are moved closer to the median, that is, the equal - divided hue - value interval is shrunk as the corresponding hue - value interval of the multi - color fluorescent sub - image.

[0107] If the white - light image is not considered when obtaining multiple equal - divided hue - value intervals by equal - division, the median of the hue - value interval can be randomly assigned to the fluorescent sub - images. If the white - light image is considered when obtaining multiple equal - divided hue - value intervals by equal - division, since the color hues of human tissues in the white - light image are usually mainly positive red, magenta, and yellow, the medians of the other hue - value intervals except the hue - value interval closest to the color hue of human tissues can be assigned to the fluorescent sub - images, so that the median assigned to each fluorescent sub - image has a greater difference from the color hue of human tissues as much as possible.

[0108] This embodiment is further described by taking the consideration of the white - light image as an example. The color - hue range of human tissues is known, and its median H0 is also known, which can be preset in advance. Of course, the processor 10 can also obtain the hues of all pixels in the white - light image and thus obtain the average hue of all pixels as the median H0. The processor 10 obtains the median H of the equal - divided hue - value interval assigned to each fluorescent sub - image through the following formula i :

[0109] where i = 1, 2, 3, ……, n;

[0110] n is the total number of fluorescent sub - images in the original fluorescent image, and H i is the median assigned to the i - th fluorescent sub - image. For the single - color fluorescent sub - image, the assigned median is used as the hue of all its pixel points; for the multi - color fluorescent sub - image, the hue - value interval (the color - representation - value interval covered by the color representations of all pixels) of the multi - color fluorescent sub - image can be obtained by expanding from the assigned median to both ends. For example, according to the preset hue - value range △H i and the assigned median H i , the hue - value interval of the multi - color fluorescent sub - image is obtained as follows:

[0111]

[0112] Thus, through the above steps 31 and 32, the assignment of different color characterization values to each fluorescence sub-image in the original fluorescence image is completed. For the fluorescence color image and its fused image obtained in this way, the color differences between various tissue structures are made as large as possible.

[0113] The high or low fluorescence intensity in the tissue structure can reflect the concentration difference of the fluorescent marker. Therefore, the fluorescence intensity can also be reflected in the fluorescence color image to provide doctors with more dimensional information. Specifically, in addition to assigning different color characterization values to each fluorescence sub-image in the original fluorescence image, the processor 10 also maps the fluorescence intensity corresponding to each pixel in the fluorescence sub-image into the brightness characterization value of the pixel according to the first mapping relationship between the preset fluorescence intensity and the brightness characterization value of the fluorescence sub-image, so as to obtain (generate) the fluorescence color image. Among them, the brightness characterization value of the pixel is positively or negatively correlated with the fluorescence intensity of the pixel. In this embodiment, the positive correlation is taken as an example for illustration, that is, for the fluorescence of a fluorescence band, the greater the fluorescence intensity, the brighter the pixel point corresponding to it on its fluorescence sub-image. The brightness characterization value characterizes the brightness of the pixel and has different names in different color models, such as brightness Y, lightness, etc.

[0114] The first mapping relationship may include a first mapping curve representing the mapping relationship between the fluorescence intensity and the brightness characterization value. For example, the first mapping curve is a curve in which the brightness characterization value changes with the change of the fluorescence intensity. In this way, the fluorescence intensity at the pixel point position can be directly mapped to obtain the brightness characterization value through the first mapping curve. Another example is, as Figure 12 shown, the first mapping curve is a curve in which the first fusion weight changes with the change of the fluorescence intensity. In this way, the fluorescence intensity at the pixel point position is multiplied by the corresponding first fusion weight on the first mapping curve to obtain the brightness characterization value. That is to say, the first fusion weight reflects the proportional relationship between the brightness characterization value and the fluorescence intensity, and different fluorescence intensities correspond to different first fusion weights. This embodiment is taken as an example for illustration.

[0115] Such as Figure 12As shown, the first mapping curve F includes a background intensity threshold Th that reflects the fluorescence background. There are three first mapping curves in the figure, namely F1, F2, and F3. The curve segment on the first mapping curve F where the fluorescence intensity is less than the background intensity threshold is the first curve segment f1 (such as the curve segment corresponding to the fluorescence intensity from 0 to Th1), and the curve segment on the first mapping curve F where the fluorescence intensity is greater than the background intensity threshold Th is the second curve segment f2 (such as the curve segment corresponding to the fluorescence intensity greater than Th1). The greater the fluorescence intensity, the slower the growth of the brightness characterization value corresponding to the points on the second curve segment f2, that is, the slower the growth of the first fusion weight corresponding to the points on the second curve segment f2, and the slope of the points on the second curve segment f2 decreases as the fluorescence intensity increases.

[0116] In some embodiments, the greater the fluorescence intensity, the faster the growth of the brightness characterization value corresponding to the points on the first curve segment f1, that is, the faster the growth of the first fusion weight corresponding to the points on the first curve segment f1, and the slope of the points on the first curve segment f1 increases as the fluorescence intensity increases, such that the first fusion weight below the background intensity threshold is relatively low, while the first fusion weight above the background intensity threshold is relatively high. In this way, the brightness characterization values corresponding to the fluorescence intensities before and after the background intensity threshold Th will differ significantly, thus well distinguishing the background light from the fluorescence. The background intensity threshold is preset, and the fluorescence intensity below the background intensity threshold is generally considered not to be fluorescence and may be noise such as background light. Of course, in some other embodiments, the processor 10 may map the brightness characterization values corresponding to the points on the first curve segment f1 to a preset background brightness, which is usually a relatively dark brightness, such as a brightness of 0. This can also eliminate the interference of the background light.

[0117] From Figure 12It can be seen that the first mapping curves corresponding to different background intensity thresholds are different. The processor 10 can select the first mapping curve suitable for the fluorescence sub-image according to the fluorescence intensity of each pixel in the fluorescence sub-image. Specifically, when collecting the original fluorescence image, due to various reasons such as collection accuracy and interference, light that is not the desired fluorescence (background light) may be collected. This causes the fluorescence sub-image in the original fluorescence image to be slightly larger than its tissue structure. And this light that is not fluorescence is usually significantly weaker than fluorescence, that is, there is an obvious boundary between the pixels formed by fluorescence and the pixels formed by background light on the fluorescence sub-image. Therefore, the processor 10 can analyze the brightness of each pixel in the fluorescence sub-image (corresponding to the fluorescence intensity), classify the brightness of each pixel in the fluorescence sub-image into two categories, for example, classify them into two categories by using the clustering method, and use the brightness of the pixels in the category with weaker brightness as the background intensity threshold. For example, use the average brightness or the highest brightness of the pixels in the category with weaker brightness as the background intensity threshold, so as to obtain the corresponding first mapping curve. Doing so is equivalent to determining that the pixels in the category with weaker brightness do not belong to the tissue structure corresponding to the fluorescence sub-image, and removing these pixels from the fluorescence sub-image. It can be seen that the higher the background light intensity, the larger the corresponding background intensity threshold. Whether the background light is strong or weak, it can adaptively eliminate it from the fluorescence sub-image of the fluorescence color image.

[0118] After each pixel of each fluorescence sub-image is assigned a hue and mapped with brightness, a fluorescence color image can be obtained (generated). Of course, in some embodiments, it is also possible to assign a hue and map the saturation to each pixel of each fluorescence sub-image, so as to obtain (generate) a fluorescence color image. Specifically, in addition to assigning different color representation values to each fluorescence sub-image in the original fluorescence image, the processor 10 also maps the fluorescence intensity corresponding to each pixel in the fluorescence sub-image into the saturation representation value of the pixel according to the second mapping relationship between the preset fluorescence intensity and the saturation representation value of the fluorescence sub-image, so as to obtain a fluorescence color image. Among them, the saturation representation value of the pixel is positively or negatively correlated with the fluorescence intensity of the pixel. In this embodiment, the positive correlation is taken as an example for illustration, that is, for the fluorescence in a fluorescence band, the greater its fluorescence intensity, the higher the saturation of the corresponding pixel point on its fluorescence sub-image. The saturation representation value represents the saturation of the pixel, and it can be the saturation itself.

[0119] The second mapping relationship may include a second mapping curve representing the mapping relationship between the fluorescence intensity and the saturation representation value. For example, the second mapping curve is a curve in which the saturation representation value changes with the change of the fluorescence intensity. In this way, the fluorescence intensity at the pixel point position can be directly mapped to obtain the saturation representation value through the second mapping curve. Another example is, Figure 12As shown, the second mapping curve is a curve in which the second fusion weight changes with the change of fluorescence intensity. Thus, the saturation characterization value can be obtained by multiplying the fluorescence intensity at the pixel position by the corresponding second fusion weight on the second mapping curve. That is to say, the second fusion weight reflects the proportional relationship between the saturation characterization value and the fluorescence intensity. Different fluorescence intensities correspond to different second fusion weights. This embodiment is used as an example for illustration.

[0120] The first mapping curve and the second mapping curve can be the same or different. Their change trends or change rules can be the same, so as to eliminate the influence of background light, enabling the doctor to better see the fluorescence intensity from the saturation of the fluorescence sub-image, and thus know the concentration or distribution of the fluorescent marker.

[0121] Continue to use Figure 12 the mapping curve in to illustrate. The second mapping curve includes the background intensity threshold Th reflecting the fluorescence background. There are three second mapping curves in the figure, namely F1, F2, and F3. The curve segment on the second mapping curve F where the fluorescence intensity is less than the background intensity threshold is the first curve segment f1 (such as the curve segment corresponding to the fluorescence intensity 0 - Th1), and the curve segment on the second mapping curve F where the fluorescence intensity is greater than the background intensity threshold Th is the second curve segment f2 (such as the curve segment corresponding to the fluorescence intensity greater than Th1). The greater the fluorescence intensity, the slower the saturation characterization value corresponding to the points on the second curve segment f2 increases, that is, the slower the second fusion weight corresponding to the points on the second curve segment f2 increases, and the slope of the points on the second curve segment f2 decreases as the fluorescence intensity increases.

[0122] In some embodiments, the greater the fluorescence intensity, the faster the saturation characterization value corresponding to the points on the first curve segment f1 increases, that is, the faster the second fusion weight corresponding to the points on the first curve segment f1 increases, and the slope of the points on the first curve segment f1 increases as the fluorescence intensity increases. In this way, the saturation characterization values corresponding to the fluorescence intensities before and after the background intensity threshold Th will differ greatly, thus well distinguishing the background light from the fluorescence. The background intensity threshold is preset, and the fluorescence intensity lower than the background intensity threshold is usually considered not to be fluorescence and may be noise such as background light. Of course, in some other embodiments, the processor 10 can map the saturation characterization values corresponding to the points on the first curve segment f1 to a preset background saturation. The background saturation is usually a relatively low saturation, such as a saturation of 0. This can also eliminate the interference of background light.

[0123] From Figure 12 it can be seen that different background intensity thresholds correspond to different second mapping curves. The processor 10 can select a second mapping curve suitable for the fluorescence sub-image according to the fluorescence intensity of each pixel in the fluorescence sub-image.

[0124] In some embodiments, after each pixel of each fluorescence sub-image is assigned a hue and mapped with brightness and saturation, a fluorescence color image is obtained (generated). For the specific process, refer to the foregoing content and will not be elaborated herein.

[0125] Certainly, in other embodiments, after a color representation value range is assigned to the multi-color fluorescence sub-image, the color representation value can also be mapped according to the fluorescence intensity of each pixel of the multi-color fluorescence sub-image within the assigned color representation value range. Specifically, after the processor 10 assigns different color representation values to each fluorescence sub-image in the fluorescence original image, for the case of generating the first fluorescence color image, after assigning different color representation value ranges to each fluorescence sub-image in the fluorescence original image, and for the case of generating the second fluorescence color image, after assigning different color representation value ranges to each multi-color fluorescence sub-image in the fluorescence original image, the processor 10 maps the fluorescence intensity corresponding to each pixel in the fluorescence sub-image into the color representation value of the pixel according to the color representation value range of the fluorescence sub-image and the third mapping relationship between the preset fluorescence intensity and the color representation value of the fluorescence sub-image, so as to obtain the first fluorescence color image and / or the second fluorescence color image. The third mapping relationship may include a third mapping curve representing the mapping relationship between the fluorescence intensity and the color representation value (such as hue). For example, the third mapping curve is a curve in which the color representation value changes with the change of the fluorescence intensity. In this way, the fluorescence intensity at the pixel position can be directly mapped to obtain the color representation value through the third mapping curve. Another example is that the third mapping curve is a curve in which the third fusion weight changes with the change of the fluorescence intensity. In this way, the fluorescence intensity at the pixel position is multiplied by the corresponding third fusion weight on the third mapping curve to obtain the color representation value. That is to say, the third fusion weight reflects the proportional relationship between the color representation value and the fluorescence intensity, and different fluorescence intensities correspond to different third fusion weights. This embodiment is described by taking this as an example. The color representation value of the pixel and the fluorescence intensity of the pixel can be positively or negatively correlated. Taking hue as an example, the higher the fluorescence intensity of the pixel, the higher or lower its hue.

[0126] After the processor 10 assigns different color representation values to each fluorescence sub-image in the fluorescence original image, it can map the brightness representation value, the saturation representation value, the color representation value, etc. according to the fluorescence intensity of the pixel, so as to obtain (generate) a fluorescence color image. For the specific process, refer to the foregoing content and will not be elaborated herein.

[0127] After displaying the fluorescence color image and / or the corresponding fusion image, if the user feels that the distinction between the fluorescence sub-images is not obvious, etc., manual adjustment can be performed to improve the fusion effect of each fluorescence sub-image. Specifically, the fluorescence sub-images of the fluorescence color image and the fusion image on the display interface can be selected by the user, and the selected fluorescence sub-image is the target fluorescence sub-image. For example, the processor 10 receives an instruction for selecting a fluorescence sub-image through the human-computer interaction device, and in response to this instruction, determines the fluorescence sub-image selected by the user as the target fluorescence sub-image. Among them, the fluorescence sub-images on the fluorescence color image and the fusion image can be selected, and the instruction for selecting a fluorescence sub-image is an instruction for selecting a fluorescence sub-image on the fluorescence color image or the fusion image. Of course, corresponding virtual buttons can also be displayed on the display interface for the user to select, so as to select the corresponding fluorescence sub-image. Furthermore, the processor 10 receives an instruction for increasing the brightness of the target fluorescence sub-image in the fluorescence color image through the human-computer interaction device, and in response to this instruction, increases the brightness characterization value corresponding to the point on the first mapping curve of the target fluorescence sub-image, as Figure 13 shown, the original first mapping curve F1 has become F2. The fusion weights corresponding to the same fluorescence intensity have basically increased. Among them, the increase in the brightness characterization value corresponding to the background intensity threshold is the largest, and the increase in the brightness characterization value corresponding to the point on the first mapping curve that is farther away from the background intensity threshold is smaller. That is, although the first mapping curve is adjusted, the change trend of its curve remains unchanged, and the background light can still be made not to be displayed or weakly displayed, and the brightness difference of the pixel points can clearly show the fluorescence intensity. In the fluorescence color image and the fusion image, if the user wants to focus on an organizational structure, the overall brightness of the fluorescence sub-image of this organizational structure can be increased, so as to observe better. Of course, the user can also lower the overall brightness of the selected fluorescence sub-image. Specifically, the processor 10 receives an instruction for decreasing the brightness of the target fluorescence sub-image in the fluorescence color image through the human-computer interaction device, and in response to this instruction, decreases the brightness characterization value corresponding to the point on the first mapping curve of the target fluorescence sub-image, such as changing from F2 to F1. Among them, the decrease in the brightness characterization value corresponding to the background intensity threshold is the largest, and the decrease in the brightness characterization value corresponding to the point on the first mapping curve that is farther away from the background intensity threshold is smaller.

[0128] In this embodiment, there is a region or window on the display interface for the user to adjust the luminance representation value A corresponding to the background intensity threshold. One luminance representation value A corresponding to a background intensity threshold corresponds to a first mapping curve. Multiple luminance representation values A corresponding to background intensity thresholds and their corresponding first mapping curves can be pre-stored, or the processor 10 can automatically generate a corresponding first mapping curve according to the function relation of the first mapping curve with the luminance representation value A corresponding to the background intensity threshold adjusted by the user as a parameter in the function relation. Therefore, when the user adjusts the luminance representation value A corresponding to the background intensity threshold, the first mapping curve is also adjusted. As Figure 13 shown, increasing the luminance representation value corresponding to the background intensity threshold from A1 to A2 means changing the first mapping curve from A1 to A2. The processor 10 receives the luminance representation value A corresponding to the background intensity threshold adjusted by the user in the region or window through the human-computer interaction device, thereby determining the adjusted first mapping curve and updating the target fluorescent sub-image according to the adjusted first mapping curve. It can be seen that the user can adjust the fusion weight corresponding to the background intensity threshold to change the fusion effect of the fluorescent images of each channel. For example, when the intensity of the fluorescence is weak, increasing the fusion weight corresponding to the background intensity threshold can increase the intensity of the fused fluorescent signal, and when the fluorescence intensity is high, weakening the fused fluorescent signal intensity can be achieved by reducing the fusion weight corresponding to the background intensity threshold. This adjustment method is simple to operate and has an intuitive effect.

[0129] This adjustment method is also applicable to the second mapping curve and the third mapping curve.

[0130] Specifically, the processor 10 receives, through the human-computer interaction device, an instruction for increasing the saturation of the target fluorescent sub-image in the fluorescent color image. In response to this instruction, the saturation characterization value corresponding to the point on the second mapping curve of the target fluorescent sub-image is increased, and the corresponding second fusion weights under the same fluorescence intensity are basically all increased. Among them, the increase in the saturation characterization value corresponding to the background intensity threshold is the largest, and the increase in the saturation characterization value corresponding to the point on the second mapping curve that is farther away from the background intensity threshold is smaller. The processor 10 receives, through the human-computer interaction device, an instruction for decreasing the saturation of the target fluorescent sub-image in the fluorescent color image. In response to this instruction, the saturation characterization value corresponding to the point on the second mapping curve of the target fluorescent sub-image is decreased. Among them, the decrease in the saturation characterization value corresponding to the background intensity threshold is the largest, and the decrease in the saturation characterization value corresponding to the point on the second mapping curve that is farther away from the background intensity threshold is smaller. In this embodiment, the display interface may also have an area or window for the user to adjust the saturation characterization value corresponding to the background intensity threshold. The saturation characterization value corresponding to one background intensity threshold corresponds to one second mapping curve. Multiple saturation characterization values corresponding to background intensity thresholds and their corresponding second mapping curves may be pre-stored, or the processor 10 may automatically generate a corresponding second mapping curve according to the function relationship of the second mapping curve with the saturation characterization value corresponding to the background intensity threshold adjusted by the user as a parameter in the function relationship. Therefore, when the user adjusts the saturation characterization value corresponding to the background intensity threshold, the second mapping curve is also adjusted. The processor 10 receives, through the human-computer interaction device, the saturation characterization value corresponding to the background intensity threshold adjusted by the user in the said area or window, thereby determining the adjusted second mapping curve, and updating the target fluorescent sub-image according to the adjusted second mapping curve.

[0131] The processor 10 receives, through the human-computer interaction device, an instruction for increasing the hue of the target fluorescent sub-image in the fluorescent color image. In response to this instruction, the hue characterization value corresponding to the point on the third mapping curve of the target fluorescent sub-image is increased, and the corresponding third fusion weights under the same fluorescence intensity are basically all increased. Among them, the increase amount of the hue characterization value corresponding to the background intensity threshold is the largest, and the increase amount of the hue characterization value corresponding to the point on the third mapping curve that is farther away from the background intensity threshold is smaller. The processor 10 receives, through the human-computer interaction device, an instruction for decreasing the hue of the target fluorescent sub-image in the fluorescent color image. In response to this instruction, the hue characterization value corresponding to the point on the third mapping curve of the target fluorescent sub-image is decreased. Among them, the decrease amount of the hue characterization value corresponding to the background intensity threshold is the largest, and the decrease amount of the hue characterization value corresponding to the point on the third mapping curve that is farther away from the background intensity threshold is smaller. In this embodiment, there may also be an area or window on the display interface for the user to adjust the hue characterization value corresponding to the background intensity threshold. The hue characterization value corresponding to one background intensity threshold corresponds to one third mapping curve. Multiple hue characterization values corresponding to background intensity thresholds and their corresponding third mapping curves may be pre-stored, or the processor 10 may automatically generate the corresponding third mapping curve according to the function relationship of the third mapping curve with the hue characterization value corresponding to the background intensity threshold adjusted by the user as a parameter in the function relationship. Therefore, when the user adjusts the hue characterization value corresponding to the background intensity threshold, the third mapping curve is also adjusted. The processor 10 receives, through the human-computer interaction device, the hue characterization value corresponding to the background intensity threshold adjusted by the user in the said area or window, thereby determining the adjusted third mapping curve, and updating the target fluorescent sub-image according to the adjusted third mapping curve.

[0132] In the fluorescent color image, which tissue structure's fluorescent sub-image uses a monochromatic fluorescent sub-image and which uses a multi-color fluorescent sub-image can be pre-set by the system or set by the user. For example, the processor 10 displays, through the human-computer interaction device, a setting window for the user to set the types of each fluorescent sub-image in the fluorescent color image. The types of fluorescent sub-images are divided into two: monochromatic fluorescent sub-images and multi-color fluorescent sub-images. That is, the user can determine whether each tissue structure marked by fluorescence is monochromatic or multi-color when displayed in the setting window, which is very flexible.

[0133] A fluorescent color image on the display interface, and its fluorescent sub-images can also be deleted, that is, the user can operate to view which tissue structures and which tissue structures not to view. Since the endoscope system has all the fluorescent data in the viewing area, and the fluorescent color image is generated from this fluorescent data, it can be achieved. Specifically, the processor 10 receives, through the human-computer interaction device, an instruction from the user to select one or more fluorescent sub-images for deletion; for example, the user first operates the human-computer interaction device to select one or more fluorescent sub-images, and then issues a deletion instruction. In response to this instruction, the processor 10 removes the selected fluorescent sub-images from the fluorescent color image and / or the corresponding fusion image. For example, in the original step of generating the fluorescent color image, the fluorescent sub-images selected by the user are removed, and the selected fluorescent sub-image will not be displayed in the newly generated fluorescent color image.

[0134] In the aforementioned step 3, there are three cases for assigning different color representation values to each fluorescent sub-image, that is, all the assigned color representation values are intervals, some of the color representation values are fixed values and some are intervals, and all the color representation values are fixed values. From Figures 3-5 it can be seen that the fluorescent color images displayed in these three cases are different, and these three cases can be divided into three display modes for the user to select. That is, the endoscope system can have three display modes: color mode, hybrid mode, and monochrome mode.

[0135] The processor 10 also provides a setting interface for the user to set the display mode. For example, the processor 10 determines the current display mode based on the user's setting operation. If the current display mode is the color mode, then the first fluorescent color image is obtained in step 3. In step 4, the processor 10 displays the first fluorescent color image and / or the fusion image corresponding to the first fluorescent color image through the human-computer interaction device, as Figure 7 and Figure 3 shown. If the current display mode is the hybrid mode, then the second fluorescent color image is obtained in step 3. In step 4, the processor 10 displays the second fluorescent color image and / or the fusion image corresponding to the second fluorescent color image through the human-computer interaction device, as Figure 8 and Figure 4 shown. If the current display mode is the monochrome mode, then the third fluorescent color image is obtained in step 3. In step 4, the processor 10 displays the third fluorescent color image and / or the fusion image corresponding to the third fluorescent color image through the human-computer interaction device, as Figure 9 and Figure 5 shown.

[0136] For the organizational structures with multiple fluorescence markings, the difficulty lies in differentiation. From the above content, it can be seen that the endoscope system provided by the present invention can not only provide fluorescence color images with good differentiation, but also the user can make various adjustments to the fluorescence sub-images in the fluorescence color images in the later stage, so that the fluorescence color images can better meet the user's needs. All these greatly improve the fusion effect between fluorescence images and between fluorescence images and white light images.

[0137] From the foregoing content, it can be seen that there are three types of fluorescence color images, and correspondingly there are three types of fusion images. The fluorescence sub-images displayed by each type are different. Although the user can set the display mode, the user may not know which display mode is more suitable. However, if these images are displayed together, limited by the area of the display interface, the size of each image displayed will be relatively small, which is inconvenient for the user to view. In this embodiment, these images are arranged and combined into multiple combinations for the user to select, and the number of images in each combination is less than 6, so as to ensure the display size of a single image and at the same time display multiple images for the user to select the best one for key viewing. Specifically, in step 4, the processor 10 displays the fluorescence color image through the human-computer interaction device, and / or displays the fusion image corresponding to the fluorescence color image, which may include: displaying multiple image combinations for the user to select, and then according to the image combination selected by the user, displaying multiple target images associated with the selected image combination. Among them, each image combination is associated with multiple target images, and the multiple target images associated with each image combination are not completely the same; the multiple target images include: multiple of the first fluorescence color image, the fusion image corresponding to the first fluorescence color image, the second fluorescence color image, the fusion image corresponding to the second fluorescence color image, the third fluorescence color image, and the fusion image corresponding to the third fluorescence color image. Usually, the number of target images associated with each image combination is the same, and the difference lies in the arrangement and combination of the target images. Taking each image combination being associated with 4 target images as an example, some image combinations can be associated with Figure 3 , Figure 4 , Figure 7 and Figure 8 , and some image combinations can be associated with Figure 4 , Figure 5 , Figure 8 and Figure 9 etc. Of course, the image combination can also take into account the white light image and the black-and-white fluorescence original image, that is, in some embodiments, the multiple target images associated with each image combination include: multiple of the first fluorescence color image, the fusion image corresponding to the first fluorescence color image, the second fluorescence color image, the fusion image corresponding to the second fluorescence color image, the third fluorescence color image, the fusion image corresponding to the third fluorescence color image, the white light image, and the black-and-white fluorescence original image.

[0138] When the processor 10 displays multiple target images associated with a selected image combination, it can optimize the display. For example, among the multiple target images associated with each image combination, there is a main target image, which is preset by the system or the user, indicating that it is more suitable for display than other target images associated with the image combination and is used as the main recommended target image. As Figure 14 shown, the processor 10 can display multiple target images associated with a selected image combination on the display interface of the human-computer interaction device. Among them, the display space occupied by the target images other than the main target image is less than half of the display space occupied by the main target image. Figure 14 In this case, the main target image is displayed in the largest display area A, and the other target images are displayed in other display areas B, C, and D with areas much smaller than A. The aspect ratio of the length and width of the display areas A, B, C, and D can be 4:3, and the four display areas are adjacent and form a large display area (the aspect ratio of the length and width of this large display area can be 16:9). In this way, the area of the display can be fully utilized.

[0139] Multi-channel fluorescence is a key technology for the next-generation fluorescence navigation endoscopy technology and is of great significance for promoting precision surgery. The image display methods of the endoscope system and the endoscope provided by the present invention can well meet the imaging and display requirements of multi-channel fluorescence and are important functions of multi-channel fluorescence imaging.

[0140] Of course, the above image display method is not limited to the fusion display of multi-channel fluorescence images and can also be applied to endoscope systems using other imaging technologies. Therefore, in one embodiment, the endoscope system includes: a processor 10, a light source 20, an image acquisition device 30, a memory 40, and a human-computer interaction device 50. Among them, the processor 10 controls the endoscope system to perform the image display method, as Figure 15 shown, including the following steps:

[0141] Step 2': The processor 10 uses at least two imaging channels through the image acquisition device 30 to image the field of view area of the endoscope to obtain at least two channel images. The optical signal characteristics for forming each channel image are different, and one optical signal characteristic is used to mark one tissue structure. If fluorescence imaging is used, then the optical signal characteristic here is fluorescence, that is Figure 1 shown in the embodiment. In this embodiment, the image acquisition device 30 can adopt narrow-band light imaging technology, laser speckle imaging technology, or both of these imaging technologies. That is, the channel images are obtained by imaging using narrow-band light imaging technology and / or laser speckle imaging technology.

[0142] Step 3': The processor 10 assigns different color representation values to each channel image; among them, the color representation values of all pixels of at least one channel image cover a color representation value interval. Similarly, the color representation values assigned to each channel image can all be a color representation value interval, or the color representation values assigned to some channel images are different fixed values, and the color representation values assigned to the other part of the channel images are color representation value intervals, and the respective color representation value intervals do not overlap. The specific process of this step is the same as the specific process of assigning different color representation values to each fluorescence sub-image in step 3 of the foregoing embodiment, that is, changing the fluorescence sub-image in the foregoing embodiment to a channel image, and the specific process of this step 3' is obtained, so it will not be elaborated here.

[0143] Step 4': The processor 10 superimposes and displays each channel image. Of course, each channel image can also be fused with a white light image to obtain a fused image. The specific process is the same as that of step 4 in the above embodiment and will not be elaborated here.

[0144] Similar to the foregoing embodiment, both are imaging the visual field area of the endoscope to obtain images of multiple tissue structures. The only difference is that one is a fluorescence image and the other is a channel image. The subsequent processing methods and display methods for the tissue structure images are the same, so they will not be elaborated here.

[0145] This document has been described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope of this document. For example, various operation steps and the components used to perform the operation steps can be implemented in different ways according to a specific application or any number of cost functions associated with the operation of the system (for example, one or more steps can be deleted, modified, or combined into other steps).

[0146] In addition, as will be understood by those skilled in the art, the principles herein may be embodied in a computer program product on a computer-readable storage medium preloaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions may be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to form a machine, such that the instructions executed on the computer or other programmable data processing apparatus may generate an apparatus for implementing the specified functions. These computer program instructions may also be stored in a computer-readable memory, which may direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory may form a manufacture, including an implementation apparatus for implementing the specified functions. The computer program instructions may also be loaded onto a computer or other programmable data processing device, thereby performing a series of operational steps on the computer or other programmable device to generate a computer-implemented process, such that the instructions executed on the computer or other programmable device may provide steps for implementing the specified functions.

[0147] While the principles herein have been shown in various embodiments, many modifications of structure, arrangement, proportions, elements, materials, and components, which are particularly adapted to specific environments and operational requirements, may be used without departing from the principles and scope of this disclosure. The above modifications and other changes or revisions will be included within the scope hereof.

[0148] The foregoing detailed description has been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes may be made without departing from the scope of this disclosure. Accordingly, the consideration of this disclosure will be in an illustrative rather than a limiting sense, and all such modifications will be included within its scope. Similarly, the advantages, other advantages, and solutions to problems of the various embodiments have been described above. However, benefits, advantages, solutions to problems, and any elements that may produce these, or solutions that make them more apparent, should not be construed as critical, essential, or necessary. The term "comprising" and any other variant thereof used herein are non-exclusive inclusions, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but also other elements not expressly listed or inherent to the process, method, system, article, or apparatus. In addition, the term "coupled" and any other variant thereof used herein refers to physical connection, electrical connection, magnetic connection, optical connection, communication connection, functional connection, and / or any other connection.

[0149] Those skilled in the art will recognize that many changes may be made to the details of the above-described embodiments without departing from the basic principles of the present invention. Accordingly, the scope of the present invention should be determined in accordance with the following claims.

Claims

1. An image display method for an endoscope, characterized in that, Including: Exciting at least two fluorescent markers within the field of view of the endoscope to emit fluorescence, where the fluorescence bands of the different types of fluorescent markers are different, and the fluorescence of one fluorescence band is used to label one tissue structure; Collecting the original fluorescence image of the field of view, where the original fluorescence image includes sub-fluorescence images formed by the fluorescence emitted by the at least two fluorescent markers respectively; Assigning different color representation values to each of the sub-fluorescence images in the original fluorescence image to generate a fluorescence color image; where the fluorescence color image includes a first fluorescence color image and / or a second fluorescence color image; the first fluorescence color image and / or the second fluorescence color image include multi-color sub-fluorescence images generated after the sub-fluorescence images are assigned color representation values, and the color representation values of all pixels of the multi-color sub-fluorescence images cover a color representation value range; Displaying the fluorescence color image, and / or, collecting the white light image of the field of view and displaying the fused image corresponding to the fluorescence color image, where the fused image corresponding to the fluorescence color image is obtained by fusing the fluorescence color image and the white light image of the field of view.

2. The method according to claim 1, wherein Each sub-fluorescence image in the first fluorescence color image is a multi-color sub-fluorescence image, and the color representation value ranges corresponding to each multi-color sub-fluorescence image do not overlap; The second fluorescence color image further includes a single-color sub-fluorescence image generated after the sub-fluorescence image is assigned a color representation value; the color representation values of all pixels of the single-color sub-fluorescence image are the same fixed value, the fixed values corresponding to each single-color sub-fluorescence image in the second fluorescence color image are different, and the color representation value ranges corresponding to each multi-color sub-fluorescence image in the second fluorescence color image do not overlap.

3. The method according to claim 2, wherein The fluorescence color image further includes a third fluorescence color image, and the third fluorescence color image includes a single-color sub-fluorescence image generated after the sub-fluorescence image is assigned a color representation value, and each sub-fluorescence image in the third fluorescence color image is the single-color sub-fluorescence image; The fixed values corresponding to each single-color sub-fluorescence image in the third fluorescence color image are different.

4. The method according to claim 2, wherein The method further includes: Displaying a setting window for the user to set the type of each sub-fluorescence image in the fluorescence color image, and the types of sub-fluorescence images are divided into two types: single-color sub-fluorescence images and multi-color sub-fluorescence images.

5. The method according to claim 2, wherein It also includes: Receiving an instruction from the user to select one or more sub-fluorescence images for deletion; In response to the instruction, removing the selected sub-fluorescence images from the fluorescence color image and / or the corresponding fused image.

6. The method according to claim 3, wherein The endoscope has three display modes: color mode, hybrid mode, and monochrome mode; the method further includes: determining the current display mode based on the user's setting operation; The displaying the fluorescence color image, and / or, collecting the white light image of the field of view and displaying the fused image corresponding to the fluorescence color image includes: When the current display mode is the color mode, display the first fluorescence color image and / or acquire the white light image of the field of view and display the fused image corresponding to the first fluorescence color image; When the current display mode is the hybrid mode, display the second fluorescence color image and / or acquire the white light image of the field of view and display the fused image corresponding to the second fluorescence color image; When the current display mode is the monochromatic mode, display the third fluorescence color image and / or acquire the white light image of the field of view and display the fused image corresponding to the third fluorescence color image.

7. The method according to claim 3, wherein The displaying the fluorescence color image, and / or, acquiring the white light image of the field of view, and displaying the fused image corresponding to the fluorescence color image, includes: Displaying multiple image combinations for the user to select; each image combination is associated with multiple target images, and the multiple target images associated with each image combination are not completely the same; the multiple target images include: multiple of the first fluorescence color image, the fused image corresponding to the first fluorescence color image, the second fluorescence color image, the fused image corresponding to the second fluorescence color image, the third fluorescence color image, and the fused image corresponding to the third fluorescence color image; According to the image combination selected by the user, display the multiple target images associated with the selected image combination.

8. The method according to claim 7, wherein Among the multiple target images associated with each image combination, one is the main target image, and the displaying the multiple target images associated with the selected image combination includes: Display the multiple target images associated with the selected image combination on the display interface, wherein the display space occupied by the target images other than the main target image is less than half of the display space occupied by the main target image.

9. The method according to claim 2, wherein The color characterization value includes a hue value, and the color characterization value interval corresponding to the multi-color fluorescent sub-image includes the hue value interval corresponding to the multi-color fluorescent sub-image; the hues between the respective fluorescent sub-images are complementary.

10. The method according to claim 2, characterized in that, The color characterization value includes a hue value, and the color characterization value interval corresponding to the multi-color fluorescent sub-image includes the hue value interval corresponding to the multi-color fluorescent sub-image; The endowing each of the fluorescent sub-images in the fluorescent original image with different color characterization values includes: According to the number of the fluorescent sub-images, equally divide the hue range to obtain multiple equally divided hue value intervals; Assign the median of each equally divided hue value interval to each of the fluorescent sub-images in the fluorescent original image, wherein the median assigned to the monochromatic fluorescent sub-image in the fluorescent original image is used as the hue value of all pixels of the monochromatic fluorescent sub-image, and the median of an equally divided hue value interval assigned to the multi-color fluorescent sub-image in the fluorescent original image is used as the median of the hue value interval corresponding to the multi-color fluorescent sub-image.

11. The method according to claim 1, wherein The endowing each of the fluorescent sub-images in the fluorescent original image with different color characterization values to generate a fluorescence color image includes: Assign different color characterization values to each of the fluorescence sub-images in the original fluorescence image; according to the first mapping relationship between the preset fluorescence intensity and the brightness characterization value of the fluorescence sub-image, map the fluorescence intensity corresponding to each pixel in the fluorescence sub-image into the brightness characterization value of the pixel; thereby generating a fluorescence color image; wherein, the brightness characterization value of the pixel is positively or negatively correlated with the fluorescence intensity of the pixel.

12. The method according to claim 1, characterized in that The step of assigning different color characterization values to each fluorescence sub-image in the original fluorescence image to generate a fluorescence color image includes: Assign different color characterization values to each of the fluorescence sub-images in the original fluorescence image; according to the second mapping relationship between the preset fluorescence intensity and the saturation characterization value of the fluorescence sub-image, map the fluorescence intensity corresponding to each pixel in the fluorescence sub-image into the saturation characterization value of the pixel; thereby generating a fluorescence color image; wherein, the saturation characterization value of the pixel is positively or negatively correlated with the fluorescence intensity of the pixel.

13. The method according to claim 2, characterized in that, The step of assigning different color characterization values to each of the fluorescence sub-images in the original fluorescence image to generate a first fluorescence color image includes: Assign different color characterization value intervals to each of the fluorescence sub-images in the original fluorescence image; According to the color characterization value interval of the fluorescence sub-image and the third mapping relationship between the preset fluorescence intensity and the color characterization value of the fluorescence sub-image, map the fluorescence intensity corresponding to each pixel in the fluorescence sub-image into the color characterization value of the pixel; thereby generating a first fluorescence color image.

14. The method according to claim 11, wherein The first mapping relationship includes a first mapping curve representing the mapping relationship between the fluorescence intensity and the brightness characterization value. The first mapping curve includes a background intensity threshold reflecting the fluorescence background. The curve segment on the first mapping curve where the fluorescence intensity is less than the background intensity threshold is the first curve segment, and the curve segment on the first mapping curve where the fluorescence intensity is greater than the background intensity threshold is the second curve segment; The greater the fluorescence intensity, the slower the growth of the brightness characterization value corresponding to the points on the second curve segment.

15. The method according to claim 14, wherein The greater the fluorescence intensity, the faster the growth of the brightness characterization value corresponding to the points on the first curve segment, or the brightness characterization values corresponding to the points on the first curve segment are all preset background brightness values.

16. The method according to claim 14 or 15, characterized in that, It further includes: Receiving an instruction for increasing the brightness of a target fluorescence sub-image in the fluorescence color image, and in response to this instruction, increasing the brightness characterization value corresponding to the points on the first mapping curve of the target fluorescence sub-image, where the increase amount of the brightness characterization value corresponding to the background intensity threshold is the largest, and the increase amount of the brightness characterization value corresponding to the points on the first mapping curve that are farther away from the background intensity threshold is smaller; and / or, Receiving an instruction for decreasing the brightness of a target fluorescence sub-image in the fluorescence color image, and in response to this instruction, decreasing the brightness characterization value corresponding to the points on the first mapping curve of the target fluorescence sub-image, where the decrease amount of the brightness characterization value corresponding to the background intensity threshold is the largest, and the decrease amount of the brightness characterization value corresponding to the points on the first mapping curve that are farther away from the background intensity threshold is smaller.

17. The method according to claim 1, wherein Both the original fluorescence image and the fluorescence color image adopt the HSV color model or the HSL color model, and the color characterization values include: hue values; or, Both the original fluorescence image and the fluorescence color image adopt the YUV color model, and the color characterization values include: chroma values.

18. An image display method for an endoscope, characterized in that, Including: Imaging the field of view area of the endoscope with at least two imaging channels to obtain at least two channel images; The optical signal characteristics forming each channel image are different, and one optical signal characteristic is used to label one tissue structure; Assigning different color characterization values to each channel image; wherein, the color characterization values of all pixels of at least one channel image cover a color characterization value interval; Overlaying and displaying each channel image.

19. The method according to claim 18, wherein The channel image is obtained by imaging with narrow-band light imaging technology and / or laser speckle imaging technology.

20. An endoscope system, characterized in that, Including: A light source for exciting at least two fluorescent markers in the field of view area of the endoscope to emit fluorescence. The fluorescence bands of the fluorescence emitted by different types of fluorescent markers are different, and the fluorescence of one fluorescence band is used to label one tissue structure; An image acquisition device for acquiring the original fluorescence image of the field of view area, and the original fluorescence image includes fluorescence sub-images respectively formed by the fluorescence emitted by the at least two fluorescent markers; A memory for storing programs; A processor for executing the program to implement the method according to any one of claims 1-19.

21. A computer-readable storage medium, characterized in that, The medium stores a program that can be executed by a processor to implement the method according to any one of claims 1-19.

Images