Endoscope image display method and apparatus, and terminal device and storage medium

Through the endoscopic image display method, the tissue profile is extracted and fused into the original white light image, solving the problem that the prior art cannot display the lesion location and tissue information at the same time, improving surgical efficiency and reducing risks.

WO2024240089A1PCT designated stage expired Publication Date: 2024-11-28CHONGQING XISHAN SCI & TECH

Patent Information

Application Number
PCT/CN2024/094009
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-05-17
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The existing endoscopic imaging display mode cannot simultaneously display the location and tissue information of fluorescently labeled tumor lesions, resulting in doctors needing to frequently switch the display mode during surgery, affecting efficiency and increasing risks.

Method used

By acquiring the original white light image of the endoscope and the marked tissue image, the tissue profile is extracted, and the target tissue profile is obtained, and the original white light image is fused to display the tissue information of the marking position while marking the lesion position.

Benefits of technology

It improves surgical efficiency, reduces surgical risks, and avoids the drawbacks of fluorescent images covering the original color and structure of the tissue in traditional fusion mode.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024094009_28112024_PF_FP_ABST
    Figure CN2024094009_28112024_PF_FP_ABST
Patent Text Reader

Abstract

An endoscope image display method, comprising: acquiring an original white-light image of an endoscope and a marked tissue image thereof; performing tissue contour extraction on the tissue image, so as to obtain a target tissue contour; and performing fusion display on the target tissue contour and the original white-light image, so as to obtain a white-light image with the target tissue contour.
Need to check novelty before this filing date? Find Prior Art

Description

Endoscopic image display method, device, terminal device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 19, 2023, with application number 2023105776643, and invention name “Endoscopic image display method, device, terminal equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of image processing, and in particular to an endoscopic image display method, apparatus, terminal device, and storage medium. Background Art

[0004] There are more and more endoscopic tissue imaging methods currently, such as fluorescence imaging, near-infrared imaging, ultraviolet imaging, etc. However, the existing endoscopic imaging display modes are relatively simple and cannot meet clinical needs well.

[0005] Taking fluorescence endoscopes as an example, the related fluorescence endoscope imaging display modes are mainly divided into white light mode, fluorescence mode and fusion mode. However, the following problems often exist:

[0006] The white light mode can only display tissue information, but cannot show the location of fluorescently marked tumor lesions; the fluorescence mode can only display black and white fluorescent images that mark the location of tumor lesions, but cannot display tissue information; the traditional fusion mode is to superimpose and fuse the fluorescence image onto the white light image. Doctors can observe the fluorescently marked lesions in the white light mode, but because the superposition of the fluorescence image covers the original color and structure of tissues such as the epidermis and blood vessels, doctors must constantly switch display modes during the operation to avoid removing important blood vessels and normal tissues, which affects the efficiency of the operation and increases the risk of the operation.

[0007] Summary of the Invention

[0008] According to various embodiments of the present application, a method, apparatus, terminal device, and storage medium for displaying an endoscopic image are provided.

[0009] The present application provides an endoscopic image display method, the endoscopic image display method comprising:

[0010] Acquire the original white light image of the endoscope and the labeled tissue image;

[0011] performing tissue contour extraction on the tissue image to obtain a target tissue contour;

[0012] The target tissue outline is fused with the original white light image to obtain a white light image with the target tissue outline.

[0013] In one embodiment, the step of extracting tissue contours from the tissue image to obtain a target tissue contour includes:

[0014] Processing the tissue image by a binary method to obtain a binary image;

[0015] The contour of the target tissue is obtained by performing contour extraction on the binary image based on an edge finding algorithm.

[0016] In one embodiment, after the step of processing the tissue image by a binarization method to obtain a binarized image, the step further includes:

[0017] Performing smoothing on the binary image to obtain a smoothed binary image;

[0018] The step of extracting the contour of the binary image based on an edge finding algorithm to obtain the contour of the target tissue comprises:

[0019] The contour of the target tissue is obtained by performing contour extraction on the smoothed binary image based on an edge finding algorithm.

[0020] In one embodiment, the step of processing the tissue image by a binary method to obtain a binary image includes:

[0021] Calculate the tissue image using a threshold calculation method to obtain a grayscale threshold;

[0022] Segmenting the tissue image into a target tissue area and a non-target tissue area according to a comparison result between the grayscale of each pixel in the tissue image and the grayscale threshold;

[0023] A binary image is obtained according to the target tissue area and the non-target tissue area.

[0024] In one embodiment, the step of segmenting the tissue image into target tissue areas and non-target tissue areas based on a comparison result between the grayscale of each pixel in the tissue image and the grayscale threshold comprises:

[0025] For each pixel point in the tissue image, if the grayscale of the pixel point is greater than the grayscale threshold, the pixel point is determined as a pixel point of the target tissue area;

[0026] If the grayscale of the pixel point is less than or equal to the grayscale threshold, the pixel point is determined to be a pixel point of a non-target tissue area;

[0027] The target tissue area and the non-target tissue area in the tissue image are determined according to the pixel points of the target tissue area and the pixel points of the non-target tissue area.

[0028] In one embodiment, the step of extracting the tissue contour from the tissue image to obtain the target tissue contour includes:

[0029] performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour;

[0030] The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises:

[0031] The enhanced target tissue outline is fused with the original white light image to obtain the white light image with the target tissue outline.

[0032] In one embodiment, the step of enhancing the target tissue contour to obtain the enhanced target tissue contour comprises:

[0033] Based on the target tissue contour, obtaining a contour area of ​​the target tissue;

[0034] Calculating the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set;

[0035] Calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set;

[0036] Obtaining contour features through contour feature calculation based on the channel contour information pixel set and the tissue contour information pixel set;

[0037] According to the contour features, a coefficient enhancement matrix is ​​calculated by a coefficient enhancement algorithm;

[0038] According to the coefficient enhancement matrix, a nonlinear adjustment coefficient is calculated by a nonlinear adjustment algorithm;

[0039] The enhanced target tissue contour is obtained by calculation based on the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

[0040] In one embodiment, the step of acquiring the original white light image of the endoscope and the marked tissue image comprises:

[0041] Deleting pixels of the original white-light image within the contour area to obtain a new white-light image;

[0042] The step of fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour comprises:

[0043] The enhanced target tissue contour is fused with the new white light image to obtain the white light image with the target tissue contour.

[0044] In one embodiment, the step of obtaining the original white light image and the marked tissue image of the endoscope further includes:

[0045] registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image;

[0046] The step of extracting tissue contours from the tissue image to obtain target tissue contours includes:

[0047] Extracting the tissue contour of the standard tissue image to obtain the target tissue contour;

[0048] The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises:

[0049] The target tissue outline is fused with the standard white light image to obtain a white light image with the target tissue outline.

[0050] The present application also provides an endoscopic image display device, comprising:

[0051] An image acquisition module, used for acquiring the original white light image of the endoscope and the marked tissue image;

[0052] A feature extraction module is used to extract the tissue contour of the tissue image to obtain the target tissue contour;

[0053] The image processing module is used to fuse the target tissue outline with the original white light image for display, so as to obtain a white light image with the target tissue outline.

[0054] In one embodiment, the feature extraction module is specifically used to:

[0055] Processing the tissue image by a binary method to obtain a binary image;

[0056] The contour of the target tissue is obtained by performing contour extraction on the binary image based on an edge finding algorithm.

[0057] In one embodiment, the feature extraction module is specifically used to:

[0058] Performing smoothing on the binary image to obtain a smoothed binary image;

[0059] The contour of the target tissue is obtained by performing contour extraction on the smoothed binary image based on an edge finding algorithm.

[0060] In one embodiment, the feature extraction module is specifically used to:

[0061] Calculate the tissue image using a threshold calculation method to obtain a grayscale threshold;

[0062] Segmenting the tissue image into a target tissue area and a non-target tissue area according to a comparison result between the grayscale of each pixel in the tissue image and the grayscale threshold;

[0063] A binary image is obtained according to the target tissue area and the non-target tissue area.

[0064] In one embodiment, the feature extraction module is specifically used to:

[0065] For each pixel point in the tissue image, if the grayscale of the pixel point is greater than the grayscale threshold, the pixel point is determined as a pixel point of the target tissue area;

[0066] If the grayscale of the pixel point is less than or equal to the grayscale threshold, the pixel point is determined to be a pixel point of a non-target tissue area;

[0067] The target tissue area and the non-target tissue area in the tissue image are determined according to the pixel points of the target tissue area and the pixel points of the non-target tissue area.

[0068] In one embodiment, the endoscopic image display device is further used to:

[0069] performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour;

[0070] The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises:

[0071] The enhanced target tissue outline is fused with the original white light image to obtain the white light image with the target tissue outline.

[0072] In one embodiment, the endoscopic image display device is further used to:

[0073] Based on the target tissue contour, obtaining a contour area of ​​the target tissue;

[0074] Calculating the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set;

[0075] Calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set;

[0076] Obtaining contour features through contour feature calculation based on the channel contour information pixel set and the tissue contour information pixel set;

[0077] According to the contour features, a coefficient enhancement matrix is ​​calculated by a coefficient enhancement algorithm;

[0078] According to the coefficient enhancement matrix, a nonlinear adjustment coefficient is calculated by a nonlinear adjustment algorithm;

[0079] The enhanced target tissue contour is obtained by calculation based on the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

[0080] In one embodiment, the endoscopic image display device is further used to:

[0081] Deleting pixels of the original white-light image within the contour area to obtain a new white-light image;

[0082] The step of fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour comprises:

[0083] The enhanced target tissue contour is fused with the new white light image to obtain the white light image with the target tissue contour.

[0084] In one embodiment, the device is further configured to:

[0085] registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image;

[0086] The step of extracting tissue contours from the tissue image to obtain target tissue contours includes:

[0087] Extracting the tissue contour of the standard tissue image to obtain the target tissue contour;

[0088] The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises:

[0089] The target tissue outline is fused with the standard white light image to obtain a white light image with the target tissue outline.

[0090] An embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and an endoscopic image display program stored in the memory and runnable on the processor. When the endoscopic image display program is executed by the processor, the steps of the endoscopic image display method described above are implemented.

[0091] An embodiment of the present application further provides a computer-readable storage medium, on which an endoscopic image display program is stored. When the endoscopic image display program is executed by a processor, the steps of the endoscopic image display method described above are implemented.

[0092] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] In order to better describe and illustrate the embodiments and / or examples of the inventions disclosed herein, reference may be made to one or more of the accompanying drawings. The additional details or examples used to describe the accompanying drawings should not be considered to limit the scope of the disclosed inventions, the presently described embodiments and / or examples, and any of the best modes currently understood for these inventions.

[0094] FIG1 is a schematic diagram of functional modules of a terminal device to which an endoscopic image display device belongs in one or more embodiments.

[0095] FIG2 is a flow chart of a method for displaying an endoscopic image in one or more embodiments.

[0096] FIG3 is another flowchart of an endoscopic image display method in one or more embodiments.

[0097] FIG4 is another flowchart of an endoscopic image display method in one or more embodiments.

[0098] FIG5 is another flowchart of an endoscopic image display method in one or more embodiments.

[0099] FIG6 is another flowchart of an endoscopic image display method in one or more embodiments.

[0100] FIG. 7 is another flowchart of an endoscopic image display method in one or more embodiments.

[0101] FIG8 is another flowchart of an endoscopic image display method in one or more embodiments.

[0102] FIG9 is a flowchart of an endoscopic image display process involved in one or more embodiments.

[0103] FIG10 is a flow chart of contour extraction involved in one or more embodiments.

[0104] FIG. 11 illustrates a tissue image and a white light image with a target tissue outline according to one or more embodiments.

[0105] FIG. 12 shows an endoscopic image display device according to one or more embodiments.

[0106] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0107] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0108] The main solution of the embodiment of this application includes: obtaining an original white-light image from an endoscope and a marked tissue image; performing tissue contour extraction on the tissue image to obtain a target tissue contour; and fusing the target tissue contour with the original white-light image to obtain a white-light image with the target tissue contour. This allows the target tissue location to be marked in the endoscopic imaging display mode while also displaying tissue information at the marked location, thereby improving surgical efficiency and reducing surgical risks.

[0109] The embodiments of the present application take into account that the relevant technical solutions based on the white light mode can only display tissue information, but cannot display the location of fluorescently marked tumor lesions; and the fluorescence mode can only display black and white fluorescent images that mark the location of tumor lesions, but cannot display tissue information; the traditional fusion mode is to superimpose and fuse the fluorescence image onto the white light image. The doctor can observe the fluorescently marked lesions in the white light mode, but because the superposition of the fluorescence image covers the original color and structure of tissues such as the epidermis and blood vessels, the doctor must constantly switch the display mode during the operation to avoid removing important blood vessels and normal tissues, which affects the efficiency of the operation and increases the risk of the operation.

[0110] Based on this, an embodiment of the present application proposes a solution that can extract the lesion contour in the tissue image and fuse it into the original white light image, thereby marking the target tissue position in the endoscopic imaging display mode while displaying the tissue information of the marked position, thereby improving surgical efficiency and reducing surgical risks.

[0111] In one embodiment, referring to Figure 1, which is a schematic diagram of the functional modules of a terminal device to which an endoscopic image display device belongs in one or more embodiments, the endoscopic image display device can be a device independent of the terminal device and capable of performing data processing, and can be hosted on the terminal device in the form of hardware or software.

[0112] The terminal device may be a medical endoscope, or a fixed terminal device with an endoscope, a camera system, or a server.

[0113] In this embodiment, referring to FIG. 1 , the terminal device to which the endoscopic image display apparatus belongs includes at least an output module 110 , a processor 120 , a memory 130 , and a communication module 140 .

[0114] The memory 130 stores an operating system and an endoscopic image display program. The endoscopic image display device can obtain the original white light image of the endoscope and the marked tissue image, extract the tissue contour of the tissue image to obtain the target tissue contour, fuse the target tissue contour with the original white light image for display, and store the obtained white light image with the target tissue contour in the memory 130. The output module 110 can be a display screen, a speaker, etc. The communication module 140 can include a WiFi module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0115] When the endoscopic image display program in the memory 130 is executed by the processor, the following steps are implemented:

[0116] Acquire the original white light image of the endoscope and the labeled tissue image;

[0117] performing tissue contour extraction on the tissue image to obtain a target tissue contour;

[0118] The target tissue outline is fused with the original white light image to obtain a white light image with the target tissue outline.

[0119] In one embodiment, when the endoscopic image display program in the memory 130 is executed by the processor, the following steps are further implemented:

[0120] Processing the tissue image by a binary method to obtain a binary image;

[0121] The contour of the target tissue is obtained by performing contour extraction on the binary image based on an edge finding algorithm.

[0122] In one embodiment, when the endoscopic image display program in the memory 130 is executed by the processor, the following steps are further implemented:

[0123] Performing smoothing on the binary image to obtain a smoothed binary image;

[0124] The contour of the target tissue is obtained by performing contour extraction on the smoothed binary image based on an edge finding algorithm.

[0125] In one embodiment, when the endoscopic image display program in the memory 130 is executed by the processor, the following steps are further implemented:

[0126] performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour;

[0127] The enhanced target tissue outline is fused with the original white light image to obtain the white light image with the target tissue outline.

[0128] In one embodiment, when the endoscopic image display program in the memory 130 is executed by the processor, the following steps are further implemented:

[0129] Based on the target tissue contour, obtaining a contour area of ​​the target tissue;

[0130] Calculating the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set;

[0131] Calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set;

[0132] Obtaining contour features through contour feature calculation based on the channel contour information pixel set and the tissue contour information pixel set;

[0133] According to the contour features, a coefficient enhancement matrix is ​​calculated by a coefficient enhancement algorithm;

[0134] According to the coefficient enhancement matrix, a nonlinear adjustment coefficient is calculated by a nonlinear adjustment algorithm;

[0135] The enhanced target tissue contour is obtained by calculation based on the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

[0136] In one embodiment, when the endoscopic image display program in the memory 130 is executed by the processor, the following steps are further implemented:

[0137] Deleting pixels of the original white-light image within the contour area to obtain a new white-light image;

[0138] The enhanced target tissue contour is fused with the new white light image to obtain the white light image with the target tissue contour.

[0139] In one embodiment, when the endoscopic image display program in the memory 130 is executed by the processor, the following steps are further implemented:

[0140] registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image;

[0141] Extracting the tissue contour of the standard tissue image to obtain the target tissue contour;

[0142] The target tissue outline is fused with the standard white light image to obtain a white light image with the target tissue outline.

[0143] This embodiment obtains an original white-light image from an endoscope and a marked tissue image; extracts tissue contours from the tissue image to obtain a target tissue contour; and fuses the target tissue contour with the original white-light image for display, thereby obtaining a white-light image with the target tissue contour. The marked tissue image can be a tissue image with the lesion location marked, thereby enabling the lesion location to be marked in the endoscopic imaging display mode while also displaying tissue information at the marked location, thereby improving surgical efficiency and reducing surgical risks.

[0144] Based on the above terminal device architecture but not limited to the above architecture, an embodiment of the method of the present application is proposed.

[0145] In one embodiment, referring to FIG. 2 , FIG. 2 is a flow chart of an endoscopic image display method according to one or more embodiments.

[0146] An embodiment of the present application provides an endoscopic image display method, the method comprising:

[0147] Step S10, acquiring the original white light image of the endoscope and the marked tissue image;

[0148] The system involved in the method of this embodiment may include a medical endoscope and an image processing device, wherein the image processing device may be integrated into the medical endoscope or carried on an image processing terminal device or server.

[0149] In order to achieve the goal of marking the location of target tissue (such as a lesion) in the endoscopic imaging display mode while displaying the tissue information at the marked location, thereby improving surgical efficiency and reducing surgical risks, this embodiment adopts a contour and image fusion display method. Specifically, the original white light image with tissue information and the tissue image with the target tissue marked are first obtained. Then, the contour of the target tissue in the tissue image is extracted to obtain the target tissue contour. Finally, the target tissue contour is fused with the original white light image to obtain a white light image with the target tissue contour.

[0150] Among them, as an implementation method, the target tissue can be a lesion tissue. Taking the lesion as an example, a tissue image with the lesion tissue marked can be obtained first, the lesion outline can be extracted from the tissue image, and then the lesion outline can be fused into the original white light image to obtain a white light image with the lesion outline, so as to achieve marking the lesion position while displaying the tissue information of the marked position, thereby improving surgical efficiency and reducing surgical risks.

[0151] In one embodiment, a white light endoscope can be used to obtain original white light images with tissue information based on the visible light spectrum of 400-700nm. The original white light images can observe tissue information inside the human body, such as mucosal color, mucosal status, vascular texture, etc., which is very helpful in providing clear images of the human body during surgery.

[0152] In this embodiment, the tissue image includes one or more of the following images: a fluorescence image, a near-infrared image, an ultraviolet image, and of course, other tissue images may also be included. In one embodiment, taking the tissue image as a fluorescence image as an example, the fluorescence image can be obtained through a fluorescence endoscope. Specifically, a fluorescent developer is injected into the patient's body, and then a fluorescence camera is used to capture the fluorescence signal to mark the lesion tissue, thereby obtaining a tissue image with the lesion location marked, which plays a key role in accurate intraoperative positioning and reducing surgical risks. The acquisition of the fluorescence image and the acquisition of the white light image can be performed simultaneously, and only the hardware of the fluorescence endoscope and the white light endoscope need to be integrated.

[0153] Due to the limitations of sensitivity, specificity, and detection rate, raw white light images cannot mark the location of lesions and thus help doctors clearly identify malignant tissues such as tumors. Labeled tissue images also have limitations. They can only distinguish between normal and lesioned tissue in black and white. Because black and white tissue images obscure the original color and structure of tissues such as the epidermis and blood vessels, they cannot reveal tissue information.

[0154] Both raw white-light images and tissue images have their own drawbacks. Even in traditional fusion modes, where fluorescence images are superimposed on raw white-light images, these drawbacks cannot be avoided. Doctors need to obtain different effective information from multiple images separately, which necessitates frequent image switching during surgery, causing significant inconvenience, reducing surgical efficiency, and increasing surgical risk. To avoid this, this embodiment proposes acquiring raw white-light images and tissue images, followed by further contour extraction and fusion processing. This allows for simultaneous acquisition of effective information from both images while avoiding the drawbacks of raw white-light and tissue images. This allows for simultaneous marking of the lesion location and display of tissue information at the marked location, improving surgical efficiency and reducing surgical risk.

[0155] Step S30, extracting the tissue contour of the tissue image to obtain the target tissue contour;

[0156] As previously mentioned, in traditional fusion methods, the fluorescence image is superimposed on the original white-light image, forcing the surgeon to obtain different valid information from each of the multiple images. This requires the surgeon to frequently switch images during surgery, which is inconvenient, reduces surgical efficiency, and increases surgical risk. To avoid this, this embodiment proposes to obtain the original white-light image and tissue image, then perform further contour extraction and fusion processing to simultaneously obtain valid information from both images while avoiding the drawbacks of the original white-light image and tissue image.

[0157] In one embodiment, the contour of the target tissue in the tissue image can be extracted by first obtaining the position and shape information of the target tissue from the tissue image, and then obtaining the contour of the target tissue. Thus, by extracting the contour of the target tissue in the tissue image, the target tissue contour occupies fewer pixels to express the position and shape of the target tissue, leaving more pixels to express the original color and structure of tissues such as the epidermis and blood vessels. This allows for the acquisition of effective information such as the position and shape of the target tissue in the tissue image, while avoiding the drawback of the tissue image overwriting the original color and structure of tissues such as the epidermis and blood vessels.

[0158] In some examples, the tissue image can be segmented into a target tissue region and a non-target tissue region by image segmentation, and then the edges of the target tissue region and the non-target tissue region can be obtained by an edge detection algorithm to obtain the target tissue contour.

[0159] In one embodiment, the target tissue may be lesion tissue, and the non-target tissue may be normal tissue. In some tissue images, such as fluorescence images, lesion tissue and normal tissue appear at different grayscale levels, with lesion tissue typically being brighter than normal tissue. Computer image processing can utilize the grayscale information of the image to represent lesion tissue as white and normal tissue as black, thereby segmenting the lesion tissue region and the normal tissue region.

[0160] As another image segmentation method, with the continuous development of deep learning technology, neural networks and other methods can also be used to automatically identify tissue images and segment the lesion tissue area and normal tissue area, thereby improving analysis efficiency and accuracy.

[0161] Then, using an edge detection algorithm, for example, the edge between the white area (lesion tissue) and the black area (normal tissue) in the fluorescence image can be detected. The pixel set of this edge in the tissue image is extracted to obtain the lesion tissue outline, i.e., the target tissue outline. The target tissue location can then be marked using the target tissue outline instead of the white area. Because the target tissue outline occupies relatively few pixels in the image, it does not affect the doctor's ability to observe the original color and structure of the lesion tissue's epidermis, blood vessels, and other tissues.

[0162] This embodiment extracts the tissue contour from the tissue image to obtain the target tissue contour, which can express the position and shape of the lesion tissue in the image with the target tissue contour occupying fewer pixels, thereby avoiding the disadvantage that the tissue image covers the original color and structure of tissues such as the epidermis and blood vessels, and can effectively help doctors analyze tissue images and diagnose diseases.

[0163] Step S50 , fusing the target tissue outline with the original white light image for display, to obtain a white light image with the target tissue outline.

[0164] In order to enable the information on the position and shape of the target tissue in the target tissue contour to be displayed on the same screen as the tissue information in the original white light image, the target tissue contour can be fused with the original white light image for display, thereby achieving the effect of information fusion. It should be noted that the method of displaying the target tissue contour on the original white light image in the embodiment of the present application is different from some traditional technologies that identify the target tissue therein and display the target tissue contour by performing image analysis or spectral analysis on the white light image. Compared with such traditional technologies, the embodiment of the present application does not need to obtain the target tissue contour through complex computational analysis and sample analysis capabilities. The target tissue contour can be obtained based on, for example, the fluorescent image of a fluorescent endoscope, thereby improving processing efficiency and accuracy.

[0165] As an implementation method, the target tissue outline can be fused onto the original white light image through image synthesis or image fusion technology to obtain a white light image with the target tissue outline. Because the target tissue outline occupies relatively few pixels in the image, more pixels in the white light image with the target tissue outline can express tissue information, such as the original color and structure of the target tissue's epidermis, blood vessels, and other tissues. Therefore, this image display method can express the location and shape of the target tissue while not covering the target tissue's color and structure. This provides a more comprehensive and clear display of the target tissue and its surrounding tissues, making it easier for doctors to identify the target tissue.

[0166] The endoscopic image display method proposed in the embodiments of the present application obtains an original white-light image from an endoscope and a marked tissue image; extracts tissue contours from the tissue image to obtain a target tissue contour; and fuses the target tissue contour with the original white-light image for display, thereby obtaining a white-light image with the target tissue contour. This method marks the target tissue location in the endoscopic imaging display mode while displaying tissue information at the marked location, thereby improving surgical efficiency and reducing surgical risks.

[0167] The target tissue can be a lesion tissue. Specifically, the lesion outline in the tissue image can be extracted and fused into the original white light image, so as to mark the lesion position in the endoscopic imaging display mode while displaying the tissue information of the marked position, thereby improving surgical efficiency and reducing surgical risks.

[0168] In one embodiment, referring to FIG. 3 , FIG. 3 is another flowchart of an endoscopic image display method in one or more embodiments.

[0169] The difference between this embodiment and the previous embodiment is that the step of extracting the tissue contour from the tissue image to obtain the target tissue contour in step S30 is refined, wherein step S30 may include:

[0170] In this embodiment, step S30, extracting the tissue contour from the tissue image to obtain the target tissue contour includes:

[0171] Step S31, processing the tissue image by a binary method to obtain a binary image;

[0172] In order to remove redundant information of tissue images and maximize the distinction between target tissue areas and non-target tissue areas, the tissue images can be processed by binary method to obtain a binary image.

[0173] Binarization is a method of image segmentation that simplifies or alters the image's representation, making it easier to analyze. Binarization works by classifying pixels in an image into two categories: black and white. Pixels with a grayscale value greater than a preset threshold are set to white, while pixels with grayscale values ​​lower than a threshold are set to black. After completing this determination for all pixels in the image, a binary image consisting of black and white is obtained.

[0174] In some embodiments, a threshold calculation method can be used to calculate a tissue image to obtain a grayscale threshold. The grayscale of each pixel in the tissue image is then compared with the grayscale threshold to segment the image into a target tissue region and a non-target tissue region. Pixels with a grayscale greater than the grayscale threshold are considered to be pixels in the target tissue region, while pixels with a grayscale less than or equal to the grayscale threshold are considered to be pixels in the non-target tissue region. This operation is repeated for all pixels in the tissue image to segment the image into the target tissue region and the non-target tissue region, thereby determining a binary image.

[0175] Among them, the specific formula of the binary method is:

[0176] Where: T is the grayscale threshold; F(x, y) is the result of the binarized image, (x, y) is the pixel coordinate of the image; 1 is the target tissue area; 0 is the non-target tissue area; f(x, y) is the grayscale of the pixel point of the tissue image.

[0177] After the tissue image is binarized, the information of the tissue image is greatly simplified, resulting in a binary image. The binary image consists only of the target tissue area and the non-target tissue area. The edge of the target tissue area and the non-target tissue area is the target tissue contour, which provides the basis for further target tissue contour extraction.

[0178] Step S33 , performing contour extraction on the binary image based on an edge finding algorithm to obtain the target tissue contour.

[0179] In order to mark the target tissue position and display the tissue information of the marked position at the same time, the edge search algorithm can be used to extract the target tissue contour that occupies fewer image pixels to replace the target tissue area that occupies more image pixels, thereby expressing the position and shape of the target tissue, so that more pixels can be used to express the original tissue information.

[0180] Among them, the edge search algorithm is an algorithm that finds a set of pixels with contrasting grayscale intensities around them to determine the edge.

[0181] In some embodiments, the grayscale intensity around the pixel point is compared with a preset threshold through specific rules. If the grayscale intensity around the pixel point is greater than the preset threshold, the pixel point belongs to a set of pixels with contrasting grayscale changes around it, and the set is the edge pixel set, thereby obtaining the edge.

[0182] For example, the Sobel edge detection method can be used to extract the target tissue contour from the binary image. The Sobel operator includes two sets of 3x3 matrices, namely the X-direction matrix and the Y-direction matrix. The specific Sobel contour extraction operator is as follows:

[0183] The above is the X-direction matrix;

[0184] The above is the Y direction matrix.

[0185] In one example, a pixel in the target tissue area of ​​a binary image is first found, and then its eight neighboring pixels are searched. The grayscales of the eight neighboring pixels of the pixel are input into the X-direction matrix calculation of the Sobel operator and into the Y-direction matrix calculation to obtain the X-direction gradient value and the Y-direction gradient value of the pixel. The square of the X-direction gradient value of the pixel and the square of the Y-direction gradient value of the pixel are added together and then squared to obtain the total gradient of the pixel. The total gradient of the pixel reflects the degree of grayscale contrast of the pixels around the pixel. The total gradient of the pixel is then compared with a pre-set edge threshold. If the total gradient of the pixel is greater than the pre-set edge threshold, the pixel is determined to be a pixel of the target tissue contour and is added to the pixel set of the target tissue contour. The above operation is repeated for the pixel points of the target tissue area in the binary image to obtain the pixel set of the target tissue contour, thereby determining the target tissue contour.

[0186] The endoscopic image display method proposed in the embodiment of the present application processes the tissue image using a binarization method to obtain a binarized image; and extracts the contour of the binarized image based on an edge finding algorithm to obtain the target tissue contour. Specifically, the tissue image is divided into a target tissue region and a non-target tissue region using the binarization method to obtain a binarized image. The target tissue contour is then extracted using the Sobel edge finding method, thereby achieving the effect of expressing the position and shape of the target tissue in the image using the target tissue contour with fewer pixels. This avoids the drawback of the tissue image covering the original color and structure of tissues such as the epidermis and blood vessels, and can effectively help doctors analyze tissue images and diagnose diseases.

[0187] 4 , which is another flowchart of an endoscopic image display method in one or more embodiments.

[0188] The difference between this embodiment and the above embodiment is that: in step S31, the step of processing the tissue image by a binary method to obtain a binary image is supplemented, wherein the supplementing step may include:

[0189] In this embodiment, after the step S31 of processing the tissue image by a binary method to obtain a binary image, the following steps are further included:

[0190] Step S32, performing smoothing processing on the binary image to obtain a smoothed binary image;

[0191] Step S33, performing contour extraction on the binary image based on an edge finding algorithm to obtain the target tissue contour includes:

[0192] Step S331 : performing contour extraction on the smoothed binary image based on an edge finding algorithm to obtain the target tissue contour.

[0193] To further improve the quality and accuracy of the binarized image, smoothing is performed on the image. Smoothing involves blurring the pixels in the image to reduce noise and irregularities. Common image filters, such as Gaussian filters or median filters, can be used to smooth the image and produce a smoothed binarized image.

[0194] In one embodiment, a shape-based opening operation using a circular structuring element can be used to perform an erosion followed by a dilation operation on the binary image. This can smoothen the edges between the segmented target tissue region and the non-target tissue region without increasing the area, remove potential noise, and obtain a smooth binary image with clearer edges. The specific formula for the shape-based opening operation is as follows:

[0195] Corrosion is defined using the point set formula

[0196] Where: X is a single-channel image; S is the structural element; x is the pixel point through which the structural element passes.

[0197] Expansion is defined using the point set formula

[0198] Where: X is a single-channel image; S is the structural element; x is the pixel point through which the structural element passes.

[0199] In another embodiment, contour extraction may be performed on the smoothed binary image to obtain a target tissue contour with a clearer and smoother contour.

[0200] The endoscopic image display method proposed in an embodiment of the present application smoothes the binary image to obtain a smoothed binary image; and then extracts the contour of the smoothed binary image based on an edge-finding algorithm to obtain the contour of the target tissue. Specifically, the present application smoothes the binary image to effectively eliminate noise and irregular areas in the image, making the edges of the target tissue area and non-target tissue area smoother and clearer, thereby optimizing the contour of the target tissue, which can help doctors analyze images and diagnose conditions more clearly and accurately.

[0201] 5 , which is another flowchart of an endoscopic image display method in one or more embodiments.

[0202] The difference between this embodiment and the above-mentioned embodiment is that: in step S30, the tissue contour is extracted from the tissue image, and further limitation is performed after the step of obtaining the target tissue contour.

[0203] In this embodiment, after step S30 of extracting tissue contours from the tissue image to obtain the target tissue contour, the following steps are further included:

[0204] Step S40, performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour;

[0205] Step S50, fusing the target tissue outline with the original white light image for display, and obtaining a white light image with the target tissue outline includes:

[0206] Step S51 : fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour.

[0207] Considering that when the target tissue outline is fused onto the original white light image, it may be mixed with the internal human tissue information in the original white light image, resulting in unclear target tissue outline. In order to present the target tissue outline more clearly and improve the distinction between the target tissue outline and the original white light image, it is necessary to enhance the target tissue outline so that the target tissue outline is more recognizable against the background of internal human tissue.

[0208] In one embodiment, this can be achieved by optimizing the hue, brightness, and saturation of the pixels of the target tissue outline. For example, against a darker background of internal human tissue, the target tissue outline can be transformed into a collection of medium-brightness pixels through enhancement processing; or against a redder background of internal human tissue, the target tissue outline can be transformed into a collection of green pixels. Depending on the environment, the enhanced processing produces different target tissue outlines, ensuring that the enhanced target tissue outline remains clearly visible against a variety of internal human tissue backgrounds.

[0209] In another embodiment, the enhanced target tissue contour can be fused into the original white light image through image synthesis or image fusion technology, so that the doctor can see the enhanced target tissue contour more clearly, which is convenient for the doctor to quickly identify the target tissue, thereby effectively improving the efficiency of the operation.

[0210] The endoscopic image display method proposed in the embodiment of the present application enhances the target tissue contour to obtain an enhanced target tissue contour; and fuses the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour. Specifically, by enhancing the hue, saturation, and brightness of the pixels of the target tissue contour, the enhanced target tissue contour can be clearly presented against the background of various internal tissues of the human body. The doctor can clearly see the enhanced target tissue contour, which facilitates the doctor to quickly identify the target tissue, thereby effectively improving the efficiency of the operation.

[0211] 6 , which is another flowchart of an endoscopic image display method in one or more embodiments.

[0212] The difference between this embodiment and the previous embodiment is that the step of enhancing the target tissue contour to obtain the enhanced target tissue contour in step S40 is refined, wherein step S40 may include:

[0213] In this embodiment, step S40, performing enhancement processing on the target tissue contour to obtain the enhanced target tissue contour includes:

[0214] Step S41, obtaining a contour area of ​​the target tissue based on the target tissue contour;

[0215] Considering that when the target tissue outline is fused onto the original white light image, it may be mixed with the internal human tissue information in the original white light image, resulting in unclear target tissue outline. In order to present the target tissue outline more clearly and improve the distinction between the target tissue outline and the original white light image, it is necessary to enhance the target tissue outline. Specifically, the hue, saturation, and brightness of the target tissue outline can be changed to make the target tissue outline more recognizable against the background of internal human tissue.

[0216] Through the target tissue contour, a pixel point set of the target tissue contour, ie, the contour area, is obtained.

[0217] Step S42, calculating the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set;

[0218] Step S43, calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set;

[0219] The target color is the color presented after the target tissue contour is enhanced. Specifically, the target color can be green. Based on the contour area, all pixel points of the green channel of the original white light image in the contour area are calculated, that is, the channel contour pixel set.

[0220] In one example, based on the contour area, for a tissue image, all pixel points in the contour area of ​​the tissue image, ie, a tissue contour pixel set, are calculated.

[0221] The specific formula is:

[0222] in:

[0223] C is the contour area; (x, y) is the pixel coordinate in the image; C I_g (x, y) is the channel contour pixel set; C FI (x, y) is the tissue contour pixel set; I_g(x, y) is the green channel pixel set of the original white light image; FI(x, y) is the tissue image pixel set.

[0224] Step S44, obtaining contour features through contour feature calculation based on the channel contour pixel set and the tissue contour pixel set;

[0225] The absolute value of the difference between the tissue contour pixel set and the channel contour pixel set is calculated to obtain the contour feature. The specific formula for contour feature calculation is:

[0226] Where: D(x,y) is the contour feature.

[0227] Step S45, calculating and obtaining a coefficient enhancement matrix using a coefficient enhancement algorithm according to the contour features;

[0228] The specific formula of the coefficient enhancement algorithm is:

[0229] Wherein: max(D(x,y)) is the maximum value of the matrix; P(x,y) is the coefficient enhancement matrix; the coefficient enhancement matrix can reflect the strength characteristics of the target tissue contour signal.

[0230] Step S46, calculating a nonlinear adjustment coefficient using a nonlinear adjustment algorithm according to the coefficient enhancement matrix;

[0231] The specific formula of the nonlinear adjustment algorithm is:

[0232] Where: λ is the weight gain coefficient, and the value of λ is in [0,∞); W (x,y) is the nonlinear adjustment coefficient.

[0233] After nonlinear adjustment, it provides a basis for subsequently displaying the enhanced target tissue contour at maximum brightness without losing information.

[0234] Step S47 , calculating the enhanced target tissue contour according to the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

[0235] The specific formula of the contour enhancement algorithm is: C fusion =W·C I_g +C FI (1-W)

[0236] Where: W is the nonlinear adjustment coefficient; C fusion To enhance the target tissue outline after processing.

[0237] The target tissue outline after enhancement processing can be displayed in green and at maximum brightness without losing information.

[0238] The endoscopic image display method proposed in the embodiment of the present application obtains the contour area of ​​the target tissue based on the contour of the target tissue; calculates the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set; calculates the pixels of the tissue image in the contour area to obtain a tissue contour pixel set; obtains contour features through contour feature operation based on the channel contour information pixel set and the tissue contour information pixel set; calculates a coefficient enhancement matrix based on the contour features through a coefficient enhancement algorithm; calculates a nonlinear adjustment coefficient based on the coefficient enhancement matrix through a nonlinear adjustment algorithm; calculates the enhanced target tissue contour based on the nonlinear adjustment coefficient, the channel contour pixel set and the tissue contour pixel set, and the contour enhancement algorithm. Specifically, the channel contour pixel set and the tissue contour pixel set are obtained through the target tissue contour, and the nonlinear adjustment parameters are obtained by the contour feature operation, the coefficient enhancement algorithm and the nonlinear adjustment algorithm to achieve the maximum brightness display of the target tissue contour without losing information. Finally, the channel contour pixel set, the tissue contour pixel set and the nonlinear adjustment parameters are combined to obtain the target tissue contour displayed in green and at the maximum brightness, that is, the enhanced target tissue contour. This makes the outline of the target tissue clearer, making it easier for doctors to analyze images and diagnose diseases.

[0239] 7 , which is another flowchart of an endoscopic image display method in one or more embodiments.

[0240] The difference between this embodiment and the above-mentioned embodiment is that: step S10 is further limited after the original white light image of the endoscope and the marked tissue image are acquired.

[0241] In this embodiment, step S10, after acquiring the original white light image and the marked tissue image of the endoscope, includes:

[0242] Step S48, deleting the pixels of the original white light image within the contour area to obtain a new white light image;

[0243] Taking into account the pixels belonging to the contour area in the original white light image, when the target tissue contour is fused into the original white light image for display, they are overlapped with the pixels of the target tissue contour, which may affect the presentation effect of the target tissue contour fused into the original white light image. Therefore, the pixels belonging to the contour area in the original white light image are deleted to obtain a new white light image, so that the target tissue contour can be clearly presented when it is fused into the new white light image.

[0244] For example, the formula for deleting pixels belonging to the contour area in the original white light image can be: I_b(x,y)=0,if I(x,y)∈C I_r(x,y)=0,if I(x,y)∈C I_g(x,y)=0,if I(x,y)∈C I_n=I_b+I_r+I_g

[0245] Where: I_b(x,y) is the blue channel pixel set of the original white light image; I_r(x,y) is the red channel pixel set of the original white light image; I_g(x,y) is the green channel pixel set of the original white light image; C is the contour area; I_n is the new white light image.

[0246] Step S51, fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour, includes:

[0247] Step S511 , fusing the enhanced target tissue contour with the new white light image to obtain the white light image with the target tissue contour.

[0248] Through image fusion technology, the enhanced target tissue contour and the new white light image are fused and displayed. The specific image fusion formula is: I_f=I_n+C fusion ;

[0249] Where: C fusion is the target tissue contour after enhancement processing; I_n is the new white light image, and I_f is the white light image with the target tissue contour.

[0250] The endoscopic image display method proposed in an embodiment of the present application deletes pixels within the contour region of the original white-light image to obtain a new white-light image; then fuses the enhanced target tissue contour with the new white-light image for display, thereby obtaining the white-light image with the target tissue contour. Specifically, the pixels within the contour region of the original white-light image are deleted to obtain a new white-light image; then, the enhanced target tissue contour is fused onto the new white-light image for display, thereby obtaining the white-light image with the target tissue contour. This makes the fused target tissue contour clearer, making it easier for doctors to analyze the image and diagnose diseases.

[0251] 8 , which is another flowchart of an endoscopic image display method in one or more embodiments.

[0252] The difference between this embodiment and the above-mentioned embodiment is that: step S10 is further limited after the original white light image of the endoscope and the marked tissue image are acquired.

[0253] In this embodiment, after step S10 of acquiring the original white light image and the marked tissue image of the endoscope, the following steps are further included:

[0254] Step S20, registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image;

[0255] Step S30, extracting the tissue contour from the tissue image to obtain the target tissue contour includes:

[0256] Step S311, extracting tissue contours from the standard tissue image to obtain target tissue contours;

[0257] Step S50, fusing the target tissue outline with the original white light image to obtain a white light image with the target tissue outline, includes:

[0258] Step S5111 , fusing the target tissue outline with the standard white light image for display, to obtain a white light image with the target tissue outline.

[0259] Considering that the original white light image and the tissue image come from different acquisition methods, may be taken at different times, different shooting angles, etc., there is a certain misalignment error between the original white light image and the tissue image. In order to improve the subsequent fusion effect of the original white light image and the target tissue contour extracted from the tissue image, the original white light image and the tissue image need to be registered.

[0260] Among them, registration is a way of mapping one image to another image through a spatial transformation, and matching the points in the two images that correspond to the same position in space one by one.

[0261] In one embodiment, the original white light image and the tissue image are registered so that points corresponding to the same spatial position in the original white light image and the tissue image are aligned one by one, so as to achieve consistency in the specifications of the two images and obtain a standard white light image and a standard tissue image.

[0262] In one embodiment, if the bit depths of the two images differ, the high-bit-depth image is converted to the specifications of a low-bit-depth image to ensure that the color depths of the two images are consistent. For example, when obtaining a standard white light image and a standard tissue image, if the image bit depths differ, a high-bit-depth image and a low-bit-depth image can be determined from the standard white light image and the standard tissue image, and the determined high-bit-depth image is converted to the specifications of a low-bit-depth image to ensure that the color depths of the standard white light image and the standard tissue image are consistent.

[0263] In one embodiment, tissue contour extraction is performed on the standard tissue image to obtain the target tissue contour. In another embodiment, the target tissue contour can be fused with the standard white light image using image fusion technology to obtain a white light image with the target tissue contour.

[0264] In one embodiment, the target tissue contour may be enhanced first, and then the enhanced target tissue contour may be fused with the standard white light image through image fusion technology to obtain a white light image with the target tissue contour.

[0265] The endoscopic image display method proposed in the embodiment of the present application obtains a standard white light image and a standard tissue image by aligning the original white light image and the tissue image; extracting the tissue contour of the standard tissue image to obtain the target tissue contour; and fusing the target tissue contour with the standard white light image for display to obtain a white light image with the target tissue contour. Specifically, by aligning the original white light image and the tissue image, the original white light image and the tissue image are spatially aligned and the error is reduced. Then, the tissue contour of the standardized tissue image is extracted to obtain the target tissue contour, and then the target tissue contour is fused with the standardized white light image for display to obtain a white light image with the target tissue contour. This reduces the error generated when acquiring the original white light image and the tissue image, making the result of fusing the target tissue contour with the original white light image more accurate, and obtaining a more accurate white light image with the target tissue contour, thereby improving the accuracy of the doctor's surgery.

[0266] An endoscopic imaging flowchart involved in an embodiment of the present application can be shown in Figure 9.

[0267] A contour extraction flow chart involved in an embodiment of the present application may be shown in FIG10 .

[0268] The tissue image and the white light image with the target tissue outline in the embodiment of the present application are shown in FIG11 .

[0269] It should be understood that, although the various steps in the flow charts shown in Figures 2 to 10 are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be performed in other orders. Moreover, for any of the flow charts in Figures 2 to 10, at least a portion of the steps may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0270] In addition, an embodiment of the present application further provides an endoscopic image display device, as shown in FIG12 , comprising:

[0271] An image acquisition module 1201 is used to acquire the original white light image and the marked tissue image of the endoscope;

[0272] A feature extraction module 1202 is configured to extract tissue contours from the tissue image to obtain a target tissue contour;

[0273] The image processing module 1203 is configured to fuse the target tissue outline with the original white light image for display, thereby obtaining a white light image with the target tissue outline.

[0274] In one embodiment, the feature extraction module 1202 is specifically configured to:

[0275] Processing the tissue image by a binary method to obtain a binary image;

[0276] The contour of the target tissue is obtained by performing contour extraction on the binary image based on an edge finding algorithm.

[0277] In one embodiment, the feature extraction module 1202 is specifically configured to:

[0278] Performing smoothing on the binary image to obtain a smoothed binary image;

[0279] The contour of the target tissue is obtained by performing contour extraction on the smoothed binary image based on an edge finding algorithm.

[0280] In one embodiment, the feature extraction module 1202 is specifically configured to:

[0281] Calculate the tissue image using a threshold calculation method to obtain a grayscale threshold;

[0282] Segmenting the tissue image into a target tissue area and a non-target tissue area according to a comparison result between the grayscale of each pixel in the tissue image and the grayscale threshold;

[0283] A binary image is obtained according to the target tissue area and the non-target tissue area.

[0284] In one embodiment, the feature extraction module 1202 is specifically configured to:

[0285] For each pixel point in the tissue image, if the grayscale of the pixel point is greater than the grayscale threshold, the pixel point is determined as a pixel point of the target tissue area;

[0286] If the grayscale of the pixel point is less than or equal to the grayscale threshold, the pixel point is determined to be a pixel point of a non-target tissue area;

[0287] The target tissue area and the non-target tissue area in the tissue image are determined according to the pixel points of the target tissue area and the pixel points of the non-target tissue area.

[0288] In one embodiment, the endoscopic image display device is further used for:

[0289] performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour;

[0290] The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises:

[0291] The enhanced target tissue outline is fused with the original white light image to obtain the white light image with the target tissue outline.

[0292] In one embodiment, the endoscopic image display device is further used for:

[0293] Based on the target tissue contour, obtaining a contour area of ​​the target tissue;

[0294] Calculating the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set;

[0295] Calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set;

[0296] Obtaining contour features through contour feature calculation based on the channel contour information pixel set and the tissue contour information pixel set;

[0297] According to the contour features, a coefficient enhancement matrix is ​​calculated by a coefficient enhancement algorithm;

[0298] According to the coefficient enhancement matrix, a nonlinear adjustment coefficient is calculated by a nonlinear adjustment algorithm;

[0299] The enhanced target tissue contour is obtained by calculation based on the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

[0300] In one embodiment, the endoscopic image display device is further used for:

[0301] Deleting pixels of the original white-light image within the contour area to obtain a new white-light image;

[0302] The step of fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour comprises:

[0303] The enhanced target tissue contour is fused with the new white light image to obtain the white light image with the target tissue contour.

[0304] In one embodiment, the device is further configured to:

[0305] registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image;

[0306] The step of extracting tissue contours from the tissue image to obtain target tissue contours includes:

[0307] Extracting the tissue contour of the standard tissue image to obtain the target tissue contour;

[0308] The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises:

[0309] The target tissue outline is fused with the standard white light image to obtain a white light image with the target tissue outline.

[0310] The principle and implementation process of the endoscopic image display in this embodiment can be referred to the above embodiments and will not be described in detail here.

[0311] Each module in the above-mentioned endoscopic image display device can be implemented in whole or in part through software, hardware, or a combination thereof. The network interface can be an Ethernet card or a wireless network card, for example. Each of the above-mentioned modules can be embedded in or independent of the server's processor in hardware form, or can be stored in the server's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0312] As used in this application, the terms "component," "module," and "system" are intended to refer to a computer-related entity, which can be hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable code, a thread of execution, a program, and / or a computer. As an illustration, both an application running on a server and a server can be a component. One or more components can reside in a process and / or a thread of execution, and a component can be located within a computer and / or distributed between two or more computers.

[0313] In addition, an embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and an endoscopic image display program stored on the memory and runnable on the processor. When the endoscopic image display program is executed by the processor, the steps of the endoscopic image display method described above are implemented.

[0314] Since the endoscopic image display program adopts all the technical solutions of all the aforementioned embodiments when executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0315] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which an endoscopic image display program is stored. When the endoscopic image display program is executed by a processor, the steps of the endoscopic image display method described above are implemented.

[0316] Since the endoscopic image display program adopts all the technical solutions of all the aforementioned embodiments when executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0317] Compared to conventional technologies, the endoscopic image display method, apparatus, terminal device, and storage medium proposed in the embodiments of this application obtain an original white-light image from an endoscope and a marked tissue image; perform tissue contour extraction on the tissue image to obtain a target tissue contour; and fuse the target tissue contour with the original white-light image for display, resulting in a white-light image with the target tissue contour. This allows the lesion location to be marked in the endoscopic imaging display mode while also displaying tissue information at the marked location, thereby improving surgical efficiency and reducing surgical risks.

[0318] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0319] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0320] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the traditional technology can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the method of each embodiment of the present application.

[0321] The above are merely exemplary embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structural or equivalent process transformations made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An endoscopic image display method, the endoscopic image display method comprising: Acquire the original white light image of the endoscope and the labeled tissue image; Extracting tissue contours from the tissue image to obtain a target tissue contour; The target tissue outline is fused with the original white light image and displayed to obtain a white light image with the target tissue outline.

2. The endoscopic image display method according to claim 1, wherein the step of extracting tissue contours from the tissue image to obtain the target tissue contour comprises: Processing the tissue image by a binary method to obtain a binary image; The contour of the binary image is extracted based on an edge finding algorithm to obtain the contour of the target tissue.

3. The method for displaying an endoscopic image according to claim 2, further comprising: Performing smoothing on the binary image to obtain a smoothed binary image; The step of extracting the contour of the binary image based on an edge finding algorithm to obtain the contour of the target tissue comprises: The contour of the target tissue is obtained by performing contour extraction on the smoothed binary image based on an edge finding algorithm.

4. The endoscopic image display method according to claim 2 or 3, wherein the step of processing the tissue image by a binary method to obtain a binary image comprises: Calculate the tissue image using a threshold calculation method to obtain a grayscale threshold; According to the comparison result of the grayscale of each pixel point in the tissue image and the grayscale threshold, the tissue image is segmented into a target tissue area and a non-target tissue area; A binary image is obtained according to the target tissue area and the non-target tissue area.

5. The endoscopic image display method according to claim 4, wherein the step of segmenting the tissue image into a target tissue area and a non-target tissue area according to a comparison result between the grayscale of each pixel in the tissue image and the grayscale threshold comprises: For each pixel point in the tissue image, if the grayscale of the pixel point is greater than the grayscale threshold, the pixel point is determined as a pixel point of the target tissue area; If the grayscale of the pixel point is less than or equal to the grayscale threshold, the pixel point is determined as a pixel point of the non-target tissue area; The target tissue area and the non-target tissue area in the tissue image are determined according to the pixel points of the target tissue area and the pixel points of the non-target tissue area.

6. The endoscopic image display method according to any one of claims 1 to 5, wherein after the step of extracting tissue contours from the tissue image to obtain the target tissue contour, the step further comprises: Performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour; The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises: The enhanced target tissue contour is fused with the original white light image to obtain the white light image with the target tissue contour.

7. The endoscopic image display method according to claim 6, wherein the step of enhancing the target tissue contour to obtain the enhanced target tissue contour comprises: Based on the target tissue contour, obtaining a contour area of ​​the target tissue; Calculate the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set; Calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set; According to the channel contour information pixel set and the tissue contour information pixel set, through contour feature calculation, Get contour features; According to the contour features, a coefficient enhancement matrix is ​​calculated by a coefficient enhancement algorithm; According to the coefficient enhancement matrix, a nonlinear adjustment coefficient is calculated by a nonlinear adjustment algorithm; The enhanced target tissue contour is calculated based on the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

8. The method for displaying endoscopic images according to claim 7, further comprising: after the step of acquiring the original white light image of the endoscope and the marked tissue image: Deleting pixels of the original white-light image within the contour area to obtain a new white-light image; The step of fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour comprises: The enhanced target tissue contour is fused with the new white light image to obtain the white light image with the target tissue contour.

9. The method for displaying an endoscopic image according to any one of claims 1 to 8, further comprising: Registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image; The step of extracting tissue contours from the tissue image to obtain the target tissue contour comprises: Extracting tissue contours from the standard tissue image to obtain target tissue contours; The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises: The target tissue contour is fused with the standard white light image and displayed to obtain a white light image with the target tissue contour.

10. An endoscope image display device, characterized in that: The endoscopic image display device comprises: An image acquisition module, used for acquiring the original white light image of the endoscope and the marked tissue image; A feature extraction module, used to extract tissue contours from the tissue image to obtain a target tissue contour; The image processing module is used to fuse the target tissue outline with the original white light image for display, so as to obtain a white light image with the target tissue outline.

11. The endoscopic image display device according to claim 10, wherein the feature extraction module is specifically used for: Processing the tissue image by a binary method to obtain a binary image; The contour of the binary image is extracted based on an edge finding algorithm to obtain the contour of the target tissue.

12. The endoscopic image display device according to claim 11, wherein the feature extraction module is specifically used for: Performing smoothing on the binary image to obtain a smoothed binary image; The contour of the target tissue is obtained by performing contour extraction on the smoothed binary image based on an edge finding algorithm.

13. The endoscopic image display device according to claim 11 or 12, wherein the feature extraction module is specifically used for: Calculate the tissue image using a threshold calculation method to obtain a grayscale threshold; According to the comparison result of the grayscale of each pixel point in the tissue image and the grayscale threshold, the tissue image is segmented into a target tissue area and a non-target tissue area; A binary image is obtained according to the target tissue area and the non-target tissue area.

14. The endoscopic image display device according to claim 13, wherein the feature extraction module is specifically used for: For each pixel point in the tissue image, if the grayscale of the pixel point is greater than the grayscale threshold, the pixel point is determined as a pixel point of the target tissue area; If the grayscale of the pixel point is less than or equal to the grayscale threshold, the pixel point is determined as a pixel point of the non-target tissue area; The target tissue area and the non-target tissue area in the tissue image are determined according to the pixel points of the target tissue area and the pixel points of the non-target tissue area.

15. The endoscopic image display device according to any one of claims 10 to 14, further used for: Performing enhancement processing on the target tissue contour to obtain an enhanced target tissue contour; The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises: The enhanced target tissue contour is fused with the original white light image to obtain the white light image with the target tissue contour.

16. The endoscopic image display device according to claim 15, further configured to: Based on the target tissue contour, obtaining a contour area of ​​the target tissue; Calculate the channel pixels of the target color of the original white light image in the contour area to obtain a channel contour pixel set; Calculating the pixels of the tissue image within the contour area to obtain a tissue contour pixel set; Obtaining contour features through contour feature calculation according to the channel contour information pixel set and the tissue contour information pixel set; According to the contour features, a coefficient enhancement matrix is ​​calculated by a coefficient enhancement algorithm; According to the coefficient enhancement matrix, a nonlinear adjustment coefficient is calculated by a nonlinear adjustment algorithm; The enhanced target tissue contour is calculated based on the nonlinear adjustment coefficient, the channel contour pixel set, the tissue contour pixel set, and a contour enhancement algorithm.

17. The endoscopic image display device according to claim 16, further configured to: Deleting pixels of the original white-light image within the contour area to obtain a new white-light image; The step of fusing the enhanced target tissue contour with the original white light image to obtain the white light image with the target tissue contour comprises: The enhanced target tissue contour is fused with the new white light image to obtain the white light image with the target tissue contour.

18. The endoscopic image display device according to any one of claims 10 to 17, further used for: Registering the original white light image and the tissue image to obtain a standard white light image and a standard tissue image; The step of extracting tissue contours from the tissue image to obtain the target tissue contour comprises: Extracting tissue contours from the standard tissue image to obtain target tissue contours; The step of fusing the target tissue contour with the original white light image to obtain a white light image with the target tissue contour comprises: The target tissue contour is fused with the standard white light image and displayed to obtain a white light image with the target tissue contour.

19. A terminal device, characterized in that: The terminal device includes a memory, a processor, and an endoscopic image display program stored in the memory and executable on the processor. When the endoscopic image display program is executed by the processor, the steps of the endoscopic image display method as described in any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an endoscopic image display program, which, when executed by a processor, implements the steps of the endoscopic image display method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Medical image segmentation method based on improved gradient vector flow model

    CN104680498A

  • Image fusion method and system

    CN111223069A

  • Image processing method and device applied to endoscope and related equipment

    CN111513660A

  • Multispectral imaging system, imaging method and storage medium

    CN113208567A

  • Image processing method, device and equipment

    CN114298980A

Cited By

  • Endoscope imaging method and system

    CN121582617A