AI-based endoscope image tumor marking method
Through the AI-based laminoscopic image tumor labeling method, the real-time identification and labeling of tumor boundaries and edges is achieved using Mask R-CNN and GAN networks, which solves the error problem caused by fluorescent laminoscopic and white laminoscopic image switching, and improves the accuracy and safety of laminoscopic surgery.
Patent Information
- Application Number
- CN202510520942.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-08
AI Technical Summary
In existing laparoscopic surgery, switching between fluorescence laparoscopic images and white laparoscopic images leads to visual memory errors of the doctor, making it difficult to accurately judge the tumor range, affecting the accuracy and safety of the surgery.
Using AI-based laminoscopic tumor labeling method, white light and fluorescent images are collected at the same time, location and the same focal length, combined with the Mask R-CNN algorithm and GAN network, real-time identification of tumor tissue boundaries and automatic labeling of safe edges are realized, and image registration and fusion technology are used to display tumor boundaries and edges in real time in ultra-high-definition white light images.
It reduces the error of tumor resection, improves the accuracy and efficiency of the surgery, reduces the risk of tumor recurrence, and improves the cure rate and long-term survival rate of malignant tumor resection.
Smart Images
Figure CN120451980A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical equipment technology, and in particular to an AI-based laparoscopic image tumor annotation method. Background Art
[0002] Fluorescence laparoscopy is commonly used in malignant tumor resection. Fluorescence laparoscopy is a laparoscopic procedure based on fluorescence imaging, which typically uses indocyanine green (ICG). This technique uses the principle that normal cells can excrete ICG well, while tumor cells cannot. This technique creates a clear boundary between the tumor tissue to be resected and healthy tissue. Proteins bind to ICG, emitting fluorescence that can be identified by special equipment, allowing the tumor's location and the surgical resection site to be determined. This allows doctors to precisely remove the tumor tissue, making the surgery more accurate and safer.
[0003] Existing laparoscopic fluorescence systems typically display fluorescent and white-light laparoscopic images on separate screens. During surgery, the surgeon's line of sight must constantly switch between the two screens, relying on visual memory to match the tumor's extent, which inevitably introduces errors. Alternatively, a fused display of fluorescent and white-light laparoscopic images is available, but these fused images offer limited detail, making it difficult to accurately identify isolated lesions based solely on visual inspection. Therefore, there is room for improvement. Summary of the Invention
[0004] This invention aims to address the shortcomings of existing technologies by proposing an AI-based method for annotating tumors in laparoscopic images. Its advantage lies in overcoming the errors caused by traditional methods, where physicians rely on visual memory to match tumor extents by comparing fluorescent and white-light laparoscopic images, thereby enabling real-time guidance and precise tumor resection.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] An AI-based laparoscopic image tumor annotation method includes the following steps:
[0007] Step 1: When in use, the white light acquisition module and the fluorescence acquisition module acquire images at the same time, location and focal length, and the image storage module stores the white light image information acquired by the white light acquisition module and the fluorescence image information acquired by the fluorescence acquisition module respectively;
[0008] Step 2: Based on the Mask R-CNN algorithm, a lightweight target recognition algorithm that separates channel convolution and spatial convolution is constructed to quickly detect and identify ultra-high-definition fluorescence image data and accurately identify tumor tissue boundaries;
[0009] The Mask R-CNN algorithm replaces the RoIpooling layer of Faster R-CNN with the RoIAlign layer, cancels the integer operation, retains the decimals, and uses bilinear interpolation to obtain the image values at the pixel points with floating-point coordinates. The formula for bilinear interpolation is as follows:
[0010] Interpolation in the x-direction:
[0011] Interpolation in the y direction:
[0012] The final result:
[0013] Among them, x, y, x1, x2, y1, y2 are coordinate points, and f(x, y) is the final result after interpolation;
[0014] At the same time, Mask R-CNN adds a mask branch to the classification and regression layer based on Faster R-CNN. The backbone of Mask R-CNN uses a PPN network. The class, box, and mask branches do not share the ROI layer, ensuring that the mask branch has more detailed information, achieving pixel-level boundary recognition and accurately identifying the boundaries of tumor cells.
[0015] Step 3: Feature extraction is performed on the two images to obtain feature points. Matching feature point pairs are found by performing similarity measurement. Image space coordinate transformation parameters are then obtained from the matched feature point pairs. Finally, the coordinate transformation parameters are used for image registration. The identified tumor boundary in the fluorescence image is registered and fused with the white-light laparoscope image, and the tumor tissue boundary is annotated in real time in the ultra-high-definition white-light laparoscope image.
[0016] Step 4: Use the built-in algorithm to calculate the tumor size, and automatically mark the safe resection margin in real time based on the safe resection margin distance data set by the system and the tumor boundary and organ boundary identified by AI; display the white light laparoscope image with tumor boundary and resection margin markings in real time to guide the physician to accurately remove the tumor tissue.
[0017] The present invention is further configured such that the white light image information and the fluorescent image information are improved through a series of preprocessing operations, the processing methods including illumination correction, blur and focus correction, filtering and denoising, thresholding, edge enhancement, image segmentation and color conversion.
[0018] The present invention is further configured such that the recognition and marking of the tumor tissue resection margin are carried out by equally expanding outward at an equal distance based on the tumor boundary according to the safety distance set by the user. However, in the actual use process, the equal-distance outward expansion will involve the safety of other tissue organs. Therefore, the resection margin is judged before the resection margin marking. For the resection margin exceeding the organ boundary, the resection boundary needs to be judged according to the specific situation; for the judgment of the organ boundary, the Mask R-CNN algorithm model is called to realize the recognition of the organ boundary.
[0019] The present invention is further configured such that the tumor boundary is set as x, the set safety distance is d, the resection margin boundary is y, and the organ boundary is Z.
[0020] When x + d < z, y = x + d; that is, the resection margin is expanded outward by the safety distance d based on the tumor boundary.
[0021] When x + d ≥ z, y = z; that is, when the calculation result exceeds the organ boundary, the organ boundary is used as the resection margin.
[0022] The present invention is further configured such that the image fusion has two parts. One part refers to fusing the tumor boundary and the white light image into one body, and the other part refers to fusing the resection margin, the tumor boundary, and the white light image into one body. Image registration is performed before image fusion to make the images consistent in time and space.
[0023] The present invention is further configured such that the image fusion adopts the GAN artificial intelligence algorithm. The GAN network mainly consists of a generator and a discriminator. The generator network mainly consists of three parts: feature extraction, feature fusion, and feature reconstruction. The main task is to extract, fuse, and reconstruct the features of the input image to generate a fused image.
[0024] The present invention is further configured such that the discriminator discriminates the false fused image generated by the generator; if it is discriminated as a false image, the parameters of the generator are continuously adjusted until the discriminator discriminates it as a true image to generate a fused annotation image with higher quality.
[0025] The adversarial relationship between G and D is:
[0026]
[0027] In the formula: x represents the input sample, z represents the noise input to the generator, P data (x) represents the real data distribution, P z (z) represents the noise distribution, D(x) represents the probability that the discriminator judges whether the real data is real, and D(G(z)) represents the probability that the discriminator judges whether the generated data is real.
[0028] An AI-based endoscopic image tumor annotation system, the image tumor annotation system applying the image tumor annotation method described in any one of claims 1 to 7. The prison intelligent correction system includes an image acquisition module, an image storage module, an image AI recognition and annotation module, and a display and human-computer interaction module; the image acquisition module is divided into a white light image acquisition module and a fluorescence image acquisition module; the display and human-computer interaction module displays the annotated ultra-high-definition white light endoscopic image, guiding the physician to accurately identify the tumor tissue boundary and resection margin, and the resection margin parameters can be set; the image AI recognition and annotation module is the data processing center, mainly realizing the recognition and annotation of images and the fusion of images.
[0029] A computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.
[0030] A computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.
[0031] The beneficial effects of the present invention are as follows:
[0032] 1. The present invention proposes an AI-based method for annotating the tumor boundary of endoscopic images, which identifies the tumor tissue boundary in the fluorescence endoscopic image based on AI and real-time annotates the tumor tissue boundary and safety resection margin in the ultra-high-definition white light endoscopic image, ensuring the pathological negativity of the resection margin, reducing tumor recurrence caused by insufficient tumor resection margin or incomplete tumor resection, and improving the cure rate and long-term survival rate of malignant tumor resection.
[0033] 2. The tumor tissue boundary and safety resection margin are real-time annotated in the ultra-high-definition white light endoscopic image, and the physician does not need to turn his head back and forth to observe the fluorescence endoscopic image and the white light endoscopic image for comparison and reference, improving the surgical efficiency and the comfort of operation;
[0034] 3. The tumor tissue boundary and safety resection margin are real-time annotated in the ultra-high-definition white light endoscopic image, solving the error caused by the physician observing the fluorescence endoscopic image and the white light endoscopic image for comparison and reference and matching the tumor range by visual memory in the traditional scheme, and realizing the real-time guidance and precise resection of tumor resection. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a schematic diagram of the annotation process of an AI-based method for annotating the tumor of endoscopic images proposed by the present invention;
[0036] Figure 2 It is a schematic diagram of the annotation of the resection margin y when x + d < z in an AI-based method for annotating the tumor of endoscopic images proposed by the present invention;
[0037] Figure 3 This is a schematic diagram of the annotation of the resection margin y when x+d≥z, an AI-based laparoscopic image tumor annotation method proposed by the present invention;
[0038] Figure 4 This is a schematic diagram of the mask branch of the AI-based laparoscopic image tumor annotation method proposed in the present invention;
[0039] Figure 5 This is a schematic diagram of the GAN artificial intelligence algorithm image fusion annotation process of the AI-based laparoscopic image tumor annotation method proposed in this invention. DETAILED DESCRIPTION
[0040] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.
[0041] The following describes in detail embodiments of the present invention, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0042] In the description of this patent, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings. They are only for the convenience of describing this patent and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on this patent.
[0043] In the description of this patent, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," "connected," and "set" should be understood in a broad sense. For example, they can refer to fixed connection or set, detachable connection or set, or integral connection or set. Those skilled in the art will understand the specific meanings of the above terms in this patent based on the specific circumstances.
[0044] Reference Figure 1-5 , an AI-based laparoscopic image tumor annotation method, comprising the following steps:
[0045] Step 1: When in use, the white light acquisition module and the fluorescence acquisition module acquire images at the same time, location and focal length, and the image storage module stores the white light image information acquired by the white light acquisition module and the fluorescence image information acquired by the fluorescence acquisition module respectively;
[0046] Step 2: Based on the Mask R-CNN algorithm, a lightweight target recognition algorithm that separates channel convolution and spatial convolution is constructed to quickly detect and identify ultra-high-definition fluorescence image data and accurately identify tumor tissue boundaries;
[0047] The Mask R-CNN algorithm replaces the RoIpooling layer of Faster R-CNN with the RoIAlign layer, cancels the integer operation, retains the decimals, and uses bilinear interpolation to obtain the image values at the pixel points with floating-point coordinates. The formula for bilinear interpolation is as follows:
[0048] Interpolation in the x-direction:
[0049] Interpolation in the y direction:
[0050] The final result:
[0051] Among them, x, y, x1, x2, y1, y2 are coordinate points, and f(x, y) is the final result after interpolation;
[0052] At the same time, Mask R-CNN adds a mask branch to the classification and regression layer based on Faster R-CNN. The backbone of Mask R-CNN uses a PPN network. The class, box, and mask branches do not share the ROI layer, ensuring that the mask branch has more detailed information, achieving pixel-level boundary recognition and accurately identifying the boundaries of tumor cells.
[0053] Step 3: Feature extraction is performed on the two images to obtain feature points. Matching feature point pairs are found by performing similarity measurement. Image space coordinate transformation parameters are then obtained from the matched feature point pairs. Finally, the coordinate transformation parameters are used for image registration. The identified tumor boundary in the fluorescence image is registered and fused with the white-light laparoscope image, and the tumor tissue boundary is annotated in real time in the ultra-high-definition white-light laparoscope image.
[0054] Step 4: Use the built-in algorithm to calculate the tumor size, and automatically mark the safe resection margin in real time based on the safe resection margin distance data set by the system and the tumor boundary and organ boundary identified by AI; display the white light laparoscope image with tumor boundary and resection margin markings in real time to guide the physician to accurately remove the tumor tissue.
[0055] Furthermore, the white light image information and the fluorescence image information are improved through a series of preprocessing operations, including illumination correction, blur and focus correction, filtering and denoising, thresholding, edge enhancement, image segmentation and color conversion.
[0056] The recognition and marking of the tumor tissue resection margin are carried out by expanding equidistantly based on the tumor boundary according to the safety distance set by the user. However, in the actual use process, the equidistant expansion will involve the safety of other tissues and organs. Therefore, the resection margin is judged before marking the resection margin. For the resection margin that exceeds the organ boundary, the resection boundary needs to be judged according to the specific situation; for the judgment of the organ boundary, the Mask R-CNN algorithm model is called to realize the recognition of the organ boundary; the tumor boundary is set as x, the set safety distance is d, the resection margin boundary is y, and the organ boundary is Z.
[0057] When x + d < z, y = x + d; that is, the resection margin is expanded by the safety distance d based on the tumor boundary.
[0058] When x + d ≥ z, y = z; that is, when the calculation result exceeds the organ boundary, the organ boundary is used as the resection margin.
[0059] Image fusion has two parts. One part refers to fusing the tumor boundary with the white light image into one, and the other part refers to fusing the resection margin with the tumor boundary and the white light image into one. Image registration is carried out before image fusion to make the images consistent in time and space; GAN artificial intelligence algorithm is used for image fusion. The GAN network is mainly composed of a generator and a discriminator. The generator network is mainly composed of three parts: feature extraction, feature fusion, and feature reconstruction. Its main task is to extract, fuse, and reconstruct the features of the input image to generate a fused image; the discriminator discriminates the false fused image generated by the generator; if it is discriminated as a false image, the parameters of the generator are continuously adjusted until the discriminator discriminates it as a true image, generating a higher-quality fused annotation image.
[0060] The adversarial relationship between G and D is:
[0061]
[0062] In the formula: x represents the input sample, z represents the noise input to the generator, P data (x) represents the real data distribution, P z (z) represents the noise distribution, D(x) represents the probability that the discriminator judges whether the real data is real, and D(G(z)) represents the probability that the discriminator judges whether the generated data is real.
[0063] An AI-based laparoscopic image tumor annotation system, the image tumor annotation system applies the image tumor annotation method of any one of claims 1 to 7, the prison intelligent correction system includes an image acquisition module, an image storage module, an image AI recognition and annotation module, and a display and human-computer interaction module; the image acquisition module is divided into a white light image acquisition module and a fluorescence image acquisition module, and white light sensors and fluorescence sensors are alternately distributed in one lens; the display and human-computer interaction module displays the annotated ultra-high-definition white light laparoscope image to guide the physician to accurately identify the tumor tissue boundary and cutting edge, and the cutting edge parameters can be set; the image AI recognition and annotation module is a data processing center, which mainly realizes image recognition and annotation and image fusion.
[0064] Based on AI, the system can identify tumor boundaries in fluorescent images in real time, and mark tumor tissue boundaries and safe resection margins in ultra-high-definition white-light laparoscope images in real time, assisting physicians in accurately removing tumor tissue. Physicians do not need to turn their heads back and forth for reference, thus achieving real-time guidance and precise removal of tumors.
[0065] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.
[0066] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.
[0067] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.
Claims
1. An AI-based laparoscopic image tumor annotation method, characterized in that: It includes the following steps: Step 1: When in use, the white light acquisition module and the fluorescence acquisition module perform image acquisition at the same time, at the same location, and with the same focal length. The image storage module stores the white light image information acquired by the white light acquisition module and the fluorescence image information acquired by the fluorescence acquisition module respectively; Step 2: Based on the Mask R-CNN algorithm, a lightweight object recognition algorithm that separates channel convolution and spatial convolution is constructed to quickly detect and recognize the ultra-high-definition fluorescence image data, and accurately recognize the boundary of the tumor tissue; The Mask R-CNN algorithm changes the RoI pooling layer of the Faster R-CNN to the RoIAlign layer, cancels the integerization operation, retains the decimal, and uses the method of bilinear interpolation to obtain the image value at the pixel point with floating-point coordinates; the formula of bilinear interpolation is as follows: Interpolation in the x-direction: Interpolation in the y direction: The final result: Where x, y, x1, x2, y1, y2 are coordinate points, and f(x, y) is the final result after interpolation; At the same time, on the basis of the Faster R-CNN, Mask R-CNN adds a mask branch in the classification and regression layers; the backbone of Mask R-CNN uses the PPN network, and the class, box branches and mask branches do not share the ROI layer, ensuring that the mask branch has more detailed information and realizing pixel-level boundary recognition to accurately recognize the boundary of tumor cells; Step 3: Feature points are extracted from the two images, matching feature point pairs are found through similarity measurement, then the image spatial coordinate transformation parameters are obtained through the matching feature point pairs, and finally the images are registered by the coordinate transformation parameters; the recognized tumor boundary of the fluorescence image is registered and fused with the white light endoscope image, and the tumor tissue boundary is registered and marked in real time in the ultra-high-definition white light endoscope image; Step 4: Use the built-in algorithm to calculate the tumor size, and based on the safety margin distance data set by the system and the tumor boundary and organ boundary recognized by AI, the safety margin is automatically marked in real time; the white light endoscope image with tumor boundary marking and margin marking is displayed in real time to guide the doctor to accurately resect the tumor tissue.
2. The AI-based laparoscopic image tumor annotation method according to claim 1, characterized in that: The white light image information and the fluorescence image information are improved through a series of preprocessing operations, and the processing methods include illumination correction, blur and focus correction, filtering and denoising, thresholding, edge enhancement, image segmentation and color conversion.
3. The AI-based laparoscopic image tumor annotation method according to claim 1, characterized in that: The recognition and marking of the tumor tissue margin are performed by expanding equidistantly based on the tumor boundary according to the safety distance set by the user. However, in the actual use process, the equidistant expansion will involve the safety of other tissues and organs. Therefore, the margin judgment is performed before the margin marking. For the margin that exceeds the organ boundary, the resection boundary needs to be judged according to the specific situation; for the judgment of the organ boundary, the Mask R-CNN algorithm model is called to realize the boundary recognition of the organ.
4. The AI-based laparoscopic image tumor annotation method according to claim 3, characterized in that: Set the tumor boundary as x, the set safety distance as d, the margin boundary as y, and the organ boundary as Z. When x + d < z, y = x + d; that is, the margin is expanded by the safety distance d based on the tumor boundary. When x+d≥z, y=z; that is, when the calculated result exceeds the organ boundary, the organ boundary is used as the cutting edge.
5. The AI-based laparoscopic image tumor annotation method according to claim 1, characterized in that: The image fusion has two parts. One part refers to fusing the tumor boundary with the white light image, and the other part refers to fusing the cutting edge with the tumor boundary and the white light image. Image registration is performed before image fusion to ensure that the images are consistent in time and space.
6. The AI-based laparoscopic image tumor annotation method according to claim 5, characterized in that: The image fusion adopts the GAN artificial intelligence algorithm. The GAN network is mainly composed of a discriminator and a generator network is mainly composed of three parts: feature extraction, feature fusion and feature reconstruction. The main task is to extract, fuse and reconstruct the features of the input image to generate a fused image.
7. The AI-based laparoscopic image tumor annotation method according to claim 6, characterized in that: The discriminator identifies the false fused image generated by the generator; if it is identified as a false image, the generator parameters are continuously adjusted until the discriminator identifies it as a true image, thereby generating a higher quality fused annotated image; The adversarial relationship between G and D is: Where: x represents the input sample, z represents the noise input to the generator, P data (x) represents the real data distribution, P z (z) represents the noise distribution, D(x) represents the probability that the discriminator judges whether the real data is real, and D(G(z)) represents the probability that the discriminator judges whether the generated data is real.
8. An AI-based laparoscopic image tumor annotation system, characterized by: The image tumor annotation system applies the image tumor annotation method described in any one of claims 1 to 7, and the prison intelligent correction system includes an image acquisition module, an image storage module, an image AI recognition and annotation module, and a display and human-computer interaction module; the image acquisition module is divided into a white light image acquisition module and a fluorescence image acquisition module; the display and human-computer interaction module displays the annotated ultra-high-definition white light laparoscope image to guide the physician to accurately identify the tumor tissue boundary and resection margin, and the resection margin parameters can be set; the image AI recognition and annotation module is a data processing center, which mainly realizes image recognition and annotation and image fusion.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.