Endoscopic navigation method and system for assisting POEM tunnel establishment

Abstract

The present invention belongs to the technical field of POEM tunnel establishment and provides an endoscopic navigation method and system for assisting in POEM tunnel establishment. The method comprises: acquiring several frames of esophageal endoscopic images, performing three-dimensional reconstruction, marking the circular muscle mucosa region in the three-dimensional reconstruction, calculating the three-dimensional centerline of the circular muscle mucosa region, and rendering it on the three-dimensional reconstruction; acquiring the current frame of an esophageal inspection image, determining whether the esophageal inspection image is a submucosal endoscopic image or an inspection-view endoscopic image; if the image is an inspection-view endoscopic image, calculating the coordinate position of the circular muscle mucosa region on the three-dimensional centerline and rendering it on the three-dimensional reconstruction; and if the image is a submucosal endoscopic image, detecting the camera movement direction and, when the angle between two adjacent camera movement directions exceeds a critical value, issuing a camera movement direction adjustment prompt. This method can assist in establishing a perfectly long, straight tunnel that is completely perpendicular to the circular muscle.

Description

Technical Field

[0001] The present invention belongs to the technical field of POEM (peroral endoscopic myotomy) tunnel establishment, and in particular relates to an endoscopic navigation method and system for assisting POEM tunnel establishment. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The cardia is located at the interface between the stomach and the esophagus, at the entrance at the upper end of the stomach. Food that usually enters from the mouth first passes through the esophagus and then through the cardia to enter the stomach.

[0004] Achalasia is an esophageal motility disorder caused by poor relaxation of the lower esophageal sphincter and loss of esophageal peristalsis, leading to food retention and symptoms such as dysphagia, regurgitation, chest pain, and weight loss. If left untreated for a long time, it can lead to chronic gastritis and even esophageal disease, and in severe cases, gastric cancer. Peroral endoscopic myotomy (POEM), a minimally invasive endoscopic procedure, is currently the first-line treatment for achalasia.

[0005] POEM involves incising the esophageal mucosa above the cardia to access the submucosa. A tunnel of a defined length (standard tunnel 10-12 cm, short tunnel 7-8 cm) is then created within the submucosa. Myotomy of the circular and longitudinal muscles deep within the submucosa is then performed within the tunnel, thereby relieving pressure on the lower esophageal sphincter. Establishing a high-quality tunnel is a key step in POEM treatment.

[0006] Establishing a long straight tunnel perpendicular to the circular muscle can create favorable conditions for achieving vertical and effective incision of the circular muscle. However, in actual clinical treatment, it is often difficult to establish a perfect long straight tunnel, especially for patients with achalasia with severe esophageal lumen distortion. The direction of the esophageal circular muscle is often twisted and changing. During the dynamic process of tunnel establishment, it is difficult to achieve a direction perpendicular to the circular muscle at all times. Summary of the Invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an endoscopic navigation method and system for assisting in the establishment of a POEM tunnel. The method dynamically establishes a straight line perpendicular to the circular muscle in real time. During the process of establishing the tunnel, the endoscopist always follows the navigation direction of this straight line to establish the tunnel. Moreover, the method can constantly remind the endoscopist of the correct direction of establishing the tunnel, and ultimately assist the endoscopist in establishing a perfect long straight tunnel completely perpendicular to the circular muscle.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A first aspect of the present invention provides an endoscopic navigation method for assisting in establishing a POEM tunnel.

[0010] An endoscopic navigation method for assisting POEM tunnel establishment, comprising:

[0011] Acquiring several frames of esophageal endoscopic images, performing three-dimensional reconstruction, marking the circular musculomucosa region in the three-dimensional reconstruction, calculating the three-dimensional centerline of the circular musculomucosa region, and rendering it on the three-dimensional reconstruction;

[0012] Obtain the current frame of the esophageal detection image and determine whether the esophageal detection image is a submucosal endoscopic image or an endoscopic image under the inspection perspective. If it is an endoscopic image under the inspection perspective, calculate the coordinate position of the circular muscle mucosa area on the three-dimensional centerline and render it on the three-dimensional reconstructed volume. If it is a submucosal endoscopic image, detect the lens movement direction. When the angle between two adjacent lens movement directions exceeds a critical value, issue a lens movement direction adjustment prompt.

[0013] Furthermore, the detection of the lens movement direction is based on the motion vectors of the current frame esophagus detection image and the previous frames of esophagus detection images.

[0014] Furthermore, the motion vector is calculated using a feature point tracking algorithm based on the optical flow principle.

[0015] Furthermore, the three-dimensional reconstruction is based on point cloud data of each frame of esophageal endoscopy image.

[0016] Furthermore, when the similarity between the current frame esophagus detection image and the previous frame esophagus detection image exceeds a threshold, the lens movement direction is detected.

[0017] Furthermore, the similarity calculation method is as follows: each frame of the esophageal detection image is converted into a grayscale image and scaled, the grayscale average value of all pixels is calculated, and the grayscale value of each pixel in the scaled image is compared with the grayscale average value, the scaled image is converted into a binary image, all pixel values ​​of the binary image are combined into a feature string, and the similarity is calculated based on the feature string between two adjacent frames of the esophageal detection image.

[0018] Furthermore, the esophageal endoscopy image used in the three-dimensional reconstruction only retains pixels of the circular musculomucosa region.

[0019] A second aspect of the present invention provides an endoscopic navigation system for assisting in the establishment of a POEM tunnel.

[0020] An endoscopic navigation system for assisting in the establishment of a POEM tunnel, comprising:

[0021] a three-dimensional reconstruction module configured to: acquire a plurality of frames of esophageal endoscopic images, perform three-dimensional reconstruction, mark the circular musculomucosa region in the three-dimensional reconstruction volume, calculate the three-dimensional centerline of the circular musculomucosa region, and render it on the three-dimensional reconstruction volume;

[0022] The endoscopic navigation module is configured to: obtain the esophageal detection image of the current frame, and determine whether the esophageal detection image is a submucosal endoscopic image or an endoscopic image under the inspection perspective; if it is an endoscopic image under the inspection perspective, calculate the coordinate position of the circular muscle mucosa area on the three-dimensional center line and render it on the three-dimensional reconstructed body; if it is a submucosal endoscopic image, detect the lens movement direction, and when the angle between two adjacent lens movement directions exceeds a critical value, issue a lens movement direction adjustment prompt.

[0023] A third aspect of the present invention provides a computer-readable storage medium.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the endoscopic navigation method for assisting POEM tunnel establishment as described in the first aspect above.

[0025] A fourth aspect of the present invention provides a computer device.

[0026] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the endoscopic navigation method for assisting POEM tunnel establishment as described in the first aspect above are implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention constantly identifies the circular muscle of the esophagus and dynamically establishes a straight line perpendicular to the direction of the circular muscle in real time. During the process of establishing a tunnel, the endoscopist always follows the navigation direction of this straight line to establish the tunnel. Moreover, the endoscopist can always be reminded of the correct direction of establishing the tunnel, ultimately achieving the establishment of a perfect long straight tunnel that is completely perpendicular to the circular muscle. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1 3D centerline schematic diagram of the first embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a three-dimensional reconstructed body shown in Example 1 of the present invention;

[0032] Figure 3 is a schematic diagram of an endoscopic image under an inspection viewing angle according to the first embodiment of the present invention;

[0033] Figure 4 Schematic diagram of an endoscopic image of the submucosal layer shown in Example 1 of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present invention. It should be noted that each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code can include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of the boxes in the flowchart and / or block diagram, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0038] Example 1

[0039] This embodiment provides an endoscopic navigation method for assisting in establishing a POEM tunnel.

[0040] This embodiment provides an endoscopic navigation method to assist in establishing a POEM tunnel. Figure 1 As shown, the esophageal circular muscle is constantly identified and a straight line perpendicular to the circular muscle is dynamically established in real time. During the process of establishing the tunnel, the endoscopist always follows the navigation direction of this straight line to establish the tunnel. This can always remind the endoscopist to establish the correct direction of the tunnel, and ultimately achieve the establishment of a perfect long straight tunnel completely perpendicular to the circular muscle.

[0041] This embodiment provides an endoscopic navigation method for assisting in POEM tunnel establishment, including:

[0042] Step 1: Label the esophageal area within the transparent cap of the endoscope and divide it into the circular and non-circular musculoskeletal regions. Train the semantic segmentation model. Preprocess the original endoscopic image based on the mask data. Pixels within the outline of the circular musculoskeletal region are retained. Pixels within the non-circular musculoskeletal region within the esophagus are set to white. Pixels outside the outline (the non-circular musculoskeletal region and non-circular musculoskeletal region in the semantically segmented image) are set to black. This divides the image into three regions.

[0043] Step 2: Based on VisualSFM software, extract the point cloud data corresponding to each frame of esophageal endoscopy image. With the help of VTK, initialize the point cloud model, encapsulate the point cloud, fit the surface, create the surface patch, and perform surface rendering and reconstruction. Figure 2 Mark the esophageal circular muscle mucosa region in the 3D reconstruction. The circular muscle mucosa region should be located on one side of the 3D reconstruction, and calculate the 3D centerline of the circular muscle mucosa region.

[0044] Among them, the calculation of the three-dimensional centerline uses the centerline extraction algorithm of the VMTK open source tool to extract the three-dimensional centerline of the circular musculomucosa area. VMTK is a collection of libraries and tools for 3D reconstruction, geometric analysis, mesh generation and surface data analysis of image-based vascular modeling.

[0045] Step 3: Train a binary classification neural network (for classifying submucosal endoscopic images and endoscopic images under the inspection perspective). Real-time detection of the endoscopic image (esophageal inspection image). When the binary classification neural network identifies the endoscopic image of the submucosal layer, it means that the circular muscle has been cut and the endoscope has entered the submucosal layer. This corresponds to the endoscopic image under the inspection perspective before entering (such as Figure 3 The following is an endoscopic image under the inspection perspective. The marked area is cut vertically into the Figure 4 Images shown: Endoscopic image of the submucosal layer).

[0046] Since the center point of the musculomucosal area in the endoscopic image under the detection perspective corresponds to the three-dimensional center line of the three-dimensional reconstruction body, it is possible to construct a correspondence between the center position of the endoscopic image entering the submucosal layer after endoscopic incision of the circular muscle and the three-dimensional center line of the three-dimensional reconstruction body.

[0047] To determine the center point of the muscularis mucosa region: calculate the circumscribed rectangle of the irregular muscularis mucosa region and obtain the center point of the rectangular frame, which is the center point of the muscularis mucosa region.

[0048] Step 4: The endoscope motion direction detection module uses the motion vector of each pre-processed input image as a motion feature and analyzes the motion state of the motion vector (lens rotation, lens forward, lens backward).

[0049] The Lucas-Kanade algorithm is used to calculate motion vectors for moving image sequences. It's a feature point tracking algorithm based on the principle of optical flow. Optical flow treats brightness changes on a two-dimensional plane as a continuous fluid, ensuring that the intensity of the brightness signal remains essentially constant along the trajectory of an object's motion. The preprocessed input image is split into n×n blocks of equal size. The optical flow between two adjacent frames is calculated, yielding n×n motion vectors.

[0050] Historical endoscopic examination data is acquired, and the motion vectors between W consecutive image frames (the current esophageal examination image and the previous several frames) are collected. The motion directions of the motion vectors relative to the X-axis in the Cartesian coordinate system are calculated and combined into a custom feature vector {motion direction between the second frame and the first frame, motion direction between the third frame and the second frame, motion direction between the fourth frame and the third frame, ... motion direction between the Wth frame (the current frame) and the W-1th frame}.

[0051] A custom neural network classification model was trained, with the network categories divided into lens rotation, lens forward, and lens backward. The judgment criteria are as follows: if the difference in movement direction exceeds a set threshold and is irregular, the colonoscope lens is considered to have rotated; if the motion vector's direction of movement remains fixed, the endoscope lens is considered to have moved, with the lens forward or backward defined based on the movement direction; if the movement direction moves toward the first and second quadrants of the Cartesian coordinate system, the lens is considered to have moved forward; if the movement direction moves toward the third and fourth quadrants of the Cartesian coordinate system, the lens is considered to have moved backward. Based on the feature vector, the trained model detects the lens movement category (lens movement direction). If the movement category remains unchanged, the change in angle between the movement directions is calculated.

[0052] It should be noted that the judgment criteria set are only for single-frame images; this application aims to determine the motion patterns (lens movement direction) of multiple consecutive frames of images, requiring a neural network to comprehensively determine the motion patterns of multiple consecutive frames of images. In addition, the judgment criteria can be annotated with the neural network data.

[0053] Step 5: During the endoscopic examination, the endoscopist first creates a 3D reconstructed model of the esophagus and determines the 3D centerline of the circular muscle-mucosal region. The program interface displays a linear navigation diagram. After incising the circular muscle and before entering the submucosal layer, the coordinate position of the circular muscle-mucosal region on the 3D centerline is calculated and displayed on the linear navigation diagram. After incising the circular muscle and entering the submucosal layer, the endoscope lens movement direction is identified in real time, and the movement trajectory is detected. When the image similarity (the similarity between the current esophageal inspection image and the previous esophageal inspection image) changes below a set threshold, the lens is considered to have not moved. When the direction detection module detects lens forward movement, the linear navigation diagram moves forward along the 3D centerline coordinate point of the 3D reconstruction (coordinate shift). When the direction detection module detects lens backward movement, the linear navigation diagram moves backward along the 3D centerline coordinate point of the 3D reconstruction. The angle change between the movement directions is calculated. If it exceeds a set threshold (critical value), it is considered a direction deviation, prompting the endoscopist to adjust the endoscope angle.

[0054] Among them, step 3 uses a matching algorithm to obtain a matching correspondence between the endoscopic image after endoscopic incision of the circular muscle and the endoscopic image under the detection perspective. Since the center point of the muscle mucosal area in the endoscopic image under the detection perspective corresponds to the three-dimensional center line of the three-dimensional reconstruction, it is possible to construct a corresponding relationship between the center position of the endoscopic image after endoscopic incision of the circular muscle and the three-dimensional center line of the three-dimensional reconstruction.

[0055] Image similarity is calculated by converting the image into a feature string using the mean hashing algorithm. The captured endoscopic color image is converted into a grayscale image, then resized to an 8×8 image size. The grayscale average of all 64 pixels is calculated and each pixel's grayscale value is compared to the average. Values ​​greater than or equal to the average are assigned a value of 1; values ​​less than the average are assigned a value of 0. The 64 pixel values ​​are combined to form a 64-bit integer, which serves as the feature string using the full-image mean hashing algorithm. The feature strings corresponding to the full-image mean hashing algorithm for two adjacent frames are compared and the Hamming distance is calculated. If the distance is less than a set threshold, the images are considered similar.

[0056] Example 2

[0057] This embodiment provides an endoscopic navigation system for assisting in establishing a POEM tunnel.

[0058] An endoscopic navigation system for assisting in the establishment of a POEM tunnel, comprising:

[0059] a three-dimensional reconstruction module configured to: acquire a plurality of frames of esophageal endoscopic images, perform three-dimensional reconstruction, mark the circular musculomucosa region in the three-dimensional reconstruction volume, calculate the three-dimensional centerline of the circular musculomucosa region, and render it on the three-dimensional reconstruction volume;

[0060] The endoscopic navigation module is configured to: obtain the esophageal detection image of the current frame, and determine whether the esophageal detection image is a submucosal endoscopic image or an endoscopic image under the inspection perspective; if it is an endoscopic image under the inspection perspective, calculate the coordinate position of the circular muscle mucosa area on the three-dimensional center line and render it on the three-dimensional reconstructed body; if it is a submucosal endoscopic image, detect the lens movement direction, and when the angle between two adjacent lens movement directions exceeds a critical value, issue a lens movement direction adjustment prompt.

[0061] It should be noted that the examples and application scenarios implemented by the above modules are the same as those in the steps of Example 1, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0062] Example 3

[0063] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the endoscopic navigation method for assisting POEM tunnel establishment as described in the first embodiment above are implemented.

[0064] Example 4

[0065] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the endoscopic navigation method for assisting POEM tunnel establishment as described in the first embodiment above are implemented.

[0066] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. An endoscopic navigation system for assisting POEM tunnel establishment, characterized in that: include: a three-dimensional reconstruction module configured to: acquire a plurality of frames of esophageal endoscopic images, perform three-dimensional reconstruction, mark the circular musculomucosa region in the three-dimensional reconstruction volume, calculate the three-dimensional centerline of the circular musculomucosa region, and render it on the three-dimensional reconstruction volume; An endoscopic navigation module is configured to: obtain a current frame of an esophageal detection image and determine whether the esophageal detection image is a submucosal endoscopic image or an inspection-view endoscopic image; if it is an inspection-view endoscopic image, calculate the coordinate position of the circular musculomucosal region on the three-dimensional centerline and render it on the three-dimensional reconstructed volume; if it is a submucosal endoscopic image, detect the lens movement direction, and when the angle between two adjacent lens movement directions exceeds a critical value, issue a lens movement direction adjustment prompt; After entering the submucosal layer, the endoscope lens movement direction is identified in real time and the movement trajectory is detected; When the similarity change between the current frame esophagus detection image and the previous frame esophagus detection image is lower than the set threshold, it is considered that the camera has not moved; The motion vector of each pre-processed input image is used as a motion feature to analyze its motion state. A trained custom neural network classification model is used to detect the lens movement category. When the movement category remains unchanged, the angle change between the movement directions is calculated. If the angle change exceeds a set threshold, the direction is determined to be deviated and a prompt to adjust the lens movement direction is issued. The movement categories are divided into lens rotation, lens forward, and lens backward. The custom neural network judgment basis is as follows: if the difference in the movement direction of the motion vector exceeds the set threshold and is irregular, the endoscope lens is judged to be rotating; if the movement direction of the motion vector remains in a fixed direction, the endoscope lens is judged to be moving, and the lens is defined as forward or backward based on the movement direction; if the movement direction moves toward the first and second quadrants of the Cartesian coordinate system, the lens is judged to be moving forward; if the movement direction moves toward the third and fourth quadrants of the Cartesian coordinate system, the lens is judged to be moving backward.

2. The endoscopic navigation system for assisting POEM tunnel establishment according to claim 1, characterized in that: The detection of the lens moving direction is based on the motion vectors of the current frame esophagus detection image and the previous frames of esophagus detection images.

3. The endoscopic navigation system for assisting POEM tunnel establishment according to claim 2, characterized in that: The motion vector is calculated using a feature point tracking algorithm based on the optical flow principle.

4. The endoscopic navigation system for assisting POEM tunnel establishment according to claim 1, characterized in that: The three-dimensional reconstruction is based on point cloud data of each frame of esophageal endoscopy image.

5. The endoscopic navigation system for assisting POEM tunnel establishment according to claim 1, characterized in that: When the similarity between the current esophagus detection image and the previous esophagus detection image exceeds a threshold, the lens movement direction is detected.

6. The endoscopic navigation system for assisting POEM tunnel establishment according to claim 5, characterized in that: The similarity calculation method includes: converting each frame of the esophageal detection image into a grayscale image and scaling it, calculating the grayscale average of all pixels, comparing the grayscale value of each pixel in the scaled image with the grayscale average, converting the scaled image into a binary image, combining all pixel values ​​of the binary image into a feature string, and calculating the similarity based on the feature string between two adjacent frames of the esophageal detection image.

7. The endoscopic navigation system for assisting POEM tunnel establishment according to claim 1, characterized in that: The esophageal endoscopy image used in the three-dimensional reconstruction only retains pixels of the circular musculomucosa region.

Images