An endoscopic puncture guidance method, system, and device

Abstract

The application discloses an endoscope puncture guiding method, system and device, and belongs to the field of medical image processing. A plurality of optical images of a puncture site outside a lesion are acquired by a camera to calculate the surface characteristics of the puncture site outside the lesion. A plurality of ultrasound images and a plurality of optical images of the lesion are acquired by an ultrasound probe and the camera. The lesion is three-dimensionally reconstructed by using the plurality of ultrasound images, and the surface three-dimensional feature points of the lesion are acquired by using the plurality of optical images. According to the surface three-dimensional information of the lesion and the surface characteristics of the puncture site outside the lesion, the positional relationship between the lesion and the puncture instrument plane is calculated, and the included angle between the current instrument puncture direction and the shortest path is obtained. Through the above steps, the puncture / surgery auxiliary guidance is realized by using ultrasound and optical images, the volume of the scope is not additionally increased, the system processing degree is fast, the real-time performance is good, different image resolutions from 480p to 1080p can be compatible, the highest frame rate output can reach 60fps, and the system detection precision is high.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to endoscopic puncture guidance methods and systems. Background Technology

[0002] An endoscope is a medical device used to directly observe the internal cavities of human organs. It generally consists of a camera, a cold light source, a main unit, a monitor, a beam guide, and lenses for different body parts. The endoscope enters the body through natural openings or small incisions, and captures images using a CCD or CMOS sensor. The images are then processed and displayed on an external monitor. Endoscopic systems are widely used for the examination and surgical treatment of human organs and tissues. Endoscopic systems offer significant advantages such as minimal invasiveness, ease of use, and short operation time, and are widely used in various departments.

[0003] Endoscopes can be divided into two types: rigid endoscopes and flexible endoscopes. Rigid endoscopes are typically used for the diagnosis and treatment of lesions that require external puncture and opening of the oral cavity, such as the thoracic cavity and abdominal cavity. Flexible endoscopes, on the other hand, are inserted into the body through natural orifices (such as the digestive tract and bronchi). They are longer and need to be flexible, allowing them to bend. Compared to rigid endoscopes, they cause less damage to the human body, are safer, and make it easier to observe the internal condition of human organs.

[0004] However, in the current technology, when doctors use flexible endoscopes for examinations or punctures, they adjust the bending angle of the endoscope by using the rear handwheel to better adapt it to the shape of the cavity and obtain the best field of view, which requires a high level of operational experience. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, one of the objectives of this invention is to provide an endoscopic puncture guidance method that can achieve puncture / surgery guidance without increasing the volume of the endoscope, thereby reducing the operational experience requirements for doctors.

[0006] In order to overcome the shortcomings of the prior art, the second objective of this invention is to provide an endoscopic puncture guidance system that can achieve puncture / surgery guidance without increasing the volume of the endoscope, thereby reducing the operational experience requirements for doctors.

[0007] In order to overcome the shortcomings of the prior art, the second objective of this invention is to provide an endoscopic puncture guidance device that can achieve puncture / surgery guidance without increasing the volume of the endoscope, thereby reducing the operational experience requirements for doctors.

[0008] One of the objectives of this invention is achieved through the following technical solution:

[0009] An endoscopic puncture-guided method includes the following steps:

[0010] S1: Use a camera to acquire multiple optical images of the lesion outside the puncture site, extract and track feature points for each adjacent frame of the acquired optical image, and use global optimization to reconstruct feature points in three-dimensional space and continuously update them in order to calculate the camera position and obtain three-dimensional feature point information of the puncture site.

[0011] S2: Acquire multiple ultrasound images and multiple optical images of the lesion, use multiple ultrasound images to perform three-dimensional reconstruction of the lesion, and use multiple optical images to acquire three-dimensional feature points on the surface of the lesion.

[0012] S3: Calculate the positional relationship between the lesion and the puncture instrument plane, project the three-dimensional lesion model onto the two-dimensional endoscopic image of the camera. At this time, the endoscope is located at the position corresponding to the maximum lesion interface in the ultrasound image. Display the outline of the internal lesion in real time on the endoscopic image, project the two-dimensional epidermal information in the endoscope onto the ultrasound image, and obtain the epidermal layer outline indicator line; draw a perpendicular line from the centroid of the lesion to the surface tissue surface. The maximum cross-section of the lesion where the perpendicular line is located is the ideal puncture plane; obtain the horizontal and vertical tangential planes of the instrument channel plane, and calculate the angles between the two planes and the puncture plane to obtain the horizontal and vertical deflection angles of the endoscope.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] S11: Rotate and translate the initial lens of the camera to acquire two initial frames;

[0015] S12: Extract feature points from two adjacent frames;

[0016] S13: Match key points between each image, estimate a matching matrix F using multiple point pairs, and obtain the matching of the projection points of each spatial point on different images;

[0017] S14: Calculate the extracted feature point matching results to obtain the camera's position parameters;

[0018] S15: Utilize camera parameters and feature point matching results to perform least squares calculations, obtain spatial information of feature points, acquire sparse three-dimensional point clouds, and complete local three-dimensional reconstruction of the lesion's external puncture site.

[0019] S16: Continue acquiring images, return to step S12 to obtain new feature point pairs, combine them with the original feature points, perform global optimization, and update camera position parameters and point cloud information.

[0020] Furthermore, in step S14, when the number of feature point pairs is less than a preset value, the process returns to step S11. When the number of feature point pairs is greater than or equal to the preset value, the camera motion is solved using 2D-2D epipolar constraints based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, thus obtaining the latest position parameters of the camera.

[0021] Further, in step S15, the least squares calculation is performed using camera parameters and feature point matching results to obtain the spatial information of the feature points. Specifically, the spatial position of the feature points is obtained by using triangulation and different 2D projections of the same spatial feature points in two consecutive frames, and by using the least squares method.

[0022] Further, in step S16, the global optimization specifically involves: storing the obtained 3D spatial feature point information; when the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, the camera pose is optimized using the BA cost function.

[0023] Furthermore, in step S2, the location of the lesion is determined by ultrasound images.

[0024] Furthermore, in step S2, multiple ultrasound images and multiple optical images are acquired by moving the endoscope at equal intervals at the lesion site. The multiple ultrasound images are an ultrasound image sequence, and the multiple optical images are an optical acoustic image sequence.

[0025] Furthermore, the three-dimensional reconstruction of the lesion using multiple ultrasound images is specifically as follows: First, the pixels on the two-dimensional ultrasound image sequence are mapped to their corresponding positions in the three-dimensional imaging space through coordinate transformation, and the pixel value is assigned to the voxel; second, voxel filling is performed by traversing all voxels in the three-dimensional imaging space, finding empty voxels, and then filling the empty voxels using an interpolation algorithm.

[0026] Furthermore, step S3 also includes: adjusting the endoscope according to the horizontal deflection angle and the vertical deflection angle, and recalculating the horizontal deflection angle and the vertical deflection angle after adjustment until the values ​​of the horizontal deflection angle and the vertical deflection angle are less than the preset values.

[0027] The second objective of this invention is achieved by the following technical solution:

[0028] An endoscopic puncture guidance system for implementing any of the above-mentioned endoscopic puncture guidance methods includes a surface positioning module, a deep positioning module, and a puncture guidance module. The surface positioning module processes optical images of the lesion to be punctured to obtain surface features of the lesion, calculates the camera position, and obtains three-dimensional feature point information of the surface features. The deep positioning module processes ultrasound images of the lesion to perform three-dimensional reconstruction of the lesion and find a suitable puncture starting position. The deep positioning module processes optical images of the lesion to obtain three-dimensional surface information of the lesion. The puncture guidance module calculates the positional relationship between the lesion and the puncture instrument plane based on the three-dimensional surface information of the lesion and the surface features of the lesion to be punctured, and determines the angle between the current instrument puncture direction and the shortest path.

[0029] The third objective of this invention is achieved by the following technical solution:

[0030] An endoscopic puncture guidance device for implementing any of the above-mentioned endoscopic puncture guidance methods includes a camera, an ultrasound probe, and a processor. The camera and the ultrasound probe are installed at the end of an endoscope. The camera acquires optical images of the puncture site outside the lesion and optical images of the lesion itself. The ultrasound probe determines the location of the lesion and acquires an ultrasound image of the lesion. The processor processes the optical images of the puncture site outside the lesion, the optical images of the lesion itself, and the ultrasound images of the lesion itself to calculate the horizontal and vertical deflection angles of the endoscope.

[0031] Compared to existing technologies, the endoscopic puncture guidance method of this invention uses a camera to acquire multiple optical images of the puncture site outside the lesion to calculate the surface features of the puncture site. Multiple ultrasound and optical images of the lesion are acquired using an ultrasound probe and a camera. The lesion is reconstructed in three dimensions using the ultrasound images, and three-dimensional feature points of the lesion surface are obtained using the optical images. Based on the three-dimensional information of the lesion surface and the surface features of the puncture site outside the lesion, the positional relationship between the lesion and the puncture instrument plane is calculated, and the angle between the current instrument puncture direction and the shortest path is determined. Through these steps, puncture / surgery-assisted guidance is achieved using ultrasound and optical images without increasing the volume of the endoscope. The system has fast processing speed, good real-time performance, and is compatible with different image resolutions from 480p to 1080p, supporting a maximum frame rate output of 60fps. The system has high detection accuracy and also features limit protection and early warning functions, providing early warning when there is a risk of collision at the endoscope / probe, preventing accidental contact and patient injury. Attached Figure Description

[0032] Figure 1 This is a flowchart of the endoscopic puncture guidance method of the present invention;

[0033] Figure 2 for Figure 1The flowchart of step S1 in the endoscopic puncture guidance method;

[0034] Figure 3 This is a schematic diagram illustrating the endoscopic puncture guidance method of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or it can be fixed through another intermediate component. When a component is said to be "connected to" another component, it can be directly connected to the other component or it may be fixed through another intermediate component. When a component is said to be "set on" another component, it can be set directly on the other component or it may be set through another intermediate component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0038] Figure 1 The endoscopic puncture guidance method of the present invention includes the following steps:

[0039] S1: Use a camera to acquire multiple optical images of the lesion outside the puncture site, extract and track feature points for each adjacent frame of the acquired optical image, and use global optimization to reconstruct feature points in three-dimensional space and continuously update them in order to calculate the camera position and obtain three-dimensional feature point information of the puncture site.

[0040] S2: Acquire multiple ultrasound images and multiple optical images of the lesion, use multiple ultrasound images to perform three-dimensional reconstruction of the lesion, and use multiple optical images to acquire three-dimensional feature points on the surface of the lesion.

[0041] S3: Calculate the positional relationship between the lesion and the puncture instrument plane, project the three-dimensional lesion model onto the two-dimensional endoscopic image of the camera. At this time, the endoscope is located at the position corresponding to the maximum lesion interface in the ultrasound image. Display the outline of the internal lesion in real time on the endoscopic image, project the two-dimensional epidermal information in the endoscope onto the ultrasound image, and obtain the epidermal layer outline indicator line; draw a perpendicular line from the centroid of the lesion to the surface tissue surface. The maximum cross-section of the lesion where the perpendicular line is located is the ideal puncture plane; obtain the horizontal and vertical tangential planes of the instrument channel plane, and calculate the angles between the two planes and the puncture plane to obtain the horizontal and vertical deflection angles of the endoscope.

[0042] Please continue reading. Figure 2 Step S1 specifically includes the following steps:

[0043] S11: Rotate and translate the initial lens of the camera to acquire two initial frames;

[0044] S12: Extract feature points from two adjacent frames;

[0045] S13: Match key points between each image, estimate a matching matrix F using multiple point pairs, and obtain the matching of the projection points of each spatial point on different images;

[0046] S14: Calculate the extracted feature point matching results to obtain the camera's position parameters;

[0047] S15: Utilize camera parameters and feature point matching results to perform least squares calculations, obtain spatial information of feature points, acquire sparse three-dimensional point clouds, and complete local three-dimensional reconstruction of the lesion's external puncture site.

[0048] S16: Continue acquiring images, return to step S12 to obtain new feature point pairs, combine them with the original feature points, perform global optimization, and update camera position parameters and point cloud information.

[0049] Specifically, step S11 aims to initialize the camera by using the camera coordinate system acquired during the acquisition of the first frame as the world coordinate system, which is then used as the coordinate system for calculation in the endoscopic puncture guidance method. In step S12, the ORB algorithm is used to extract feature points from adjacent frames and calculate the BRIEF descriptor. In step S13, the Fast Approximate Nearest Neighbor (FLANN) algorithm is used to perform descriptor matching on all feature points to obtain feature point pairs. In step S14, if the number of feature point pairs is less than a preset value, the process returns to step S11; if the number of feature point pairs is greater than or equal to the preset value, the camera motion is solved using 2D-2D epipolar constraints based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, thus obtaining the latest camera position. In step S15, through triangulation, the spatial position of the feature point is obtained by using the least squares method to solve for the different 2D projections of the same spatial feature point in the preceding and following frames. In step S16, the obtained 3D spatial feature point information is stored. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, the camera pose is optimized using the BA cost function.

[0050] Specifically, in step S14, the preset value is 8, and the camera motion is solved using 2D-2D epipolar constraints.

[0051] In step S2, the location of the lesion is determined using ultrasound images. Specifically, the endoscope is inserted into the body cavity to begin exploration. When the lesion appears in the ultrasound sector scan image, the appropriate endoscope angle is adjusted, the camera is initialized, and the first optical image is obtained. The endoscope is moved, and the movement is performed at equal intervals by reading the displacement sensor at the tip of the endoscope, until the lesion disappears from the ultrasound image. Multiple ultrasound images are acquired, forming an ultrasound image sequence. While acquiring ultrasound images, the camera acquires multiple optical images of the lesion surface, forming an optical-acoustic image sequence. The acquisition of surface three-dimensional feature points is completed simultaneously using the optical images.

[0052] The specific steps for three-dimensional reconstruction of the lesion using multiple ultrasound images are as follows: First, the pixels on the two-dimensional ultrasound image sequence are mapped to their corresponding positions in the three-dimensional imaging space through coordinate transformation, and the pixel value is assigned to a voxel; second, voxel filling is performed by traversing all voxels in the three-dimensional imaging space, finding empty voxels, and then filling the empty voxels using an interpolation algorithm.

[0053] Step S2 also includes: recording the position of the endoscope at the largest point of the lesion interface in the ultrasound image sequence, and moving the endoscope back to that position after the exploration is completed to prepare for the puncture guidance module to insert the needle.

[0054] In step S3, the positional relationship between the lesion and the puncture instrument plane is calculated as S31. Specifically, S31 involves: based on the three-dimensional information of the lesion obtained in step S2, performing coordinate transformation through the spatial positional relationship matrix F between the instrument channel plane and the ultrasound probe to obtain the positional relationship between the lesion and the instrument plane. The horizontal deflection angle and the vertical deflection angle are calculated as in step S32.

[0055] Step S3 also includes step S33: Adjust the endoscope according to the horizontal deflection angle and the vertical deflection angle, and repeat steps S31 and S32 after adjustment to recalculate the horizontal deflection angle and the vertical deflection angle until the values ​​of the horizontal deflection angle and the vertical deflection angle are less than the preset values.

[0056] This invention also relates to an endoscopic puncture guidance system for implementing any of the above-mentioned endoscopic puncture guidance methods, comprising a surface positioning module, a deep positioning module, and a puncture guidance module. The surface positioning module processes optical images of the lesion to be punctured to obtain surface features of the lesion, calculates the camera position, and obtains three-dimensional feature point information of the surface features. The deep positioning module processes ultrasound images of the lesion to perform three-dimensional reconstruction of the lesion and find a suitable puncture starting position. The deep positioning module processes optical images of the lesion to obtain three-dimensional information of the lesion surface. The puncture guidance module calculates the positional relationship between the lesion and the puncture instrument plane based on the three-dimensional information of the lesion surface and the surface features of the lesion to be punctured, and determines the angle between the current instrument puncture direction and the shortest path.

[0057] like Figure 3 As shown, the present invention also relates to an endoscopic puncture guidance device for implementing any of the above-described endoscopic puncture guidance methods, comprising a camera (optical lens), an ultrasound probe, and a processor, wherein the camera and the ultrasound probe are mounted on the endoscope tip. An instrument channel is located between the camera and the ultrasound probe, and the positional relationship between the instrument channel, the camera, and the ultrasound probe is defined. The camera acquires optical images of the area to be punctured outside the lesion and optical images of the lesion itself. The ultrasound probe determines the location of the lesion and acquires an ultrasound image of the lesion. The processor processes the optical images of the area to be punctured outside the lesion, the optical images of the lesion itself, and the ultrasound images of the lesion to calculate the horizontal and vertical deflection angles of the endoscope.

[0058] This invention provides an endoscopic puncture guidance method that uses a camera to acquire multiple optical images of the puncture site outside the lesion, calculates the surface features of the puncture site, and uses an ultrasound probe and camera to acquire multiple ultrasound and optical images of the lesion. The ultrasound images are used to perform three-dimensional reconstruction of the lesion, and the optical images are used to acquire three-dimensional feature points on the lesion's surface. Based on the three-dimensional information of the lesion's surface and the surface features of the puncture site, the positional relationship between the lesion and the puncture instrument plane is calculated, and the angle between the current instrument puncture direction and the shortest path is determined. Through these steps, puncture / surgery-assisted guidance is achieved using ultrasound and optical images without increasing the volume of the endoscope. The system has fast processing speed, good real-time performance, and is compatible with different image resolutions from 480p to 1080p, supporting a maximum frame rate output of 60fps. The system has high detection accuracy and also features limit protection and early warning functions, providing early warning when there is a risk of collision at the endoscope / probe end to prevent accidental contact and patient injury.

[0059] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These are all equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of these fall within the protection scope of the present invention.

Claims

1. An endoscopic puncture guiding device for implementing an endoscopic guidance method, characterized in that, The method includes the following steps: S1: Multiple optical images of the lesion to be punctured are acquired using a camera installed at the end of the endoscope. The camera coordinate system at the time of acquiring the first frame image is used as the reference coordinate system. Feature points are extracted and matched on two adjacent frames of optical images to obtain feature point pairs. The camera motion is solved using 2D-2D epipolar constraints based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, thus obtaining the latest position parameters of the camera. Through triangulation, the spatial position of the feature point is obtained by using the different 2D projections of the same spatial feature point in the two frames and the least squares method. A sparse three-dimensional point cloud is obtained to complete the local three-dimensional reconstruction of the lesion to be punctured. Images are acquired again, new feature point pairs are obtained, and they are combined with the original feature points for global optimization to update the camera position parameters and point cloud information. S2: The endoscope is continuously moved at equal intervals at the lesion site to acquire multiple ultrasound images to form a two-dimensional ultrasound image sequence. Simultaneously, multiple optical images are acquired through the camera to form a two-dimensional optical image sequence. Based on the reference coordinate system, the pixels in the two-dimensional ultrasound image sequence are mapped to their corresponding positions in the three-dimensional imaging space through coordinate transformation, and the pixel values ​​of the pixels in the two-dimensional ultrasound image sequence are assigned to voxels. All voxels in the three-dimensional imaging space are traversed, empty voxels are found, and empty voxels are filled through an interpolation algorithm to obtain a three-dimensional voxel model at the lesion site. Using the two-dimensional optical image sequence, the surface three-dimensional feature points at the lesion site are acquired synchronously in the same way as in step S1. The endoscope movement position corresponding to the maximum lesion interface in the two-dimensional ultrasound image sequence is recorded. S3: Calculate the positional relationship between the lesion and the puncture instrument plane: The endoscope is located at the position corresponding to the maximum lesion interface recorded in step S2. The camera displays a two-dimensional optical image of the internal lesion in real time. The three-dimensional voxel model of the lesion obtained in step S2 is projected onto the two-dimensional optical image of the camera to display the outline of the internal lesion in real time on the two-dimensional optical image. The three-dimensional feature points of the lesion surface obtained in step S2 are projected onto the imaging plane of the two-dimensional ultrasound image to obtain the surface outline indicator line of the lesion. A perpendicular line is drawn from the centroid of the lesion to the surface tissue surface. The maximum cross-section of the lesion where the perpendicular line is located is the ideal puncture plane. The horizontal and vertical tangent planes of the instrument channel plane are obtained. The angles between the horizontal and vertical tangent planes of the instrument channel plane and the puncture plane are calculated respectively to obtain the horizontal and vertical deflection angles of the endoscope.

2. The endoscopic puncture guiding device according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Rotate and translate the initial lens of the camera to acquire two initial frames; S12: Extract feature points from two adjacent frames; S13: Match key points between each image, estimate a matching matrix F using multiple point pairs, and obtain the matching of the projection points of each spatial point on different images; S14: Calculate the extracted feature point matching results to obtain the camera's position parameters; S15: Utilize camera parameters and feature point matching results to perform least squares calculations, obtain spatial information of feature points, acquire sparse three-dimensional point clouds, and complete local three-dimensional reconstruction of the lesion's external puncture site. S16: Continue acquiring images, return to step S12 to obtain new feature point pairs, combine them with the original feature points, perform global optimization, and update camera position parameters and point cloud information.

3. The endoscopic puncture guiding device according to claim 2, characterized in that: In step S14, when the number of feature point pairs is less than a preset value, return to step S11. When the number of feature point pairs is greater than or equal to the preset value, solve the camera motion using 2D-2D epipolar constraints based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, and obtain the latest position parameters of the camera.

4. The endoscopic puncture guiding device according to claim 2, characterized in that: In step S15, the least squares calculation is performed using camera parameters and feature point matching results to obtain the spatial information of the feature points. Specifically, the spatial position of the feature points is obtained by using triangulation and different 2D projections of the same spatial feature points in two consecutive frames, and by using the least squares method.

5. The endoscopic puncture guiding device according to claim 2, characterized in that: In step S16, the global optimization is specifically performed as follows: the obtained 3D spatial feature point information is stored. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, the camera pose is optimized using the BA cost function.

6. The endoscopic puncture guiding device according to claim 1, characterized in that: In step S2, the location of the lesion is determined by ultrasound images.

7. The endoscopic puncture guiding device according to claim 1, characterized in that: Step S3 further includes: adjusting the endoscope according to the horizontal deflection angle and the vertical deflection angle, and recalculating the horizontal deflection angle and the vertical deflection angle after adjustment until the values ​​of the horizontal deflection angle and the vertical deflection angle are less than the preset values.

8. The endoscopic puncture guiding device according to claim 1, characterized in that: The device includes a camera, an ultrasound probe, and a processor. The camera and ultrasound probe are mounted on the endoscope tip. The camera acquires optical images of the area to be punctured outside the lesion and optical images of the lesion itself. The ultrasound probe determines the location of the lesion and acquires ultrasound images of the lesion. The processor processes the optical images of the area to be punctured outside the lesion, the optical images of the lesion itself, and the ultrasound images of the lesion itself to calculate the horizontal and vertical deflection angles of the endoscope.

9. An endoscopic puncture guidance system, comprising the endoscopic puncture guidance device as described in any one of claims 1-8, characterized in that: The device includes a surface positioning module, a deep positioning module, and a puncture guidance module. The surface positioning module processes optical images of the lesion to be punctured to obtain surface features of the lesion, calculates the camera position, and obtains three-dimensional feature point information of the surface features. The deep positioning module processes ultrasound images of the lesion to perform three-dimensional reconstruction of the lesion and find a suitable puncture starting position. The deep positioning module processes optical images of the lesion to obtain three-dimensional surface information of the lesion. The puncture guidance module calculates the positional relationship between the lesion and the puncture instrument plane based on the three-dimensional surface information of the lesion and the surface features of the lesion to be punctured, and determines the angle between the current instrument puncture direction and the shortest path.

Images