An endoscopic instrument guidance method and system

Abstract

The application discloses an endoscope instrument guiding method and system, and belongs to the field of medical image processing. The real-time position of a camera is obtained by feature extraction and matching of an endoscope image frame sequence collected. Then, when a doctor operates an instrument, the angle and extension distance of the instrument are obtained by a sensor, and the relative position relationship between the camera and the instrument is calculated. Finally, the overlapping part of the instrument and the camera field of view in the three-dimensional space is obtained, and is projected to the corresponding camera screen to realize real-time guiding. Through the above steps, the extension position of the endoscope instrument is obtained only by two additional sensors, real-time guiding is realized, the system is small in size, does not increase the additional burden of the equipment, is high in processing speed and real-time performance, is compatible with different image resolutions from 480p to 1080p, can support a frame rate output of 60fps at most, is high in position detection accuracy and good in consistency, and can maintain high instrument positioning and guiding accuracy during long-time examination / surgery (more than 2h).

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to endoscopic instrument guidance methods and systems. Background Technology

[0002] A medical endoscope is a minimally invasive medical device used to assist in examinations or surgeries. An endoscope comprises an optical lens, a camera, and mechanical auxiliary devices. It enters the human body through natural openings or tiny external incisions to observe internal diseased tissues, assisting in diagnosis, treatment, and surgical procedures. Compared to traditional medical methods, endoscopic systems offer significant advantages such as minimal invasiveness, ease of use, and shorter procedure time, and are widely used in various departments.

[0003] An endoscope enters the human body through natural orifices. The insertion section at the front of the endoscope is a flexible structure, containing a camera, light source, water / air channel, and instrument channel. The rear of the endoscope is the operating handle, which connects to the main unit. When using the endoscope for biopsies or surgery, the surgeon controls the direction of the endoscope using the handle. After the endoscope is fixed in place, surgical instruments are inserted into the instrument channel. The extension angle of the instruments is controlled by the lifting forceps wire inside the operating channel to complete the operation. Because the endoscope camera and the instrument channel exit are not on the same plane, instruments often fall out of the camera's field of view. In such cases, the surgeon cannot see the instruments in the image and must rely on experience to adjust the instruments to the appropriate angle, making operation somewhat inconvenient.

[0004] Currently, most commonly used endoscopic surgery / examination guidance methods require the assistance of large external equipment, such as CT, ultrasound and MRI imaging equipment. Intraoperative guidance is carried out through multi-modal image registration or three-dimensional modeling, which involves a large amount of computation, large space occupation of large equipment, and high cost. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, one of the objectives of this invention is to provide an endoscopic instrument guidance method that enables instrument guidance during surgery / examination without the need for large equipment, with low computational load, high processing speed, good real-time performance, and strong scalability.

[0006] To overcome the shortcomings of the prior art, the second objective of this invention is to provide an endoscopic instrument guidance method that enables the instrument system during surgery / examination to operate without large equipment, with low computational load, high processing speed, good real-time performance, and strong scalability.

[0007] One of the objectives of this invention is achieved through the following technical solution:

[0008] An endoscopic instrument guidance method includes the following steps:

[0009] S1: The camera acquires multiple endoscopic images. Feature extraction and matching are performed on adjacent frames. The camera pose is calculated based on the matching results. After acquiring the camera pose, the spatial information of all feature points is obtained. The camera pose is optimized based on the spatial information.

[0010] S2: When operating the instrument, the angle and distance of the instrument's movement relative to the instrument channel are obtained through the sensor, thereby obtaining the position of the instrument head relative to the instrument channel and calculating the relative positional relationship between the camera and the instrument.

[0011] S3: Connect the head end of the instrument and the instrument channel plane in three-dimensional space, construct an extension line, extend the instrument into the field of view of the camera, and project the extension line onto the camera image to achieve prediction and guidance of the instrument position.

[0012] Furthermore, steps S1 and S2 are performed in parallel, and step S1 is performed throughout the entire endoscopic instrument guidance process.

[0013] Furthermore, step S1 specifically includes the following steps:

[0014] S11: Camera initialization: Rotate and translate the initial lens to acquire two initial frames;

[0015] S12: Feature extraction: Extract feature points from two adjacent frames of images and calculate the BRIEF descriptor;

[0016] S13: Perform descriptor matching on all feature points to obtain feature point pairs;

[0017] S14: Camera motion solution: When the number of feature point pairs is less than the preset value, return to step S11. When the number of feature point pairs is greater than or equal to the preset value, solve the camera motion based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, and obtain the latest position of the camera.

[0018] S15: Obtain spatial information of feature points: Through triangulation, using the different 2D projections of the same spatial feature point in two consecutive frames, the spatial position of the feature point is obtained by solving the least squares method.

[0019] S16: Camera pose optimization: Store the obtained 3D spatial feature point information. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, camera pose optimization is performed.

[0020] S17: Continue acquiring images, repeating steps S12 to S16 until endoscopic instrument guidance is complete.

[0021] Furthermore, in step S11, the camera coordinate system used when acquiring the first frame image is taken as the world coordinate system and used as the coordinate system calculated in the endoscopic instrument guidance method.

[0022] Furthermore, in step S14, the preset value is 8, and the camera motion is solved using 2D-2D epipolar constraints.

[0023] Furthermore, step S2 specifically includes the following steps:

[0024] S21: Alignment of instrument with initial calculation plane: Insert the instrument into the instrument channel of the endoscope, read the position sensor reading during insertion, stop the operation when the reading reaches the internal length of the endoscope instrument channel, and reset the position sensor reading to zero to ensure that the instrument tip is aligned with the protruding plane of the channel;

[0025] S22: Operate the instrument and acquire motion parameters: Operate the instrument and obtain the distance the instrument has moved and the angle of deflection relative to its initial position by reading the readings of the position sensor and the rotary encoder, respectively;

[0026] S23: Calculation of the position of the instrument head: Since the deflection of the instrument is a single-degree-of-freedom motion, the positional relationship of the instrument head relative to the instrument channel plane is determined based on the movement distance and the deflection angle.

[0027] S24: Determining the positional relationship between the instrument and the camera: The positional relationship between the instrument channel plane and the camera is fixed and known. The positional relationship between the instrument and the camera is obtained by the positional relationship between the instrument head end and the instrument channel plane.

[0028] Furthermore, in step S23, the positional relationship between the instrument head end and the instrument channel plane is a rotation matrix R and a translation vector t.

[0029] Furthermore, step S3 specifically includes the following steps:

[0030] S31: Find the nearest and farthest feature points: Read the camera position at the current moment, obtain the set of feature points extracted by camera localization, calculate the distance between all feature points and the camera, and obtain the farthest point and the nearest point to the camera respectively.

[0031] S32: Acquire camera view frustum:

[0032] S33: Find the intersection of the instrument extension lines: Connect the head end and the end end plane of the instrument, draw out a ray that intersects with the camera's visual cone, obtain two intersection points, and solve for the spatial coordinates of the two intersection points;

[0033] S34: Constructing the guide line: According to the camera projection equation, the three-dimensional coordinates of the two intersection points are projected onto the camera's pixel plane to obtain two-dimensional pixel coordinates. The two intersection points are then connected in the camera's imaging image to obtain the instrument guide line, thus achieving guidance.

[0034] Furthermore, step S32 specifically includes the following steps:

[0035] S321: Connect the camera point to the nearest and farthest points respectively, and take the planes perpendicular to the connecting lines as the nearest and farthest viewing cone planes.

[0036] S322: Calculate the angle θ between the nearest and farthest view cone planes by simultaneously rotating both planes in opposite directions. We obtain mutually parallel view cone planes;

[0037] S323: Draw rays from the camera's horizontal and vertical field of view, intersecting with the nearest / farthest view cone planes to obtain the near clipping plane and the far clipping plane. The middle part enclosed by the two clipping planes is the visible view cone space of the camera.

[0038] The second objective of this invention is achieved by the following technical solution:

[0039] An endoscopic instrument guidance system for implementing the aforementioned endoscopic instrument guidance method includes a camera positioning module, an instrument position detection module, and an instrument guidance module. The camera positioning module calculates and optimizes the camera pose based on images acquired by the camera. The instrument position detection module includes an encoder and a position sensor. The encoder obtains the angle of movement of the instrument relative to the instrument channel, and the position sensor obtains the distance of movement of the instrument relative to the instrument channel. The position of the instrument tip relative to the instrument channel is calculated using the angle and distance. The instrument guidance module connects the instrument tip and the channel plane in three-dimensional space, constructs an extension line, extends the instrument into the field of view of the camera, and projects the extension line onto the camera image to achieve instrument position prediction and guidance.

[0040] Compared to existing technologies, the endoscopic instrument guidance method of this invention extracts and matches features from the acquired endoscopic image frame sequence, and calculates the real-time position of the camera. Subsequently, when the doctor operates the instrument, the angle and extension distance of the instrument are obtained through sensors, and the relative positional relationship between the camera and the instrument is calculated. Finally, the overlapping part of the instrument and the camera's field of view is obtained in three-dimensional space and projected onto the corresponding camera image to achieve real-time guidance. Through the above steps, the extension position of the endoscopic instrument is obtained only through two additional sensors to achieve real-time guidance. The system is small in size and does not add extra burden to the equipment. The system has a fast processing speed, good real-time performance, and is compatible with different image resolutions from 480p to 1080p. It can support a maximum frame rate output of 60fps. The system has high position detection accuracy and good consistency, and can maintain high instrument positioning and guidance accuracy during long-term examinations / surgeries (more than 2 hours). Attached Figure Description

[0041] Figure 1 This is a flowchart of the endoscopic instrument guidance method of the present invention;

[0042] Figure 2for Figure 1 A flowchart for optimizing camera pose in endoscopic instrument guidance methods;

[0043] Figure 3 for Figure 1 Flowchart of instrument position detection and instrument guidance in endoscopic instrument guidance methods;

[0044] Figure 4 This is a schematic diagram of the front end structure of the endoscope of the present invention;

[0045] Figure 5 This is a schematic diagram of the instrument guidance method for endoscopes according to the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or it can be fixed through another intermediate component. When a component is said to be "connected to" another component, it can be directly connected to the other component or it may be fixed through another intermediate component. When a component is said to be "set on" another component, it can be set directly on the other component or it may be set through another intermediate component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0049] Figure 1 The endoscopic instrument guidance method of the present invention includes the following steps:

[0050] S1: The camera acquires multiple endoscopic images. Feature extraction and matching are performed on adjacent frames. The camera pose is calculated based on the matching results. After acquiring the camera pose, the spatial information of all feature points is obtained. The camera pose is optimized based on the spatial information.

[0051] S2: When operating the instrument, the angle and distance of the instrument's movement relative to the instrument channel are obtained through the sensor, thereby obtaining the position of the instrument head relative to the instrument channel and calculating the relative positional relationship between the camera and the instrument.

[0052] S3: Connect the head end of the instrument and the instrument channel plane in three-dimensional space, construct an extension line, extend the instrument into the field of view of the camera, and project the extension line onto the camera image to achieve prediction and guidance of the instrument position.

[0053] Steps S1 and S2 are performed in parallel, and step S1 is used throughout the entire endoscopic instrument guidance process to achieve real-time guidance and correction of the endoscopic instrument.

[0054] Please continue reading. Figure 2 Step S1 specifically includes the following steps:

[0055] S11: Camera initialization: Rotate and translate the initial lens to acquire two initial images. Use the camera coordinate system when acquiring the first image as the world coordinate system, which will be used as the coordinate system calculated in the endoscopic instrument guidance method.

[0056] S12: Feature extraction: Use the ORB algorithm to extract feature points in two adjacent frames of images and calculate the BRIEF descriptor;

[0057] S13: Use the Fast Approximate Nearest Neighbor (FLANN) algorithm to perform descriptor matching on all feature points to obtain feature point pairs;

[0058] S14: Camera motion solution: When the number of feature point pairs is less than the preset value, return to step S11. When the number of feature point pairs is greater than or equal to the preset value, solve the camera motion using 2D-2D epipolar constraints based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, and obtain the latest position of the camera.

[0059] S15: Obtain spatial information of feature points: Through triangulation, using the different 2D projections of the same spatial feature point in two consecutive frames, the spatial position of the feature point is obtained by solving the least squares method.

[0060] S16: Camera pose optimization: Store the obtained 3D spatial feature point information. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, the BA cost function is used to optimize the camera pose.

[0061] S17: Continue acquiring images, repeating steps S12 to S16 until endoscopic instrument guidance is complete.

[0062] Specifically, in step S14, the preset value is 8, and the camera motion is solved using 2D-2D epipolar constraints.

[0063] Please continue reading. Figure 3 as well as Figure 4 Step S2 specifically includes the following steps:

[0064] S21: Alignment of instrument with initial calculation plane: Insert the instrument into the instrument channel of the endoscope, read the position sensor reading during insertion, stop the operation when the reading reaches the internal length of the endoscope instrument channel, and reset the position sensor reading to zero to ensure that the instrument tip is aligned with the protruding plane of the channel;

[0065] S22: Operate the instrument and acquire motion parameters: Operate the instrument and obtain the distance the instrument has moved and the angle of deflection relative to its initial position by reading the readings of the position sensor and the rotary encoder, respectively;

[0066] S23: Calculation of the position of the instrument head: Since the deflection of the instrument is a single-degree-of-freedom motion, the positional relationship of the instrument head relative to the instrument channel plane is determined based on the movement distance and the deflection angle.

[0067] S24: Determining the positional relationship between the instrument and the camera: The positional relationship between the instrument channel plane and the camera is fixed and known. The positional relationship between the instrument and the camera is obtained by the positional relationship between the instrument head end and the instrument channel plane.

[0068] Specifically, in step S23, the positional relationship between the instrument head end and the instrument channel plane is the rotation matrix R and the translation vector t.

[0069] Please continue reading. Figure 5 Step S3 specifically includes the following steps:

[0070] S31: Find the nearest and farthest feature points: Read the camera position at the current moment, obtain the set of feature points extracted by camera localization, calculate the distance between all feature points and the camera, and obtain the farthest point and the nearest point to the camera respectively.

[0071] S32: Acquire camera view frustum:

[0072] S33: Find the intersection of the instrument extension lines: Connect the head end and the end end plane of the instrument, draw out a ray that intersects with the camera's visual cone, obtain two intersection points, and solve for the spatial coordinates of the two intersection points;

[0073] S34: Constructing the guide line: According to the camera projection equation, the three-dimensional coordinates of the two intersection points are projected onto the camera's pixel plane to obtain two-dimensional pixel coordinates. The two intersection points are then connected in the camera's imaging image to obtain the instrument guide line, thus achieving guidance.

[0074] Specifically, step S32 includes the following steps:

[0075] S321: Connect the camera point to the nearest and farthest points respectively, and take the planes perpendicular to the connecting lines as the nearest and farthest viewing cone planes.

[0076] S322: Calculate the angle θ between the nearest and farthest view cone planes by simultaneously rotating both planes in opposite directions. We obtain mutually parallel view cone planes;

[0077] S323: Draw rays from the camera's horizontal and vertical field of view, intersecting with the nearest / farthest view cone planes to obtain the near clipping plane and the far clipping plane. The middle part enclosed by the two clipping planes is the visible view cone space of the camera.

[0078] This invention also relates to an endoscopic instrument guidance system for implementing the aforementioned endoscopic instrument guidance method, comprising a camera positioning module, an instrument position detection module, and an instrument guidance module. The camera positioning module calculates and optimizes the camera pose based on images acquired by the camera. The instrument position detection module includes an encoder and a position sensor. The encoder obtains the angle of movement of the instrument relative to the instrument channel, and the position sensor obtains the distance of movement of the instrument relative to the instrument channel. The position of the instrument tip relative to the instrument channel is calculated using the angle and distance. The instrument guidance module connects the instrument tip and the channel plane in three-dimensional space, constructs an extension line, extends the instrument into the field of view of the camera, and projects the extension line onto the camera image to achieve instrument position prediction and guidance.

[0079] This invention provides an endoscopic instrument guidance method that extracts and matches features from acquired endoscopic image frame sequences, calculating the real-time position of the camera. Subsequently, while the doctor operates the instrument, sensors obtain the instrument's angle and extension distance, calculating the relative positional relationship between the camera and the instrument. Finally, the overlapping portion of the instrument's and camera's field of view is obtained in three-dimensional space and projected onto the corresponding camera image to achieve real-time guidance. Through these steps, only two additional sensors are needed to obtain the extension position of the endoscopic instrument, achieving real-time guidance. The system is compact, does not add extra burden to the equipment, has a fast processing speed, good real-time performance, is compatible with different image resolutions from 480p to 1080p, supports a maximum frame rate output of 60fps, and exhibits high position detection accuracy and good consistency. It maintains high instrument positioning and guidance accuracy even during long examinations / surgeries (over 2 hours).

[0080] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These are all equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of these fall within the protection scope of the present invention.

Claims

1. A method for guiding endoscopic instruments, characterized in that, Includes the following steps: S1: The camera acquires multiple endoscopic images. Feature extraction and matching are performed on adjacent frames. The camera pose is calculated based on the matching results. After acquiring the camera pose, the spatial information of all feature points is obtained. The camera pose is optimized based on the spatial information. S2: When operating the instrument, the distance the instrument moves and the angle of deflection relative to the initial position are obtained by reading the readings of the position sensor and the rotary encoder, respectively, thereby obtaining the position of the instrument head relative to the instrument channel and calculating the relative positional relationship between the camera and the instrument. S3: Connect the instrument head and the instrument channel plane in three-dimensional space, construct an extension line to extend the instrument into the camera's field of view, and project the extension line onto the camera image to predict and guide the instrument's position; Step S3 specifically includes the following steps: S31: Find the nearest and farthest feature points: Read the camera position at the current moment, obtain the set of feature points extracted by camera localization, calculate the distance between all feature points and the camera, and obtain the farthest point and the nearest point to the camera respectively. S32: Acquire camera view frustum: S33: Find the intersection of the instrument extension lines: Connect the head end and the end end plane of the instrument, draw out a ray that intersects with the camera's visual cone, obtain two intersection points, and solve for the spatial coordinates of the two intersection points; S34: Constructing the guide line: According to the camera projection equation, the three-dimensional coordinates of the two intersection points are projected onto the camera's pixel plane to obtain two-dimensional pixel coordinates. The two intersection points are then connected in the camera's imaging image to obtain the instrument guide line, thus achieving guidance.

2. The endoscopic instrument guidance method according to claim 1, characterized in that: Steps S1 and S2 are performed in parallel, and step S1 is performed throughout the entire endoscopic instrument guidance process.

3. The endoscopic instrument guidance method according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Camera initialization: Rotate and translate the initial lens to acquire two initial frames; S12: Feature extraction: Extract feature points from two adjacent frames of images and calculate the BRIEF descriptor; S13: Perform descriptor matching on all feature points to obtain feature point pairs; S14: Camera motion solution: When the number of feature point pairs is less than the preset value, return to step S11. When the number of feature point pairs is greater than or equal to the preset value, solve the camera motion based on the feature point pairs to obtain the camera rotation matrix R and translation matrix t, and obtain the latest position of the camera. S15: Obtain spatial information of feature points: Through triangulation, using the different 2D projections of the same spatial feature point in two consecutive frames, the spatial position of the feature point is obtained by solving the least squares method. S16: Camera pose optimization: Store the obtained 3D spatial feature point information. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, camera pose optimization is performed. S17: Continue acquiring images, repeating steps S12 to S16 until endoscopic instrument guidance is complete.

4. The endoscopic instrument guidance method according to claim 3, characterized in that: In step S11, the camera coordinate system when the first frame image is acquired is used as the world coordinate system, which is also used as the coordinate system calculated in the endoscopic instrument guidance method.

5. The endoscopic instrument guidance method according to claim 3, characterized in that: In step S14, the preset value is 8, and the camera motion is solved using 2D-2D epipolar constraints.

6. The endoscopic instrument guidance method according to claim 1, characterized in that: Step S2 specifically includes the following steps: S21: Alignment of instrument with initial calculation plane: Insert the instrument into the instrument channel of the endoscope, read the position sensor reading during insertion, stop the operation when the reading reaches the internal length of the endoscope instrument channel, and reset the position sensor reading to zero to ensure that the instrument tip is aligned with the protruding plane of the channel; S22: Operate the instrument and acquire motion parameters: Operate the instrument and obtain the distance the instrument has moved and the angle of deflection relative to its initial position by reading the readings of the position sensor and the rotary encoder, respectively; S23: Calculation of the position of the instrument head: Since the deflection of the instrument is a single-degree-of-freedom motion, the positional relationship of the instrument head relative to the instrument channel plane is determined based on the movement distance and the deflection angle. S24: Determining the positional relationship between the instrument and the camera: The positional relationship between the instrument channel plane and the camera is fixed and known. The positional relationship between the instrument and the camera is obtained by the positional relationship between the instrument head end and the instrument channel plane.

7. The endoscopic instrument guidance method according to claim 6, characterized in that: In step S23, the positional relationship between the instrument head end and the instrument channel plane is the rotation matrix R and the translation vector t.

8. The endoscopic instrument guidance method according to claim 1, characterized in that: Step S32 specifically includes the following steps: S321: Connect the camera point to the nearest and farthest points respectively, and take the planes perpendicular to the connecting lines as the nearest and farthest viewing cone planes. S322: Calculate the angle θ between the nearest and farthest view cone planes by simultaneously rotating both planes in opposite directions. This results in mutually parallel frustum planes; S323: Draw rays from the camera's horizontal and vertical field of view, intersecting with the nearest / farthest view cone planes to obtain the near clipping plane and the far clipping plane. The middle part enclosed by the two clipping planes is the visible view cone space of the camera.

9. An endoscopic instrument guidance system for implementing the endoscopic instrument guidance method as described in any one of claims 1-8, characterized in that: The system includes a camera positioning module, an instrument position detection module, and an instrument guidance module. The camera positioning module calculates and optimizes the camera pose based on images captured by the camera. The instrument position detection module includes an encoder and a position sensor. The encoder obtains the angle of movement of the instrument relative to the instrument channel, and the position sensor obtains the distance of movement of the instrument relative to the instrument channel. The position of the instrument tip relative to the instrument channel is calculated using the angle and distance. The instrument guidance module connects the instrument head and the channel plane in three-dimensional space, constructs an extension line, extends the instrument into the field of view of the camera, and projects the extension line onto the camera image to predict and guide the instrument position.

Images