Real-time operation sampling pen positioning method based on single RGB camera

Abstract

The invention discloses a real-time operation sampling pen positioning method based on a single RGB camera, and the method comprises the following steps: 1, enabling a camera to shoot an initial image of an operation region in a close range before an operation; after the operation is started, the camera is pulled far to a position suitable for the operation, and the current image of the operation is shot in real time; 2, in the real-time operation process, the computer processing unit continuously detects the small markers on the sampling pen through the camera, and the position of the pen point of the sampling pen is calculated in real time based on the perspective transformation matrix; 3, in the operation process, a two-step method is adopted to avoid interruption of continuous tracking due to shielding of the tip of the sampling pen. According to the method, the real-time feedback capability of the operation is improved, and the positioning precision of the sampling pen is kept.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular is a real-time surgical sampling pen positioning method based on a single RGB camera. Background Art

[0002] At present, in solid tumor resection surgery, how to accurately determine the tumor boundary is the key to the success of the operation. Most of the existing surgical navigation technologies rely on multi-camera arrays, depth cameras, or radiological imaging technology (such as CT, MRI, etc.) combined with real-time tracking algorithms to locate surgical instruments. The advantages of these methods are that they can provide high spatial resolution and accurate instrument positioning. However, such technologies usually face the following challenges and shortcomings:

[0003] 1. Multi-camera array: Although the multi-camera system can achieve relatively accurate three-dimensional positioning, its installation is complex, costly, and has high requirements for the operating room environment. The viewing angles of multiple cameras may be blocked, making it difficult to continuously track surgical instruments. (Wang J, Song S, Ren H, et al. Surgical instrument tracking by multiple monocular modules and a sensor fusion approach [J]. IEEE Transactions on Automation Science and Engineering, 2018, 16 (2): 629-639).

[0004] 2. Depth camera technology: Depth camera technology uses infrared or structured light to capture depth information and can measure the spatial position of the instrument more accurately. However, this technology has high requirements on equipment cost and real-time performance, especially in surgical environments with drastic changes in light, which may affect positioning accuracy. (Pei-Lun Y, Chun-Sheng H, Yu-Qing H, et al. Surgical navigation system based on the visual object tracking algorithm [C] / / 2018 4th Annual International Conference on Network and Information Systems for Computers (ICNISC). IEEE, 2018: 160-164).

[0005] 3. Image-based surgical navigation system: Currently, some surgical navigation systems rely on CT or MRI images for surgical planning and real-time guidance. However, such imaging systems require a lot of data preprocessing, rely on preoperative images, cannot quickly respond to changes in instrument position during surgery, and lack real-time performance.

[0006] 4. Deep learning surgical navigation: In recent years, surgical navigation technology based on deep learning has developed, but such methods rely on large-scale medical data for training. However, the acquisition of medical imaging data is often limited by ethical issues and equipment costs, and the universality of deep learning models is limited due to the diversity of surgical instruments. The computing resources required for model training and reasoning are also relatively large, and it is not easy to deploy in real-time surgical scenarios. (Neves CA, Tran ED, Blevins NH, et al. Deep learning automated segmentation of middle skull-base structures for enhanced navigation [C] / / International Forum of Allergy & Rhinology. 2021, 11 (12)). Summary of the invention

[0007] The purpose of the present invention is to solve the problems of high equipment cost, insufficient real-time performance, and complex multi-camera or depth camera settings in existing surgical navigation technology. Although the existing navigation system can achieve relatively accurate positioning in the surgical scene, it has high environmental requirements during deployment and operation, and cannot be economically and efficiently applied to various types of surgical scenes. In particular, there is a problem of unstable accuracy when the sampling pen is tilted at a large angle or blocked by the sampling pen. A real-time surgical sampling pen positioning method based on a single RGB camera is provided. This method can improve the real-time feedback capability of the surgery and maintain the positioning accuracy of the sampling pen.

[0008] The technical solution for achieving the purpose of the present invention is:

[0009] A real-time surgical sampling pen positioning method based on a single RGB camera includes a positioning system, the system includes the following components: an RGB camera for real-time shooting, a computer processing unit and a sampling pen, a small marker is attached to the sampling pen, the camera is connected to the computer processing unit by wire, and collects surgical images in real time and transmits image data to the computer processing unit for processing, the sampling pen does not need to be directly connected to the computer processing unit or the camera hardware, but is positioned by the position change of the small marker, the camera is placed in an appropriate position to shoot the surgical area, so as to ensure that the images can be collected and transmitted uninterruptedly during the operation, the small marker must be located in the image shot by the camera, so as to achieve accurate sampling pen positioning, the method includes the following steps:

[0010] Step 1: Before the operation begins, the camera takes an initial image of the operation area at close range to ensure that the initial image clearly covers the entire operation area. This initial image serves as a reference for subsequent positioning calibration. After the operation begins, the camera is moved away to a position suitable for the operation to avoid obstructing the doctor's operation, and the current image of the operation is taken in real time.

[0011] Step 2: During real-time operation, the computer processing unit continuously detects the small marker on the sampling pen through the camera and calculates the position of the sampling pen tip in real time based on the perspective transformation matrix. The specific process is as follows:

[0012] Starting with image preprocessing, the video frames in the video acquired by the camera are expanded, and a 10-pixel border is added on both sides of each extracted video frame to prevent edge detection problems; the video frame image is converted into a grayscale image, and then threshold processing is performed to generate a binary image for contour detection. Contour lines are detected from the binary image to find nested contour lines that match the positioning mark pattern. The positioning mark pattern consists of a bull's eye pattern - a large blank area is surrounded by a dark outer circle, that is, each positioning mark pattern consists of three nested squares, and each nested square has a different contour length; the inner and outer nested contour hierarchy of the positioning mark pattern itself helps to separate the positioning mark pattern from other image features. According to the design principles of the QR code, the expected ratio of the perimeter of the outermost nested square to the innermost nested square of the positioning mark pattern is 2 to 3:1. For each detected mark point, the computer processing unit uses the geometric moment method to calculate the coordinates of the center point of the mark point. The calculation of the geometric moment method is as follows:

[0013]

[0014] Where Ci is the coordinate of the center point of the i-th marker pattern, I(x, y) is the intensity of the pixel point (x, y) in the image, and the computer processing unit can accurately calculate the coordinate of the center point of each positioning marker pattern through the calculation of the geometric moment method;

[0015] The computer processing unit solves the perspective transformation matrix M according to the coordinates of the four detected marking points, so as to transform the image plane under the camera's viewing angle into the reference square coordinate system. Assuming that the coordinates of the center points of the four positioning marking patterns in the camera image are P_1(x_1, y_1), P_2(x_2, y_2), P_3(x_3, y_3) and P_4(x_4, y_4), the side length of the reference square is a, and a is determined by the distance between the center points of two adjacent positioning marking patterns. The corresponding points of the center points of the four positioning marking patterns in the reference square coordinate system are (0, 0), (a, 0), (a, a) and (0, a). The perspective transformation matrix M is calculated by the following equation:

[0016]

[0017] Among them, (x i ,y i ) and (x i ′ ,y i ′ ) are the coordinates of the marked points in the camera image and the corresponding coordinates in the reference coordinate system, respectively. The matrix M is solved by the correspondence between the known marked points in the camera image and the corresponding points in the reference coordinate system to generate a perspective transformation matrix;

[0018] Through the matrix M, the computer processing unit can map the coordinates of the small marker in the image to the reference square coordinate system, and calculate the actual position P of the sampling pen tip in the real coordinate system according to the relative position of the small marker after the perspective transformation. real , the formula is as follows:

[0019] P real =(0.5a,n×a)

[0020] Where n is the ratio coefficient of the length of the sampling pen tip to the side length a of the small marker;

[0021] Then apply the inverse of the perspective transformation matrix M -1 Further obtain the image coordinates P of the sampling pen tip under the current camera viewing angle img , the calculation formula is as follows:

[0022] P img =M -1 ·P real

[0023] Finally, the calculated sampling pen tip image coordinates P img After further smoothing by EMA filter, the error caused by hand shaking is reduced. img (t) is the coordinate of the sampling pen tip after the current filtering. The filtering formula is as follows:

[0024] P′ img (t) = α·P img (t)+(1-α)·P′ img (t-1)

[0025] Among them, α is the smoothing coefficient. Through this filtering process, the computer processing unit can stably track the position of the sampling pen tip in a noisy environment and provide real-time feedback;

[0026] Step 3: During the operation, a two-step method was used to avoid interruption of continuous tracking due to occlusion of the tip of the sampling pen. The first step of the two-step method was to capture an initial image covering the entire surgical area before the operation began. This image will serve as a reference for subsequent positioning.

[0027] The second step is to manually mark at least four pairs of feature points in the current camera image that correspond to those in the initial image during the operation. Through these feature point pairs, the perspective transformation matrix H is calculated to map the camera's view to the initial image. The perspective transformation matrix is ​​obtained by solving the transformation relationship between the feature points in the two views. For each pair of feature points, let the coordinates in the camera image be (x cam ,y cam ), the corresponding coordinates in the initial image are (x init ,y init ), then the transformation relationship between the coordinates in the camera image and the corresponding coordinates in the initial image is expressed as:

[0028]

[0029] By solving a set of linear equations, H minimizes the reprojection error of all point pairs. After obtaining H, the position P of the tip of the sampling pen in the camera image can be obtained. cam (t) The position P of the tip of the sampling pen mapped to the initial image init (t):

[0030] P init (t) = H -1 P cam (t)

[0031] This ensures that even if the camera cannot directly observe the tip of the sampling pen due to occlusion or angle problems, the computer processing unit can still infer the actual position of the tip of the sampling pen based on the previously captured initial image, thereby ensuring continuous position information tracking during the operation.

[0032] The positioning system of this technical solution relies on a single RGB camera and a small marker attached to the sampling pen. By calculating the perspective transformation matrix, the surgical image captured by the camera is mapped to the actual position of the tip of the sampling pen. The method of this technical solution uses grayscale threshold segmentation, edge detection and contour analysis technology to detect the position of the marker on the sampling pen. Through the geometric features of the small marker, the positioning system can track the spatial position of the instrument in real time, and use exponential moving average filtering (EMA) to suppress the jitter caused by hand tremors, further improving the stability of the positioning of the tip of the sampling pen.

[0033] The specific effects of this technical solution are:

[0034] 1. Low cost and high adaptability: This technical solution method only uses a single RGB camera, avoiding the high cost and complex deployment of multi-camera or depth camera systems, greatly reducing equipment costs, and has strong adaptability and can be widely used in various surgical scenarios.

[0035] 2. High-precision real-time feedback: By using perspective transformation matrix and geometric moment method, the computer processing unit can accurately locate the tip of the sampling pen in complex surgical scenes. When displayed in the current image, the error is kept within 1.525 mm, and when perspective is returned to the initial image, the error is within 2.525 mm, which meets the surgical requirements.

[0036] 3. Suppress jitter: The EMA filtering algorithm is used to effectively suppress the high-frequency noise caused by hand jitter, ensuring stable real-time feedback during the operation.

[0037] 4. Solve the occlusion problem: Through initial image registration, this technical solution method can solve the positioning error caused by instrument occlusion or angle change, and ensure continuous tracking of the sampling pen in the surgical scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of patch samples and specifications of positioning marks in the embodiment;

[0039] Figure 2 It is a schematic diagram of the proportions of the sampling pen, the sampling pen tip, and the positioning patch in the embodiment;

[0040] Figure 3 Schematic diagram of the framework and image preprocessing flow of the embodiment, a is the framework diagram, and b is the image preprocessing flow chart. DETAILED DESCRIPTION

[0041] The content of the present invention is further described below in conjunction with the drawings and embodiments, but the present invention is not limited thereto.

[0042] Example:

[0043] Reference Figure 3 A real-time surgical sampling pen positioning method based on a single RGB camera includes a positioning system, the system includes the following components: an RGB camera for real-time shooting, a computer processing unit and a sampling pen, a small marker is attached to the sampling pen, the camera is connected to the computer processing unit by wire, and collects surgical images in real time and transmits image data to the computer processing unit for processing, the sampling pen does not need to be directly connected to the computer processing unit or the camera hardware, but is positioned by the position change of the small marker, the camera is usually placed in an appropriate position to capture the surgical area to ensure that the image can be continuously collected and transmitted during the operation, the small marker must be located in the image captured by the camera to achieve accurate sampling pen positioning, the method includes the following steps, such as Figure 3 As shown in a:

[0044] Step 1: Before the operation begins, the camera takes an initial image of the operation area at close range to ensure that the initial image clearly covers the entire operation area. This initial image serves as a reference for subsequent positioning calibration. After the operation begins, the camera is moved away to a position suitable for the operation to avoid obstructing the doctor's operation, and the current image of the operation is taken in real time.

[0045] Step 2: During real-time operation, the computer processing unit continuously detects the small marker on the sampling pen through the camera and calculates the position of the sampling pen tip in real time based on the perspective transformation matrix. The specific process is as follows:

[0046] like Figure 3 As shown in FIG. 2 , starting from image preprocessing, the video frames in the video acquired by the camera are expanded, and a 10-pixel border is added on both sides of each extracted video frame to prevent edge detection problems; the video frame image is converted into a grayscale image, and then threshold processing is performed to generate a binary image for contour detection. Contour lines are detected from the binary image to find nested contour lines that match the positioning mark pattern. The positioning mark pattern consists of a bull's eye pattern - a large blank area is surrounded by a central mark, and the blank area itself is surrounded by a dark outer circle, that is, each positioning mark pattern consists of three nested squares, and each nested square has a different contour length; the inner and outer nested contour hierarchy of the positioning mark pattern itself helps to separate the positioning mark pattern from other image features. In this example, four positioning mark patterns are set on the small marker. The specifications of the small marker sample and the positioning mark pattern are shown in FIG. Figure 1As shown, CA, CB, and CC represent the contour lines corresponding to the outermost, middle, and innermost squares, respectively. The hierarchical relationship is verified by checking the perimeter ratio of CA and CC. According to the design principle of the QR code, the expected ratio of the perimeter of the outermost nested square to the innermost nested square of the mark is 2 to 3:1. For each detected mark point, the computer processing unit uses the geometric moment method to calculate the coordinates of the center point of the mark point. The calculation of the geometric moment method is as follows:

[0047]

[0048] Where Ci is the coordinate of the center point of the i-th marker, and I(x, y) is the intensity of the pixel point (x, y) in the image. Through the calculation of the geometric moment method, the computer processing unit can accurately calculate the coordinates of the center point of each marker pattern;

[0049] The computer processing unit solves the perspective transformation matrix M based on the coordinates of the four detected marking points, so as to transform the image plane under the camera's viewing angle into the reference square coordinate system. Assume that the coordinates of the four marking points in the camera image are P_1(x_1, y_1), P_2(x_2, y_2), P_3(x_3, y_3) and P_4(x_4, y_4), and the side length of the reference square is a, which is determined by the distance between the center points of two adjacent positioning marking patterns. In this embodiment, a is 3 cm. The placement of the marking points and the small marking objects and the sampling pen is as follows: Figure 2 As shown, the corresponding points of the center points of the four positioning mark patterns in the reference square coordinate system are (0,0), (a,0), (a,a) and (0,a), and the perspective transformation matrix M is calculated by the following equation:

[0050]

[0051] Among them, (x i ,y i ) and (x i ′ ,y i ′ ) are the coordinates of the marked points in the camera image and the corresponding coordinates in the reference coordinate system, respectively. The matrix M is solved by the correspondence between the known marked points in the camera image and the corresponding points in the reference coordinate system to generate a perspective transformation matrix;

[0052] Through the matrix M, the computer processing unit can map the coordinates of the small marker in the image to the reference square coordinate system, and calculate the actual position P of the sampling pen tip in the real coordinate system according to the relative position of the small marker after the perspective transformation. real , the formula is as follows:

[0053] P real=(0.5a,n×a)

[0054] Wherein, n is the ratio coefficient of the length of the sampling pen tip to the side length a of the small marker, and in this embodiment, n is 6.3;

[0055] Then apply the inverse of the perspective transformation matrix M -1 Further obtain the image coordinates P of the sampling pen tip under the current camera viewing angle img , the calculation formula is as follows:

[0056] P img =M -1 ·P real

[0057] Finally, the calculated sampling pen tip image coordinates P img After further smoothing by EMA filter, the error caused by hand shaking is reduced. img (t) is the coordinate of the sampling pen tip after the current filtering. The filtering formula is as follows:

[0058] P′ img (t) = α·P img (t)+(1-α)·P′ img (t-1)

[0059] Among them, α is the smoothing coefficient, which is set to 0.11 in the experiment. Through this filtering process, the computer processing unit can stably track the position of the sampling pen tip in a noisy environment and provide real-time feedback;

[0060] Step 3: During the operation, a two-step method was used to avoid interruption of continuous tracking due to occlusion of the tip of the sampling pen. The first step of the two-step method was to capture an initial image covering the entire surgical area before the operation began. This image will serve as a reference for subsequent positioning.

[0061] The second step is to manually mark at least four pairs of feature points in the current camera image that correspond to those in the initial image during the operation. Through these feature point pairs, the perspective transformation matrix H is calculated to map the camera's view to the initial image. The perspective transformation matrix is ​​obtained by solving the transformation relationship between the feature points in the two views. For each pair of feature points, let the coordinates in the camera image be (x cam ,y cam ), the corresponding coordinates in the initial image are (x init ,y init ), then the transformation relationship between the coordinates in the camera image and the corresponding coordinates in the initial image is expressed as:

[0062]

[0063] By solving a set of linear equations, H minimizes the reprojection error of all point pairs. After obtaining H, the position P of the tip of the sampling pen in the camera image can be obtained. cam (t) The position P of the tip of the sampling pen mapped to the initial image init (t):

[0064] P init (t) = H -1 P cam (t)

[0065] This ensures that even if the camera cannot directly observe the tip of the sampling pen due to occlusion or angle problems, the computer processing unit can still infer the actual position of the tip of the sampling pen based on the previously captured initial image, thereby ensuring continuous position information tracking during the operation.

[0066] The above is only a preferred implementation mode of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time surgical sampling pen positioning method based on a single RGB camera, characterized in that: The method comprises a positioning system, which comprises the following components: an RGB camera for real-time shooting, a computer processing unit and a sampling pen, a small marker is attached to the sampling pen, the camera is connected to the computer processing unit by wire, and collects the operation picture in real time and transmits the collected image data to the computer processing unit for processing, the sampling pen does not need to be directly connected to the computer or camera hardware, but is positioned by the position change of the small marker, the camera is placed in a position where the operation area can be photographed to ensure that the picture can be collected and transmitted uninterruptedly during the operation, the small marker must be located in the picture taken by the camera to achieve accurate positioning of the sampling pen, and the method comprises the following steps: Step 1: Before the operation begins, the camera takes an initial image of the operation area at close range to ensure that the initial image clearly covers the entire operation area. This initial image serves as a reference for subsequent positioning calibration. After the operation begins, the camera is moved away to a position suitable for the operation to avoid obstructing the doctor's operation, and the current image of the operation is taken in real time. Step 2: During the real-time operation, the computer processing unit continuously detects the small marker on the sampling pen through the camera and calculates the position of the tip of the sampling pen in real time based on the perspective transformation matrix; Step 3: During the operation, a two-step method was used to avoid interruption of continuous tracking due to occlusion of the sampling pen tip. The first step of the two-step method was to capture an initial image covering the entire surgical area before the operation began. This image will serve as a reference for subsequent positioning. The second step is to manually mark at least four pairs of feature points in the current camera image that correspond to the initial image during the operation, align the real-time camera view with the initial image, and calculate the perspective transformation matrix H through these feature point pairs. The camera's view is mapped to the initial image to infer the position of the sampling pen tip, thereby ensuring continuous position information tracking during the operation.

2. The real-time surgical sampling pen positioning method based on a single RGB camera according to claim 1, characterized in that: In step 2, the computer processing unit continuously detects the small marker on the sampling pen through the camera, and calculates the position of the tip of the sampling pen in real time based on the perspective transformation matrix. The specific process is as follows: Starting with image preprocessing, the video frames in the video captured by the camera are expanded, and a 10-pixel border is added on both sides of each extracted video frame; the video frame image is converted into a grayscale image, and then thresholded to generate a binary image for contour detection. Contour lines are detected from the binary image to find nested contour lines that match the positioning mark pattern. The positioning mark pattern consists of a bull's eye pattern - a large blank area around the central mark, which is itself surrounded by a dark outer circle, that is, each positioning mark pattern consists of three nested squares, and each nested square has a different contour length; the inner and outer nested contour hierarchy of the mark itself helps to separate the positioning mark pattern from other image features. According to the design principles of the QR code, the expected ratio of the perimeter of the outermost nested square of the mark to the innermost nested square is 2 to 3:

1. For each detected mark point, the computer processing unit uses the geometric moment method to calculate the coordinates of the center point of the mark point. The calculation of the geometric moment method is as follows: Among them, C i is the coordinate of the center point of the i-th mark, I(x,y) is the intensity of the pixel point (x,y) in the image, and the computer processing unit can accurately calculate the coordinate of the center point of each positioning mark pattern through the calculation of the geometric moment method; The computer processing unit solves the perspective transformation matrix M based on the coordinates of the four detected marker points so as to transform the image plane under the camera's viewing angle into the reference square coordinate system. Assuming that the coordinates of the centers of the four positioning marker patterns in the camera image are P_1(x_1, y_1), P_2(x_2, y_2), P_3(x_3, y_3) and P_4(x_4, y_4), the side length of the reference square is a, and a is determined by the distance between the center points of two adjacent positioning markers. The corresponding points of the centers of the four positioning marker patterns in the reference square coordinate system are (0, 0), (a, 0), (a, a) and (0, a). The perspective transformation matrix M is calculated by the following equation: Among them, (x i ,y i ) and (x′ i ,y′ i ) are the coordinates of the marked points in the camera image and the corresponding coordinates in the reference coordinate system, respectively. The matrix M is solved by the correspondence between the known marked points in the camera image and the corresponding points in the reference coordinate system to generate a perspective transformation matrix; Through the matrix M, the computer processing unit can map the coordinates of the small marker in the image to the reference square coordinate system, and calculate the actual position P of the sampling pen tip in the real coordinate system according to the relative position of the small marker after the perspective transformation. real , the formula is as follows: P real =(0.5a,n×a) Where n is the ratio coefficient of the length of the sampling pen tip to the side length a of the small marker; Then apply the inverse of the perspective transformation matrix M -1 Further obtain the image coordinates P of the sampling pen tip under the current camera viewing angle img , the calculation formula is as follows: P img =M -1 ·P real Finally, the calculated sampling pen tip image coordinates P img After further smoothing by EMA filter, the error caused by hand shaking is reduced. img (t) is the coordinate of the sampling pen tip after the current filtering. The filtering formula is as follows: P′ img (t)=α·P img (t)+(1-α)·P′ img (t-1) Among them, α is the smoothing coefficient. Through this filtering process, the computer processing unit can stably track the position of the sampling pen tip in a noisy environment and provide real-time feedback.

3. The real-time surgical sampling pen positioning method based on a single RGB camera according to claim 1, characterized in that: The specific process of the second step of the two-step method in step 3 is: The second step is to manually mark at least four pairs of feature points in the current camera image that correspond to the initial image during the operation, and align the real-time camera view with the initial image. Through these feature point pairs, the perspective transformation matrix H is calculated to map the camera's view to the initial image. The perspective transformation matrix is ​​obtained by solving the transformation relationship between the feature points in the two views. For each pair of feature points, let the coordinates in the camera image be (x cam ,y cam ), the corresponding coordinates in the initial image are (x init ,y inii ), then the transformation relationship between the coordinates in the camera image and the corresponding coordinates in the initial image is expressed as: By solving a set of linear equations, H minimizes the reprojection error of all point pairs. After obtaining H, the position P of the tip of the sampling pen in the camera image can be obtained. cam (t) The position P of the tip of the sampling pen mapped to the initial image init (t): P init (t)=H -1 P cam (t) This ensures that even if the camera cannot directly observe the tip of the sampling pen due to occlusion or angle problems, the computer processing unit can still infer the actual position of the tip of the sampling pen based on the previously captured initial image, thereby ensuring continuous position information tracking during the operation.

Images