An efficient surgery monitoring system combining endoscope and exoscope

Abstract

The application discloses a kind of inside and outside mirror fusion's high efficiency surgical monitoring system, including camera module, light source module, power management module, data storage module, image processing module, image display module, support and controller module;Camera module includes outside view mirror camera module and endoscope camera module, light source module is used to provide light source for power management module, image processing module is used to handle the image data obtained by camera module, splicing fusion endoscope and outside view mirror partition image, and present on image display module, controller module is used to coordinate the work of endoscope camera module and outside view mirror camera module.The application can selectively use endoscope and outside view mirror by device, while supporting the simultaneous use of endoscope and outside view mirror in some special scenarios.In addition, the application also provides the position positioning function of endoscope image, facilitates medical staff to adjust endoscope position and observation angle, indicates the specific position of endoscope image in outside view mirror surgical area, greatly expands surgical field of view and angle.

Description

Technical Field

[0001] The invention belongs to the technical field of medical devices and relates to a high-efficiency surgical monitoring system integrating endoscope and endoscope. Background Art

[0002] In recent years, endoscopes have been widely used in neurosurgery due to their inherent advantages, such as a wide viewing angle, a free visual axis, and minimal surgical trauma. However, when using an endoscope alone, neurovascular structures may not be visible through the endoscope as they pass through them. Therefore, during endoscopic surgery, there is a risk of damage to neurovascular structures outside the scope of the endoscope.

[0003] Exoscopic systems were developed to address the shortcomings of surgical microscopes and neuroendoscopy in clinical applications. With their wider field of view, longer focal length, more comfortable operation, and smaller size and weight, they are gradually becoming a viable alternative to traditional surgical microscopes. To address complex surgical environments, a device that integrates endoscopy and endoscopy is urgently needed. Summary of the Invention

[0004] When using an exoscopic system, the exoscopic system transmits images of the surgical area to a display in real time via a camera for the doctor to observe and operate. However, the images of the surgical area obtained have blind spots due to occlusion by organs or tissues. During the movement of the endoscope, the corresponding position information of the local surgical area image obtained in the global image cannot be displayed. To solve the above problems, the present invention discloses an efficient surgical monitoring system that integrates endoscopes and endoscopes, which can obtain more comprehensive image information during the operation and meet the requirements of modern surgical operations.

[0005] The specific plan is as follows:

[0006] An efficient surgical monitoring system integrating endoscopes and endoscopes is characterized in that it includes a camera module, a light source module, a power management module, a data storage module, an image processing module, an image display module, a bracket and a controller module; the camera module includes an exoscopic camera module and an endoscopic camera module, the light source module is used to provide light to the power management module, the image processing module is used to process the image data obtained by the camera module and present it on the image display module, and the controller module is used to coordinate the operation of the endoscopic camera module and the exoscopic camera module.

[0007] Furthermore, the camera module includes an exoscopic camera module and an endoscopic camera module; the exoscopic camera module is connected to the bracket and is used to obtain image information in the surgical field of view, including at least two imaging optical paths and an image sensor, and the image sensor is placed behind the imaging optical path; the endoscopic camera module is used to detect the front end and side conditions of the endoscope.

[0008] Furthermore, the imaging optical path includes three movable optical modules, namely the magnification group, the compensation group and the focusing group. The focusing group realizes the continuous adjustable change of the working distance, the magnification group and the focusing group cooperate to realize the change of the overall optical magnification, and the compensation group is used to maintain the clarity of the image during the zooming process; the endoscope camera module is equipped with a main view and a side view dual camera. The main view camera is used to detect the front end of the endoscope, and the side view camera is used to detect the side of the endoscope.

[0009] Furthermore, the light source module includes a lighting system and a light source control unit; the lighting system includes a light source and a light guide, the light source is connected to the endoscope and the exoscope respectively through the light guide, and transmits light to the front end of the endoscope and the exoscope camera module, and the light source control unit is used to provide light source brightness adjustment, spectrum adjustment, light source stability control and uniform lighting detection functions for the endoscope and the exoscope.

[0010] Furthermore, the power management module is used to provide and manage the power supply of each module of the system.

[0011] Furthermore, the data storage module is used to store video data and sensor data.

[0012] Furthermore, the image processing module is used to provide position navigation during surgery. When the exoscope and endoscope work together, the exoscope camera module obtains the image of the surgical area and transmits it to the image processing module, and the image processing module performs feature detection through an algorithm; when the endoscope moves, the image obtained by the endoscope camera module is transmitted to the image processing module for feature recognition; the image processing module partitions the global surgical area image obtained by the exoscope, and performs feature matching analysis on the feature information of the endoscope image with each area of ​​the exoscope image; after completing the feature matching, the image processing module presents the images of the endoscope and the exoscope on the image display module in a spliced, fused and expanded form.

[0013] Furthermore, the feature matching algorithm performs feature recognition and matching on the endoscope and exterior mirror images. When the matching degree is high, the image processing module obtains the endoscope image and the corresponding exterior mirror partition image.

[0014] Furthermore, the image fusion algorithm stitches and fuses the endoscopic image and the corresponding exoscopic partition image in an extended form and presents them on the image display module.

[0015] Furthermore, the image display module includes a medical display screen, which is used to selectively output exoscopic or endoscopic images according to actual working conditions, or to display them on the screen simultaneously when the endoscope and exoscopic images are working.

[0016] Furthermore, the bracket is used to adjust or fix the angle of the exterior mirror, and includes a robotic arm and a robotic arm control module. The robotic arm includes a robotic arm, a robotic arm, and a camera arm connected in sequence, and each robotic arm is connected through a mechanical joint. The robotic arm control module provides motion control and posture adjustment of the robotic arm.

[0017] Furthermore, the controller module is used to coordinate the work of the endoscope camera module and the exoscope camera module. The controller module allows doctors to use it in two modes. The first mode is to use the endoscope or the exoscope alone, and the second mode is to use the endoscope and the exoscope at the same time. The position of the endoscope detection image is obtained through the image module, and the images of the endoscope and the exoscope are displayed on the same screen at the same time.

[0018] Furthermore, the controller module is connected to the light source module, the power management module, and the image processing module; when an endoscope or an exoscope is used alone, the medical staff controls the light source illumination through the controller module and selects to provide power to one of the devices, and the image of the endoscope or exoscope is presented on the display module; when an exoscope and an endoscope are used at the same time, the image processing module uses an algorithm to identify and match feature points of the endoscopic image and the exoscope image respectively, obtains the relative position of the endoscopic image in the surgical area, and after completing the feature matching, the image processing module presents the images of the endoscope and the exoscope in a fused and expanded form on the display module.

[0019] The beneficial effects of the present invention are:

[0020] The present invention enables the selective use of endoscopes and exoscopes through the device, and in some special scenarios, supports the simultaneous use of endoscopes and exoscopes. In addition, the present invention provides a position positioning function for endoscopic images, which facilitates medical staff to adjust the position and observation angle of the endoscope, indicating the specific location of the endoscopic image in the exoscopic surgical area, greatly expanding the surgical field of view and angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a system diagram of the present invention.

[0022] Figure 2 This is a flowchart for implementing the image processing module of the present invention.

[0023] Figure 3 Schematic diagram of the integral graph algorithm in the embodiment. DETAILED DESCRIPTION

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0025] like Figure 1 As shown, the present invention provides an efficient surgical monitoring system integrating endoscope and endoscope, including a light source module, a camera module, a power management module, a data storage module, an image processing module, an image display module, a bracket and a controller module.

[0026] In this embodiment, the camera module includes an exoscopic camera module and an endoscope camera module. The exoscopic camera module is used to obtain image information in the surgical field of view. The camera system in this embodiment includes two imaging light paths, which have three movable optical modules, namely a zoom group, a compensation group and a focusing group. The focusing group realizes continuous adjustable change of the working distance, and the zoom group and the focusing group cooperate to realize the change of the overall optical magnification. The compensation group is used to maintain the clarity of the image during the zoom process. The endoscope camera module is equipped with a main view and a side view dual camera. The main view camera can detect the front end of the endoscope, and the side view camera can detect the side. Full-view detection is possible without bending the endoscope, making the detection process more comprehensive and convenient.

[0027] In this embodiment, the light source module includes a lighting system and a light source control unit. The lighting system in this embodiment uses the same light source device to provide lighting for the endoscope and the exterior mirror. The exterior mirror and the endoscope light source usually use a xenon lamp or an LED light source. In this embodiment, an LED light source is used. Compared with a xenon lamp, an LED light source has the characteristics of long life, low energy consumption, good stability and low heat generation. The light source transmits light to the front end of the camera module of the endoscope and the exterior mirror through two light guides. The light guide usually adopts an optical fiber structure. The optical fiber core is used to guide the optical fiber emitted by the light source to the endoscope camera. There is a protective layer outside the optical fiber core. The interfaces at both ends of the light guide are customized according to the type of light source interface.

[0028] In this embodiment, the light source control unit can provide functions for adjusting the brightness and spectrum of the light sources of the endoscopes and the endoscopic mirrors, controlling the light source stability, and detecting uniform illumination. In this embodiment, medical personnel can choose to use either the endoscope or the endoscopic mirror alone or simultaneously, depending on different application scenarios. Medical personnel use the light source control unit to illuminate either or both endoscopes, and can adjust the brightness during surgery based on the image quality of the display screen.

[0029] In this embodiment, the power management module is used to provide and manage power supply to each module of the system.

[0030] In this embodiment, the data storage module is used to store video data and sensor data for easy recovery and analysis.

[0031] In this embodiment, the image processing module is used to provide position navigation during surgery. When the exoscope and endoscope work together, the exoscope camera module obtains the image of the surgical area and transmits it to the image processing module, and the image processing module performs feature detection through an algorithm; when the endoscope moves, the image obtained by the endoscope camera module is transmitted to the image processing module for feature recognition; the image processing module partitions the global surgical area image obtained by the exoscope, and performs feature matching analysis on the feature information of the endoscope image with each area of ​​the exoscope image; after completing the feature matching, the image processing module presents the images of the endoscope and exoscope in a fused and expanded form on the display screen.

[0032] In this embodiment, the image display device utilizes a medical display screen and an optical camera. Medical display screens can be of various types, including 2K, 4K, 2D, and 3D. Preferably, a polarized 3D 4K display screen is used in this embodiment. The optical camera is used to process the exoscopic images, performing stitching and correction. The two sets of images are then processed by the image processing host for three-dimensional synthesis and output in a specific format to the corresponding 3D display device. The 3D display screen can selectively output either the exoscopic or endoscopic images, depending on the actual situation, or display both images simultaneously on the screen while the endoscope and endoscope are operating. To achieve optimal observation, this embodiment also includes 3D glasses.

[0033] In this embodiment, the types of supports include pneumatic robotic arms, electric robotic arms, hydraulic robotic arms, cable-driven robotic arms, etc. Preferably, this embodiment can use an electric robotic arm, the robotic arm is fixed on the box, and is connected to the robotic arm through the arm mechanical joint, the robotic arm and the camera arm are interconnected through the arm mechanical joint, and the camera arm and the camera system are interconnected through the arm mechanical joint. Mechanical joints include planar joints, spherical joints, annular joints, spiral joints, etc. Preferably, the mechanical joints in this embodiment can use spherical joints, which can rotate around three mutually perpendicular axes to provide 3 to 6 degrees of freedom. Therefore, the robotic arm in this embodiment is flexible and has a large range of movement. The robotic arm can move the camera module of the exoscope or endoscope to the required position by dragging it.

[0034] In this embodiment, the controller module is used to coordinate the operation of the endoscope and exoscope systems. In this embodiment, medical staff can optionally select two operating modes using the controller module. In the first mode, medical staff choose to use the exoscope or endoscope alone according to the scenario. The controller module is connected to the light source module, the power management module, and the image processing module. When the endoscope or exoscope is used alone, the medical staff controls the light source illumination through the controller module and chooses to provide power to one of the devices. When used alone, the image of the endoscope or exoscope is presented on the display screen. In the second mode, the exoscope and endoscope are used at the same time. The image processing module uses an algorithm to identify and match the feature points of the endoscopic image and the exoscope image respectively to obtain the relative position of the endoscopic image in the surgical area. After completing the feature matching, the image processing module can optionally fuse the images of the endoscope and exoscope and present them on the display screen in an expanded form.

[0035] An image always has unique pixels, which we can consider to be the characteristics of the image, called feature points. We select certain feature points from the image and perform local analysis on them, rather than observing the entire image. As long as there are enough detectable points of interest in the image, and these points are unique and have stable characteristics, they can be accurately located. Endoscopes and exoscopes have different image scales.

[0036] In this embodiment, the image processing module uses a feature matching algorithm and an image fusion algorithm (such as Figure 2 shown).

[0037] In this embodiment, the feature matching algorithms mainly include methods based on Sift, Harris, ORB, LBP, Surf, etc. Optionally, since the Surf algorithm greatly improves the speed and stability of feature point detection and has strong applications in object matching in some real-time video streams, the Surf algorithm is used to perform feature recognition and matching on endoscope and exoscope images.

[0038] The implementation process of the Surf algorithm is as follows:

[0039] S1. Generate integral graph

[0040] Assume that l(x,y) is a pixel point at any position in the image, that is, its integral image is the sum of all pixel values ​​in the entire rectangular area from the pixel point to the origin. The formula is as follows:

[0041]

[0042] right Figure 3 The grayscale value of each pixel in the area of ​​the square ABCD is summed up as follows:

[0043] ∑=l ∑(A) -1 ∑ (B) +1 ∑ (C) -1 ∑ (D)

[0044] S2, locating feature points

[0045] In the Surf algorithm, the Hessian matrix of each pixel point in the image is calculated to extract the feature points. The purpose of constructing the Hessian matrix is to generate stable edge points (abrupt points) in the image, which lays a foundation for subsequent feature extraction. The specific process is as follows:

[0046] Given a point in an image, the Hessian matrix H = (l, σ) at the point l and with a scale σ is:

[0047]

[0048] where the parameter L xx (l, σ), L xy (l, σ), and L yy (l, σ) are the second-order differential Convolve with the image I at the point l = (x, y).

[0049] To detect feature points on the same image in different scale spaces, an important property of feature points is realized, i.e., its scale invariance. By replacing the Gaussian filter with a box filter, the scale space of the image can be constructed by convolving the integral image with box filters of different sizes and different directions. According to the determinant of the Hessian matrix, extreme points are extracted. Due to the similar replacement of the box filter, the determinant needs to be weighted, i.e.:

[0050] det(H) = D xx D yy -(wD xy ) 2

[0051] where w is the weight value, generally taking the value of 0.9, which aims to balance the error caused by the use of the box filter approximation.

[0052] S3, determining the main direction of the feature point

[0053] Taking the accurate feature point as the origin of the circular region to be constructed, a circular region with a radius of 6 scales is established around it. Then, the response value of the Haar wavelet corresponding to all pixel points in its region is calculated. A 60° sector region is rotated in the circular region by the same size of degrees, and the response values in the region are accumulated to obtain the maximum value, which is set as the main direction.

[0054] S4. Construct feature descriptor

[0055] The information of the detected feature points needs to be described by feature vectors. First, a square neighborhood with a side length of 20 scales is constructed, and the feature point is located at its center. Then, its direction is rotated to the main direction of the feature point and divided into 16 sub-areas. Then, Harr is used to calculate the response of all pixels in each small area in the x and y directions, and the total response of the small area is counted to obtain a 4-dimensional feature vector as follows:

[0056] v=[∑dx,∑dy,∑∣dx∣,∑∣dy∣]

[0057] Finally, the 4-dimensional feature vectors of all sub-regions are put together to form a 64-dimensional feature descriptor of the Surf feature point.

[0058] S5. Feature matching

[0059] Surf determines the degree of match by calculating the Euclidean distance between two feature descriptors. The shorter the Euclidean distance, the better the match between the two feature points. Surf also uses the Hessian matrix trace. If the matrix traces of two feature points have the same sign, it means that the contrast changes in the same direction. If they are different, the contrast changes in the two feature points are in opposite directions. Even if the Euclidean distance is 0, the feature point is directly excluded.

[0060] To display the endoscopic image at the corresponding location on the exterior mirror image, the imaging module partitions the exterior mirror image into regions of equal size. In this embodiment, the image is divided into nine regions of equal area. The Surf algorithm is then used to perform feature matching on the endoscopic image. When a high degree of match is achieved, the image processing module uses an image fusion algorithm to merge the endoscopic image with the corresponding exterior mirror partition image in an expanded form and present it on the display.

[0061] In this embodiment, the image fusion algorithms mainly include direct stitching, image weighted addition, pyramid fusion, and Poisson fusion algorithms. Optionally, because the Poisson image fusion algorithm utilizes gradient information, it can achieve a smoother transition for the gaps at the image splicing locations, greatly enhancing the fusion effect and the sense of realism. It is applicable to scenarios such as panoramic image stitching and Mona Lisa face-changing special effects. The Poisson image fusion algorithm is used to stitch and fuse the endoscopic image and the corresponding exoscopic sub-area image.

[0062] The algorithm assumes that within the fusion region Ω, the target image f and the source image f * The divergence div is equal, and the boundary value of f is equal to f * The boundary values ​​of are equal, that is, the variational equation is satisfied:

[0063]

[0064] Where v is the guidance vector field, and for image fusion tasks, v is the image gradient.

[0065] The Δ gradient operator is:

[0066] v=(u,v)=Δg

[0067] The Laplace operator is:

[0068]

[0069] The div divergence operator is:

[0070]

[0071] So the div operator is equivalent to the Laplace operator:

[0072]

[0073] The core of the Poisson image fusion process is to solve the central pixel value f of the four neighborhoods of the Laplace window p The process:

[0074]

[0075] Transform into linear vector form Af=b:

[0076]

[0077] Where A matrix is ​​the coefficient matrix of Laplace operator, f p is the pixel value at point p in the source image, and b is the Laplace filter result at point p in the source image, that is, the divergence of point p. By sliding the Laplace window within the fusion area Ω, the value of each pixel in the fusion area can be obtained.

[0078] The Poisson image fusion algorithm process is:

[0079] S1. Get the Mask area of ​​the source image Src.

[0080] S2. In the Mask, at position p, use the Laplace operator to construct the linear equation system Af=b.

[0081] S3. Optionally, use the Jacobi iteration method, Gauss-Seidel method or other linear equation solving methods to solve Af=b and extract f. p , set it to the grayscale value of the corresponding position p in the target image.

[0082] S4. Slide the Laplace operator window within the Mask and repeat S2-S4 until all points within the Mask are traversed and the fusion is completed.

[0083] The technical means disclosed in the solutions of the present invention are not limited to those disclosed in the above-mentioned embodiments, but also include technical solutions composed of any combination of the above-mentioned technical features. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An efficient surgical monitoring system integrating endoscope and endoscope, characterized in that: It includes a camera module, a light source module, a power management module, a data storage module, an image processing module, an image display module, a bracket and a controller module; the camera module includes an exoscopic camera module and an endoscopic camera module, the light source module is used to provide light to the power management module, the image processing module is used to process the image data obtained by the camera module and present it on the image display module, and the controller module is used to coordinate the work of the endoscopic camera module and the exoscopic camera module; the image processing module is used to provide position navigation during surgery, when the exoscopic camera and the endoscope work together, the exoscopic camera module obtains the image of the surgical area and transmits it to the image processing module, and the image processing module performs feature detection through a feature algorithm; when the endoscope moves, the image obtained by the endoscopic camera module is transmitted to the image processing module for feature recognition; the image processing module partitions the global surgical area image obtained by the exoscopic camera, and performs feature matching analysis on the feature information of the endoscopic image with each area of ​​the exoscopic image; After completing the feature matching, the endoscopic image and the corresponding exoscopic partition image are obtained, and the images are spliced ​​and fused in an expanded form through the image fusion algorithm, and finally transmitted to the image display module for display.

2. The efficient surgical monitoring system integrating endoscopy and endoscopy according to claim 1, characterized in that: The exoscopic camera module is connected to the bracket and is used to obtain image information in the surgical field of view. It includes at least two imaging optical paths and image sensors, and the image sensor is placed behind the imaging optical path; the endoscope camera module is used to detect the front end and side conditions of the endoscope; the imaging optical path includes three movable optical modules, namely a magnification group, a compensation group and a focusing group. The focusing group realizes continuous adjustable change of the working distance, and the magnification group and the focusing group cooperate to realize the change of the overall optical magnification. The compensation group is used to maintain the clarity of the image during the zooming process; the endoscope camera module is equipped with a main view and a side view dual camera. The main view camera is used to detect the front end of the endoscope, and the side view camera is used to detect the side condition of the endoscope.

3. The efficient surgical monitoring system integrating endoscopy and endoscopy according to claim 1, characterized in that: The light source module includes a lighting system and a light source control unit; the lighting system includes a light source and a light guide, the light source is connected to the endoscope and exoscope respectively through the light guide, and transmits light to the front end of the endoscope and exoscope camera module, and the light source control unit is used to provide light source brightness adjustment, spectrum adjustment, light source stability control and uniform lighting detection functions for the endoscope and exoscope.

4. The efficient surgical monitoring system combining endoscopy and endoscopy according to claim 1, characterized in that: The power management module is used to provide and manage the power supply of each system module, and the data storage module is used to store video data and sensor data.

5. The efficient surgical monitoring system integrating endoscopy and endoscopy according to claim 1 is characterized in that: The image processing module adopts a feature matching algorithm to perform feature recognition and matching on the endoscope and exoscopic images. When the matching degree is high, the endoscope image and the corresponding exoscopic partition image are obtained.

6. The efficient surgical monitoring system combining endoscopy and endoscopy according to claim 1, characterized in that: The image processing module adopts an image fusion algorithm to splice and fuse the endoscopic image and the corresponding exoscopic partition image in an expanded form, and presents them on the image display module.

7. The efficient surgical monitoring system combining endoscopy and endoscopy according to claim 1, characterized in that: The image display module includes a medical display screen, which is used to selectively output an exoscopic or endoscopic image according to actual working conditions, or to display the image on the screen simultaneously when the endoscope and the exoscopic image are working.

8. The efficient surgical monitoring system integrating endoscopy and endoscopy according to claim 1 is characterized in that: The bracket is used to adjust or fix the angle of the exterior mirror, and includes a robotic arm and a robotic arm control module. The robotic arm includes a robotic arm, a robotic arm, and a camera arm connected in sequence, and each robotic arm is connected through a mechanical joint. The robotic arm control module provides motion control and posture adjustment of the robotic arm.

9. The efficient surgical monitoring system integrating endoscopy and endoscopy according to claim 1, characterized in that: The controller module is used to coordinate the work of the endoscope camera module and the exoscope camera module. The controller module allows doctors to use it in two modes. The first mode is to use an endoscope or an exoscope alone, and the second mode is to use an endoscope and an exoscope at the same time. The position of the endoscope detection image is obtained through the image processing module, and the images of the endoscope and the exoscope are spliced ​​and fused, and then displayed on the same screen at the same time; the controller module is connected to the light source module, the power management module, and the image processing module; when the endoscope or the exoscope is used alone, the medical staff controls the light source illumination through the controller module, selects to provide power to one of the devices, and the image of the endoscope or the exoscope is presented on the display module; when the exoscope and the endoscope are used at the same time, the image processing module uses the feature matching algorithm to identify and match the feature points of the endoscopic image and the exoscope image respectively, and obtains the relative position of the endoscopic image in the surgical area. After completing the feature matching, the image processing module uses the image fusion algorithm to present the images of the endoscope and the exoscope in a spliced, fused and expanded form on the display module.

Images