Efficient operation monitoring system with fusion of endoscope and endoscope

Abstract

The invention discloses an efficient operation monitoring system with fusion of an endoscope and an endoscope. The efficient operation monitoring system comprises a camera module, a light source module, a power management module, a data storage module, an image processing module, an image display module, a support and a controller module. The camera shooting module comprises an exterior mirror camera shooting module and an endoscope camera shooting module, the light source module is used for providing a light source for the power management module, and the image processing module is used for processing image data obtained by the camera shooting module, splicing and fusing endoscope and exterior mirror partition images and displaying the images on the image display module. The controller module is used for coordinating work of the endoscope camera module and the exterior mirror camera module. The endoscope and the exterior mirror can be selectively used through the device, and meanwhile in some special scenes, simultaneous use of the endoscope and the exterior mirror is supported. Besides, the invention further provides a position positioning function of the endoscope image, so that medical personnel can conveniently adjust the position and the observation angle of the endoscope and indicate the specific position of the endoscope image in an operation area of the exterior mirror, and the operation view field and the operation angle are greatly expanded.

Description

Technical Field

[0001] The invention belongs to the technical field of medical devices and relates to a high-efficiency surgery monitoring system integrating endoscope and endoscope. Background Art

[0002] In recent years, endoscopes have been widely used in neurosurgery due to their wide viewing angle, free visual axis, and minimal surgical trauma. In the scenario where an endoscope is used alone, when the endoscope passes through the neurovascular structure, the neurovascular structure may not be visible through the endoscope. Therefore, in endoscopic surgery, there is a risk of damage to the neurovascular structure outside the range of the endoscope.

[0003] The exoscopic system was proposed based on the shortcomings of surgical microscopes and neuroendoscopic technology in clinical applications. The surgical exoscopic system has gradually shown its advantages in replacing traditional surgical microscopes due to its wider field of view, longer focal length, more comfortable operation, smaller size and weight. In order to cope with complex surgical environments, there is an urgent need for a device that combines endoscopes and endoscopes. Summary of the invention

[0004] When using an exoscopic device, the exoscopic system transmits the image of the surgical area to a display in real time through a camera for the doctor to observe and operate. However, the surgical area image obtained has blind spots due to the occlusion of organs or tissues. When the endoscope is moving, the position information corresponding to the local surgical area image obtained in the global image cannot be displayed. In order to solve the above problems, the present invention discloses an efficient surgical monitoring system that integrates endoscopes and endoscopes, which can obtain more comprehensive image information during the operation and meet the requirements of modern surgical operations.

[0005] The specific plan is as follows:

[0006] An efficient surgical monitoring system integrating endoscope and endoscope is characterized in that it comprises a camera module, a light source module, a power management module, a data storage module, an image processing module, an image display module, a bracket and a controller module; the camera module comprises an exoscopic camera module and an endoscopic camera module, the light source module is used to provide light source to the power management module, the image processing module is used to process image data obtained by the camera module and present it on the image display module, and the controller module is used to coordinate the work of the endoscopic camera module and the exoscopic camera module.

[0007] Furthermore, the camera module includes an exoscopic camera module and an endoscopic camera module; the exoscopic camera module is connected to the bracket and is used to obtain image information in the surgical field of view, including at least two imaging optical paths and an image sensor, and the image sensor is placed behind the imaging optical path; the endoscopic camera module is used to detect the front end and side conditions of the endoscope.

[0008] Furthermore, the imaging optical path includes three movable optical modules, namely a magnification group, a compensation group and a focusing group. The focusing group realizes a continuously adjustable change of the working distance, the magnification group and the focusing group cooperate to realize the change of the overall optical magnification, and the compensation group is used to maintain the clarity of the image during the zooming process; the endoscope camera module is equipped with a main view and a side view dual camera. The main view camera is used to detect the front end of the endoscope, and the side view camera is used to detect the side of the endoscope.

[0009] Furthermore, the light source module includes a lighting system and a light source control unit; the lighting system includes a light source and a light guide, the light source is respectively connected to the endoscope and the exoscope through the light guide, and the light is transmitted to the front end of the endoscope and the exoscope camera module, and the light source control unit is used to provide light source brightness adjustment, spectrum adjustment, light source stability control and uniform lighting detection functions for the endoscope and the exoscope.

[0010] Furthermore, the power management module is used to provide and manage power supply of each module of the system.

[0011] Furthermore, the data storage module is used to store video data and sensor data.

[0012] Furthermore, the image processing module is used to provide position navigation during surgery. When the exoscope and the endoscope work together, the exoscope camera module obtains the image of the surgical area and transmits it to the image processing module, and the image processing module performs feature detection through an algorithm; when the endoscope moves, the image obtained by the endoscope camera module is transmitted to the image processing module for feature recognition; the image processing module partitions the global surgical area image obtained by the exoscope, and performs feature matching analysis on the feature information of the endoscope image with each area of ​​the exoscope image; after completing the feature matching, the image processing module presents the images of the endoscope and the exoscope on the image display module in a spliced, fused and expanded form.

[0013] Furthermore, the feature matching algorithm performs feature recognition and matching on the endoscope and the exterior mirror images, and when the matching degree is high, the image processing module obtains the endoscope image and the corresponding exterior mirror partition image.

[0014] Furthermore, the image fusion algorithm stitches and fuses the endoscopic image and the corresponding exoscopic partition image in an extended form and presents them on the image display module.

[0015] Furthermore, the image display module includes a medical display screen, which is used to select and output an exoscopic or endoscopic image according to actual working conditions, or to display the image on the screen simultaneously when the endoscope and the exoscopic image are working.

[0016] Furthermore, the bracket is used to adjust or fix the angle of the exterior mirror, and includes a robotic arm and a robotic arm control module. The robotic arm includes a robotic arm, a robotic arm, and a camera arm connected in sequence, and each robotic arm is connected through a mechanical joint. The robotic arm control module provides motion control and posture adjustment of the robotic arm.

[0017] Furthermore, the controller module is used to coordinate the work of the endoscope camera module and the exoscope camera module. The controller module allows the doctor to use it in two modes. The first mode is to use the endoscope or the exoscope alone, and the second mode is to use the endoscope and the exoscope at the same time. The position of the endoscope detection image is obtained through the image module, and the images of the endoscope and the exoscope are displayed on the same screen at the same time.

[0018] Furthermore, the controller module is connected to the light source module, the power management module, and the image processing module; when an endoscope or an exoscope is used alone, the medical staff controls the light source illumination through the controller module and selects to provide power to one of the devices, and the image of the endoscope or the exoscope is presented on the display module; when an exoscope and an endoscope are used at the same time, the image processing module uses an algorithm to identify and match feature points of the endoscopic image and the exoscopic image respectively, obtains the relative position of the endoscopic image in the surgical area, and after completing the feature matching, the image processing module presents the images of the endoscope and the exoscope in a fused and expanded form on the display module.

[0019] The beneficial effects of the present invention are:

[0020] The present invention can selectively use endoscopes and exoscopes through the device, and in some special scenarios, supports the simultaneous use of endoscopes and exoscopes. In addition, the present invention also provides a position positioning function for endoscopic images, which facilitates medical staff to adjust the position and observation angle of the endoscope, and indicates the specific position of the endoscopic image in the exoscopic surgical area, greatly expanding the surgical field of view and angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a system diagram of the present invention.

[0022] Figure 2 This is a flowchart for implementing the image processing module of the present invention.

[0023] Figure 3 Schematic diagram of the integral graph algorithm in the embodiment. DETAILED DESCRIPTION

[0024] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0025] like Figure 1 As shown, the present invention provides an efficient surgical monitoring system integrating endoscope and endoscope, including a light source module, a camera module, a power management module, a data storage module, an image processing module, an image display module, a bracket and a controller module.

[0026] In this embodiment, the camera module includes an exoscopic camera module and an endoscope camera module. The exoscopic camera module is used to obtain image information in the surgical field of view. The camera system in this embodiment includes two imaging optical paths, which have three movable optical modules, namely a zoom group, a compensation group and a focusing group. The focusing group realizes the continuous adjustable change of the working distance, and the zoom group and the focusing group cooperate to realize the change of the overall optical magnification. The compensation group is used to maintain the clarity of the image during the zooming process. The endoscope camera module is equipped with a main view and a side view dual camera. The main view camera can detect the front end of the endoscope, and the side view camera can detect the side. Full-view detection is possible without bending the endoscope, making the detection process more comprehensive and convenient.

[0027] In this embodiment, the light source module includes a lighting system and a light source control unit. The lighting system in this embodiment uses the same light source device to provide lighting for the inner and outer mirrors, and the light sources of the outer mirror and the endoscope usually use xenon lamps or LED light sources. In this embodiment, LED light sources are used. Compared with xenon lamps, LED light sources have the characteristics of long life, low energy consumption, good stability and low heat generation. The light source transmits light to the front end of the camera module of the endoscope and the outer mirror through two light guides. The light guide usually adopts an optical fiber structure. The optical fiber core is used to guide the optical fiber emitted by the light source to the endoscope camera, and there is a protective layer outside the optical fiber core. The interfaces at both ends of the light guide are customized according to the type of light source interface.

[0028] In this embodiment, the light source control unit can provide functions of adjusting the brightness of the light source of the internal and external mirrors, adjusting the spectrum, controlling the light source stability, and detecting uniform illumination. In this embodiment, medical staff can choose to use one of the internal and external mirrors alone or simultaneously according to different application scenarios. Medical staff use the light source control unit to provide illumination for one of the internal and external mirrors or both, and can choose to adjust the brightness according to the image effect on the display screen during the operation.

[0029] In this embodiment, the power management module is used to provide and manage the power supply of each module of the system.

[0030] In this embodiment, the data storage module is used to store video data and sensor data for easy recovery and analysis.

[0031] In this embodiment, the image processing module is used to provide position navigation during surgery. When the exoscope and the endoscope work together, the exoscope camera module obtains the image of the surgical area and transmits it to the image processing module, and the image processing module performs feature detection through an algorithm; when the endoscope moves, the image obtained by the endoscope camera module is transmitted to the image processing module for feature recognition; the image processing module partitions the global surgical area image obtained by the exoscope, and performs feature matching analysis on the feature information of the endoscope image with each area of ​​the exoscope image; after completing the feature matching, the image processing module presents the images of the endoscope and the exoscope on the display screen in a fused and expanded form.

[0032] In this embodiment, the image display device uses a medical display screen and an optical camera. The medical display screen can be of various types, including a 2K display screen, a 4K display screen, a 2D display screen, and a 3D display screen. Preferably, a polarized 3D 4k display screen is used in this embodiment. The optical camera is used to process the image of the exoscopic mirror, perform splicing and correction, and the two sets of images are processed by the image processing host for three-dimensional synthesis and then output to the corresponding 3D display device in a specific output form. The 3D display screen can choose to output the exoscopic or endoscopic image according to actual conditions, or display it on the screen simultaneously when the endoscope and endoscope are working. In order to achieve the observation effect, 3D glasses are also provided in this embodiment.

[0033] In this embodiment, the support types include pneumatic mechanical arms, electric mechanical arms, hydraulic mechanical arms, cable-driven mechanical arms, etc. Preferably, this embodiment can use an electric mechanical arm, the mechanical arm is fixed on the box, connected to the mechanical arm through the mechanical joint of the arm, the mechanical arm and the camera arm are interconnected through the mechanical joint of the arm, and the camera arm and the camera system are interconnected through the mechanical joint of the arm. Mechanical joints include plane joints, spherical joints, annular joints, spiral joints, etc. Preferably, the mechanical joints in this embodiment can use spherical joints, which can rotate around three mutually perpendicular axes and provide 3 to 6 degrees of freedom. Therefore, the mechanical arm in this embodiment is flexible and has a large range of movement. The mechanical arm can move the camera module of the exoscopic or endoscope to the required position by dragging it.

[0034] In this embodiment, the controller module is used to coordinate the work of the endoscope and the exoscopic system. In this embodiment, the medical staff can optionally select two operation modes using the controller module. In the first mode, the medical staff chooses to use the exoscopic or endoscope alone according to the scene. The controller module is connected to the light source module, the power management module, and the image processing module. When the endoscope or the exoscopic is used alone, the medical staff controls the light source illumination through the controller module and chooses to provide power for one of the devices. When used alone, the image of the endoscope or the exoscopic is presented on the display screen. In the second mode, the exoscopic and endoscope are used at the same time. The image processing module uses an algorithm to identify and match the feature points of the endoscope image and the exoscopic image respectively, and obtains the relative position of the endoscope image in the surgical area. After completing the feature matching, the image processing module can choose to fuse and expand the images of the endoscope and the exoscopic and present them on the display screen.

[0035] There are always unique pixels in an image, which we can consider as the features of the image, called feature points. Select certain feature points from the image and perform local analysis on the image instead of observing the entire image. As long as there are enough detectable points of interest in the image, and these points of interest are different and have stable features, they can be accurately located. The image scales obtained by endoscopes and exoscopes are different.

[0036] In this embodiment, the image processing module uses a feature matching algorithm and an image fusion algorithm (such as Figure 2 shown).

[0037] In this embodiment, the feature matching algorithms mainly include methods based on Sift, Harris, ORB, LBP, Surf, etc. Optionally, since the Surf algorithm greatly improves the speed and stability of feature point detection and has strong applications in some real-time video stream object matching, the Surf algorithm is used to perform feature recognition and matching on endoscope and exoscopic images.

[0038] The implementation process of the Surf algorithm is as follows:

[0039] S1. Generate integral graph

[0040] Assume that l(x,y) is a pixel point at any position in the image, that is, its integral image is the sum of all pixel values ​​in the entire rectangular area from the pixel point to the origin. The formula is as follows:

[0041]

[0042] right Figure 3 The gray value of each pixel in the area of ​​the square ABCD in is summed up, and the formula is as follows:

[0043] ∑=l ∑(A)-l ∑ (B)+l ∑ (C)-l ∑ (D)

[0044] S2. Positioning feature points

[0045] In the Surf algorithm, when extracting feature points, the integral image is mainly used to calculate the Hessian matrix of each pixel in the image. The purpose of constructing the Hessian matrix is ​​to generate stable edge points (mutation points) of the image, which lays a good foundation for subsequent feature extraction. The specific process is as follows:

[0046] Given a point in an image, the Hessian matrix H = (l, σ) at that point l and with scale σ is:

[0047]

[0048] Among them, the parameter L xx (l,σ),L xy (l,σ),L yy (l,σ) are Gaussian second-order differentials Convolve with image I at point l = (x, y).

[0049] It is required to detect feature points on the same image in different scale spaces to realize an important property of feature points, namely, its scale invariance. By using the box filter to similarly replace the Gaussian filter, the scale space of the image can be constructed by convolution operations of box filters of different sizes and integral images of different directions, and the extreme points are extracted according to the sum determinant of the Hessian matrix. Due to the similar replacement of the box filter, the determinant at this time needs to be added with weights, that is:

[0050] det(H)=D xx D yy -(wD xy ) 2

[0051] Among them, w is the weight, which is generally taken as 0.9, in order to balance the error caused by the use of box filter approximation.

[0052] S3. Determine the main direction of the feature point

[0053] Use the precise feature point as the origin of the circular area to be constructed, establish a circular area with a radius of 6 scales around it, and then calculate the response values ​​of the Haar wavelet corresponding to all pixels in its area. Then use a 60° fan-shaped area to rotate the circular area by the same degree, accumulate all the response values ​​in the area, and set the largest value as the main direction.

[0054] S4. Construct feature descriptor

[0055] The information of the detected feature points needs to be described by feature vectors. First, construct a square neighborhood with a side length of 20 scales, and the feature point is at its center, then rotate its direction to the main direction of the feature point, divide it into 16 sub-areas, and then use Harr to calculate the response of all pixels in each small area in the x and y directions, and count the total response of the small area to obtain a 4-dimensional feature vector as follows:

[0056] v=[∑dx,∑dy,∑∣dx∣,∑∣dy∣]

[0057] Finally, the 4-dimensional feature vectors of all sub-regions are put together to form a 64-dimensional feature descriptor of the Surf feature point.

[0058] S5. Feature matching

[0059] Surf determines the matching degree by calculating the Euclidean distance between two feature descriptors. The shorter the Euclidean distance, the better the matching degree of the two feature points. Surf also adds the judgment of the Hessian matrix trace. If the matrix traces of two feature points have the same positive and negative signs, it means that the two features have contrast changes in the same direction. If they are different, it means that the contrast changes of the two feature points are in opposite directions. Even if the Euclidean distance is 0, it is directly excluded.

[0060] In order to display the endoscope image at the corresponding position of the external mirror image, the image module partitions the external mirror image into regions of equal area according to the size of the external mirror image. In this embodiment, the image is divided into 9 regions of equal area, and the Surf algorithm and the endoscope image are sequentially matched for features. When the matching degree is high, the image processing module uses the image fusion algorithm to splice and fuse the endoscope image and the corresponding external mirror partition image in an extended form and present it on the display.

[0061] In this embodiment, the image fusion algorithm mainly includes direct stitching method, image weighted addition method, pyramid fusion method, Poisson fusion algorithm, etc. Optionally, since the Poisson image fusion algorithm uses gradient information, the gaps at the image splicing can be smoothly transitioned, which greatly enhances the fusion effect and improves the fusion realism. It is applied to scenes such as panoramic image stitching and Mona Lisa face-changing special effects, and the Poisson image fusion algorithm is adopted to realize the stitching and fusion of the endoscope image and the corresponding exterior mirror partition image.

[0062] The algorithm assumes that within the fusion region Ω, the target image f and the source image f * The divergence div is equal to that of f, and the boundary value of f is equal to f * The boundary values ​​of are equal, that is, the variational equation is satisfied:

[0063]

[0064] Where v is the guidance vector field, and for image fusion tasks, v is the image gradient.

[0065] The Δ gradient operator is:

[0066] v=(u,v)=Δg

[0067] The Laplace operator is:

[0068]

[0069] The div divergence operator is:

[0070]

[0071] So the div operator is equivalent to the Laplace operator:

[0072]

[0073] The core of the Poisson image fusion process is to solve the central pixel value f of the 4-neighborhood of the Laplace window. p The process:

[0074]

[0075] Transform into linear vector form Af = b:

[0076]

[0077] The A matrix is ​​the coefficient matrix of the Laplace operator, f p is the pixel value at point p of the source image, and b is the Laplace filtering result at point p of the source image, that is, the divergence of point p. By sliding the Laplace window in the fusion area Ω, the value of each pixel in the fusion area can be obtained.

[0078] The process of Poisson image fusion algorithm is as follows:

[0079] S1. Obtain the Mask area of ​​the source image Src.

[0080] S2. In the Mask, at position p, use the Laplace operator to construct a linear equation system Af=b.

[0081] S3. Optionally, use the Jacobi iteration method, Gauss-Seidel method or other linear equation solving methods to solve Af=b and extract f. p , set it to the grayscale value of the corresponding position p in the target image.

[0082] S4. Slide the Laplace operator window within the Mask and repeat S2-S4 until all points within the Mask are traversed and the fusion is completed.

[0083] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. An efficient surgical monitoring system integrating endoscope and endoscope, characterized in that: It includes a camera module, a light source module, a power management module, a data storage module, an image processing module, an image display module, a bracket and a controller module; the camera module includes an exterior camera module and an endoscope camera module, the light source module is used to provide light to the power management module, the image processing module is used to process the image data obtained by the camera module and present it on the image display module, and the controller module is used to coordinate the work of the endoscope camera module and the exterior camera module.

2. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The exoscopic camera module is connected to the bracket and is used to obtain image information in the surgical field of view, including at least two imaging optical paths and an image sensor, and the image sensor is placed behind the imaging optical path; the endoscope camera module is used to detect the front end and side conditions of the endoscope; the imaging optical path includes three movable optical modules, namely a magnification group, a compensation group and a focusing group, the focusing group realizes continuous and adjustable change of the working distance, the magnification group and the focusing group cooperate to realize the change of the overall optical magnification, and the compensation group is used to maintain the clarity of the image during the zooming process; the endoscope camera module is equipped with a main view and a side view dual camera, the main view camera is used to detect the front end of the endoscope, and the side view camera is used to detect the side of the endoscope.

3. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The light source module includes a lighting system and a light source control unit; the lighting system includes a light source and a light guide, the light source is connected to the endoscope and the exoscope respectively through the light guide, and the light is transmitted to the front end of the endoscope and the exoscope camera module, and the light source control unit is used to provide light source brightness adjustment, spectrum adjustment, light source stability control and uniform lighting detection functions for the endoscope and the exoscope.

4. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The power management module is used to provide and manage the power supply of each module of the system, and the data storage module is used to store video data and sensor data.

5. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The image processing module is used to provide position navigation during surgery. When the exoscopic scope and the endoscope work together, the exoscopic camera module obtains the image of the surgical area and transmits it to the image processing module, and the image processing module performs feature detection through a feature algorithm; when the endoscope moves, the image obtained by the endoscope camera module is transmitted to the image processing module for feature recognition; the image processing module divides the global surgical area image obtained by the exoscopic scope into partitions, and performs feature matching analysis on the feature information of the endoscope image and each area of ​​the exoscopic image; After completing the feature matching, the endoscopic image and the corresponding exoscopic partition image are obtained, and the images are stitched and fused in an extended form through the image fusion algorithm, and finally transmitted to the image display module for display.

6. The efficient surgical monitoring system combining endoscope and endoscope according to claim 5, characterized in that: The image processing module adopts a feature matching algorithm to perform feature recognition and matching on the endoscope and the exterior mirror images, and when the matching degree is high, an endoscope image and a corresponding exterior mirror partition image are obtained.

7. The efficient surgical monitoring system combining endoscope and endoscope according to claim 5, characterized in that: The image processing module adopts an image fusion algorithm to splice and fuse the endoscopic image and the corresponding exoscopic partition image in an extended form, and presents them on the image display module.

8. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The image display module includes a medical display screen, which is used to select and output an exoscopic or endoscopic image according to actual working conditions, or to display the image on the screen simultaneously when the endoscope and the exoscopic image are working.

9. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The bracket is used to adjust or fix the angle of the exterior mirror, and includes a robotic arm and a robotic arm control module. The robotic arm includes a robotic arm, a robotic arm, and a camera arm connected in sequence, and each robotic arm is connected through a mechanical joint. The robotic arm control module provides motion control and posture adjustment of the robotic arm.

10. The efficient surgical monitoring system combining endoscope and endoscope according to claim 1, characterized in that: The controller module is used to coordinate the work of the endoscope camera module and the exoscope camera module. The controller module allows doctors to use it in two modes. The first mode is to use an endoscope or an exoscope alone, and the second mode is to use an endoscope and an exoscope at the same time. The position of the endoscope detection image is obtained through the image processing module, and the images of the endoscope and the exoscope are spliced ​​and fused, and then displayed on the same screen at the same time; the controller module is connected to the light source module, the power management module, and the image processing module; when the endoscope or the exoscope is used alone, the medical staff controls the light source illumination through the controller module, selects to provide power to one of the devices, and the image of the endoscope or the exoscope is presented on the display module; when the exoscope and the endoscope are used at the same time, the image processing module recognizes and matches the feature points of the endoscope image and the exoscope image respectively through the feature matching algorithm, obtains the relative position of the endoscope image in the surgical area, and after completing the feature matching, the image processing module presents the images of the endoscope and the exoscope on the display module in the form of splicing, fusion and expansion through the image fusion algorithm.

Images