Intelligent surgical exterior mirror system

Abstract

The invention discloses an intelligent surgical exterior mirror system which comprises a working trolley, a pneumatic mechanical arm, an intelligent multi-light-path imaging system arranged at the tail end of the pneumatic mechanical arm, a 3D display device and an auxiliary control device. The working trolley comprises a box body, a display rotating bracket, a four-star foot base and universal trundles; the pneumatic mechanical arm comprises a pneumatic mechanical large arm, a pneumatic mechanical small arm, a pneumatic camera branch arm and a spherical joint; the intelligent multi-optical-path imaging system comprises a plurality of integrated zoom imaging optical paths, an intelligent focusing controller and an annular integrated chip. The 3D display device comprises a polarization type 3D main screen, a polarization type 3D auxiliary screen and a head-mounted display device. The head-mounted display equipment comprises VR glasses and AR glasses; the auxiliary control device comprises a pressure-sensitive pedal. According to the system, the requirements of modern surgical operations are met, and more intelligent services, more efficient signal transmission and more comfortable use experience are provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices and relates to an intelligent surgical exoscopic system. Background Art

[0002] In recent years, with the rapid development of medical imaging technology, the field of surgical operations has gradually moved towards minimally invasive and precise operations. As an important auxiliary tool, the exoscopic system has played an important role in minimally invasive surgery, open surgery, remote collaboration, etc. The exoscopic system transmits the image of the surgical area to the display in real time through a camera for doctors to observe and operate, allowing doctors to perform precise surgical operations with minimal exposure of the patient's internal tissues, thereby reducing surgical trauma, shortening postoperative recovery time, and improving surgical results.

[0003] There are many types of surgical exoscopic systems available on the market, but there are still some problems that cannot be ignored. For example, the intelligence and convenience of exoscopic scopes are low, and they often do not have the functions of multimodal imaging and multifunctional sensing, and the lens cannot automatically zoom; the obstruction of the shadowless lamp during surgery has a great interference with the imaging effect of the surgical field, affecting the quality of surgery; the information transmission method is complex and is easily disturbed during the transmission process, resulting in display delays or reduced image quality.

[0004] Therefore, there is an urgent need for a new surgical exoscopic system to address the deficiencies in existing technologies, provide more intelligent services, more efficient signal transmission, and a more comfortable user experience to meet the needs of modern surgical operations. Summary of the invention

[0005] Existing exoscopic scopes are less intelligent and convenient, often lacking the functions of multimodal imaging and multifunctional sensing, and the lens cannot automatically zoom; during surgery, the occlusion of the shadowless lamp has a significant interference with the imaging effect of the surgical field, affecting the quality of surgery; the information transmission method is complex and is easily disturbed during the transmission process, resulting in display delays or reduced image quality. In order to solve the above problems, the present invention discloses an intelligent surgical exoscopic system that meets the needs of modern surgical operations, provides more intelligent services, more efficient signal transmission, and a more comfortable user experience.

[0006] The specific plan is as follows:

[0007] An intelligent surgical exoscopic system, characterized in that it includes a work trolley, a pneumatic mechanical arm, an intelligent multi-optical path imaging system arranged at the end of the pneumatic mechanical arm, a 3D display device and an auxiliary control device; the work trolley includes a box body, a display rotating bracket, a four-star foot base and universal casters, wherein the display rotating bracket and the four-star foot base are respectively installed on the top and bottom of the box body, and the universal casters are cooperatively installed at the four top corners of the four-star foot base; the pneumatic mechanical arm includes a pneumatic mechanical arm, a pneumatic mechanical arm and a pneumatic camera arm, wherein the lower end of the pneumatic mechanical arm is installed on the box body, the pneumatic mechanical arm has two sections, one of which is connected to the upper end of the pneumatic mechanical arm through a big arm ball joint, the two sections and the other section and the pneumatic camera arm are connected through a small arm ball joint, and the pneumatic camera arm is connected to the pneumatic camera arm through a small arm ball joint. The arm is connected to the intelligent multi-path imaging system through a split-arm spherical joint; the intelligent multi-path imaging system includes integrated multiple zoom imaging optical paths, an intelligent focus controller and a ring-shaped integrated chip; the ring-shaped integrated chip is installed on the periphery of the multiple zoom imaging optical paths, and integrates thermal imaging sensors, gas sensors, Raman spectrum sensors and temperature and humidity sensors, and is connected to the periphery with a miniature ultrasonic probe and a miniature optical fiber probe; the 3D display device includes a polarized 3D main screen, a polarized 3D secondary screen and a head-mounted display device; the polarized 3D main screen and the polarized 3D secondary screen are connected to the box through a display rotating bracket and can be rotated on a fixed axis; the head-mounted display device includes VR glasses and AR glasses; the auxiliary control device includes a pressure-sensitive foot pedal for controlling the orientation and height of the intelligent multi-path imaging system.

[0008] As a further improvement of the present invention, a robotic arm control system, an image processing host and a light source control system are provided at the front end of the box, a drawer for storing medical equipment, etc. is provided at the rear end of the box, and an operation screen and heat dissipation holes are provided on the outside of the box. The operation screen is used to start and shut down the exterior mirror system, manually adjust the pneumatic robotic arm when necessary, set the image processing host output mode, and control the imaging mode; the heat dissipation holes are used to dissipate heat from the robotic arm control system, the image processing host and the light source control system to prevent the exterior mirror from overheating and causing danger; the robotic arm control system is used to provide motion control, posture adjustment, automatic positioning, balance control, safety functions, braking functions, sensor feedback and integration functions.

[0009] As a further improvement of the present invention, the image processing host is used to process the image information obtained by the camera system and output it in the output mode required by the user. The output mode includes a naked-eye 3D mode and an HMD mode. The naked-eye 3D mode is suitable for a polarized 3D main screen and a polarized 3D secondary screen, and the HMD mode is suitable for a head-mounted display device.

[0010] As a further improvement of the present invention, the image processing host provides intraoperative navigation function and path planning function, wherein the specific implementation method of the intraoperative navigation function is: using pre-acquired medical imaging data to construct a three-dimensional model of the patient's anatomical structure, aligning the real-time position and posture of the exoscope with the three-dimensional model, realizing real-time positioning of the exoscope in the patient's body, displaying the current position and field of view of the exoscope on the three-dimensional model, and providing intraoperative navigation for the doctor; the specific implementation method of the path planning function is: according to the surgical objectives and anatomical structures, automatically generating the optimal path for the exoscope to reach the target site, combining with the navigation system, guiding the exoscope to move along the planned path, avoiding key blood vessels and nerve tissues.

[0011] As a further improvement of the present invention, the image processing host also solves the problem of decreased brightness of the surgical field due to obstruction of the shadowless lamp during surgery, and comprehensively utilizes a brightness compensation model based on deep learning, multi-scale analysis and edge preservation technology. The specific implementation method is: training a convolutional neural network CNN for surgical field brightness compensation, using occluded and non-occluded surgical field image data for learning so that it can automatically estimate and compensate for local brightness; applying the CNN model to surgical field images of multiple scales, and using multi-scale fusion technology to obtain clearer and more natural enhancement results; at the same time, an edge-aware loss function is introduced into the CNN model, and an edge-preserving filter is applied in the post-processing stage to ensure that the enhanced image can retain key structural details and avoid image distortion or artifacts.

[0012] As a further improvement of the present invention, the image processing host also solves the problem of decreased brightness of the surgical field caused by the shadowless lamp being blocked during surgery. The specific implementation method is as follows: training a CNN model for surgical field brightness compensation, which is improved based on the existing image enhancement network structure, such as U-Net, ResNet, etc., and adjusted according to actual needs. For example, the network depth can be increased, the convolution kernel size can be adjusted, etc., to improve the learning ability and generalization ability of the model; the CNN model is applied to surgical field images of multiple scales, and multi-scale fusion technology is used to obtain clearer and more natural enhancement results; at the same time, an edge-aware loss function is introduced, and an edge-preserving filter is applied in the post-processing stage to ensure that the enhanced image can retain key structural details and avoid image distortion or artifacts;

[0013] The loss function is defined as:

[0014] L = L_mse + λ × L_edge,

[0015] Among them, L_mse is the mean square error loss function, which is used to measure the pixel-level difference between the enhanced image and the real image; L_edge is the edge loss function, which is used to measure the difference in edge information between the enhanced image and the real image; the Sobel operator or Canny operator is used to extract the image edge and calculate the difference in edge pixels; λ is the weight coefficient, which is used to balance the importance of the two loss functions;

[0016] Among them, L_mse is defined as:

[0017] L_mse=1 / N×Σ(I_e-I_gt) 2 ,

[0018] Among them, I_e is the enhanced image, I_gt is the real image, and N is the total number of pixels;

[0019] L_edge is defined as:

[0020] L_edge=1 / M×Σ(E_e-E_gt) 2 ,

[0021] Among them, E_e is the edge information of the enhanced image, E_gt is the edge information of the real image, and M is the total number of edge pixels.

[0022] As a further improvement of the present invention, the light source control system provides light source brightness adjustment, spectrum adjustment, light source stability control, and uniform illumination detection functions; wherein the spectrum adjustment function of the light source control system is used for multimodal imaging; multimodal imaging includes white light imaging, ICG fluorescence imaging, and narrow-band imaging; the light source control system is connected to the zoom imaging optical path via an optical fiber, and there are multiple optical fibers for transmitting light of different wavelengths to the front end of the exterior mirror.

[0023] As a further improvement of the present invention, the multiple zoom imaging optical paths integrated in the intelligent multi-optical path imaging system are used for simultaneous multi-viewpoint 3D imaging and multimodal imaging; the two imaging optical paths can respectively simulate the imaging systems of the left and right eyes of the human body, and simultaneously collect two sets of images of the same surgical field, which are output to the corresponding 3D display device after three-dimensional synthesis processing by the image processing host; the 3D display devices each correspond to two imaging optical paths, and the remaining optical paths are used for multimodal imaging; the multiple zoom imaging optical paths are distributed in a circular array around the central axis of the intelligent multi-optical path imaging system, satisfying an optical zoom ratio of more than 10 times, and having automatic focus and viewing angle adjustment functions; the intelligent focus controller adopts a rotary button design, and the specific implementation method is as follows: the button can be pressed and rotated, pressing is used to switch between automatic focus and manual focus modes, and rotating is used for manual focus, accompanied by sound feedback to inform the operator of the focusing status.

[0024] As a further improvement of the present invention, the thermal imaging sensor is used to help the surgeon perform tissue perfusion assessment, tumor or inflammation detection, and the results are superimposed on the polarized 3D main screen in real time or presented separately on the polarized 3D secondary screen; the gas sensor is used to detect changes in gas composition in the surgical area, which helps to detect complications early and can perform tissue metabolism monitoring, hypoxia detection, infection or necrosis indication; the Raman spectroscopy sensor provides the surgeon with real-time tissue composition analysis capabilities non-invasively, and the sensor is provided with a small high-sensitivity spectrometer and a sensor data processing unit, and the results are superimposed on the polarized 3D main screen in real time or presented separately on the polarized 3D secondary screen; the temperature and humidity sensor is used to monitor the temperature and humidity of the surgical environment.

[0025] As a further improvement of the present invention, the micro-ultrasound probe is a miniaturized ultrasonic imaging device that provides the surgeon with real-time deep tissue structure information, assists in surgical navigation, and improves its accuracy.

[0026] As a further improvement of the present invention, the transmission method of the 3D scene information obtained after processing by the image processing host and the 3D display device, and the information transmission method of the pressure-sensitive foot pedal and the robotic arm control system are all 5G communication wireless transmission; the pressure-sensitive foot pedal is composed of a base, a pedal, and a fixed block, based on a pressure sensor and a tilt sensor, and there are two on the left and right, which control the orientation and height of the intelligent multi-path imaging system by stepping on it in different ways, and the fixed block is used to fix or release the position of the intelligent multi-path imaging system; the operation screen can help the operator customize different functional combinations of the pedal.

[0027] The beneficial effects of the present invention are:

[0028] 1. It realizes the intraoperative navigation and path planning functions, providing more intelligent services and comfortable user experience for surgeons and patients.

[0029] 2. It can effectively solve the problem of decreased brightness of the surgical field caused by obstruction of the shadowless lamp, provide the surgeon with a clear, natural and detail-rich surgical field view at all times, and improve the quality and efficiency of surgery.

[0030] 3. It provides multi-modal imaging function, which collects surgical field information through the integration of multiple imaging optical paths and performs imaging of other modalities, which helps surgeons to make better judgments and facilitates multi-surgeon collaborative surgery.

[0031] 4. Combining the advantages of manual zoom and automatic zoom to cope with special situations or meet personal preferences, the zoom function has a preset magnification, but also allows the operator to manually fine-tune. Combined with the highly integrated solution of the ring integrated chip, it can realize multifunctional sensing in a very small space, providing comprehensive tissue and environmental information for surgical exoscopes, while maintaining the compactness and operational flexibility of the equipment.

[0032] 5. Both the 3D display device and the pressure-sensitive foot pedal use 5G wireless transmission to transmit information, which simplifies the equipment layout, facilitates equipment expansion and upgrading, improves safety and flexibility, and further meets the needs of modern surgical operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the three-dimensional structure of an intelligent surgical exoscopic system of the present invention.

[0034] Figure 2 It is a side structural schematic diagram of an intelligent surgical exoscopic system of the present invention.

[0035] Figure 3 It is a schematic diagram of the rear structure of an intelligent surgical exoscopic system of the present invention.

[0036] Figure 4 A schematic diagram of a multi-view 3D imaging method of an intelligent multi-light path imaging system according to an embodiment of the present invention.

[0037] Figure 5 This is a schematic structural diagram of a pressure-sensitive foot pedal of an intelligent surgical exoscopic system of the present invention.

[0038] Figure 6 A schematic diagram of a stacking of a ring-shaped integrated chip according to an embodiment of the present invention, wherein arrows represent the direction of data transmission.

[0039] List of reference numerals:

[0040] 1. Universal caster; 2. Four-star foot base; 3. Box; 4. Heat dissipation hole; 5. Operation screen; 6.1. Display rotation bracket; 7. Pneumatic mechanical arm; 8. Pneumatic mechanical arm; 9.1. Arm spherical joint; 9.2. Arm spherical joint; 9.3. Branch arm spherical joint; 10. Pneumatic camera branch arm; 11. Intelligent multi-optical path imaging system; 12. Intelligent focus controller; 13. Polarized 3D main screen; 14. Polarized 3D sub-screen; 15. Head-mounted display device; 16. Drawer; 17.1. Zoom imaging optical path; 18. Ring integrated chip; 19.1. Base; 20.1. Pedal; 21.1. Fixed block; 22. System on chip; 23. Thermal imaging sensor layer; 24. Gas sensor layer; 25. Raman spectrum sensor layer; 26. Temperature and humidity sensor layer. DETAILED DESCRIPTION

[0041] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0042] The present invention provides an intelligent surgical exoscopic system, such as Figure 1 ,2 , 3, including: a work trolley, a pneumatic mechanical arm, an intelligent multi-light path imaging system 11 arranged at the end of the pneumatic mechanical arm, and a 3D display device. The work trolley includes a box 3, a display rotating bracket 6.1, a four-star foot base 2, and a universal caster 1. The pneumatic mechanical arm includes a pneumatic mechanical arm 7, a pneumatic mechanical arm 8, a pneumatic camera arm 10, and a spherical joint, and the spherical joint includes a large arm spherical joint 9.1; a small arm spherical joint 9.2; and a branch arm spherical joint 9.3.

[0043] Specifically, the pneumatic mechanical arm 7 in this embodiment is fixed on the box body 3, connected to the pneumatic mechanical arm 8 through the arm spherical joint 9.1, the two pneumatic mechanical arms 8 and the pneumatic camera arm 10 are connected to each other through the arm spherical joint 9.2, and the pneumatic camera arm 10 is connected to the camera system through the arm spherical joint 9.3. The spherical joint in the embodiment can rotate around three mutually perpendicular axes, providing three degrees of freedom, so the pneumatic mechanical arm in this embodiment is flexible and has a large range of movement, and the longest arm span is 1600mm.

[0044] Specifically, the intelligent multi-light path imaging system 11 includes five integrated zoom imaging light paths 17.1, an intelligent focus controller 12, and a ring integrated chip 18. Figure 4 As shown, the five zoom imaging optical paths 17.1 in this embodiment are distributed in a circular array around the central axis of the intelligent multi-optical path imaging system 11, with a resolution of 3840×2160, an optical zoom ratio of 13 times, and automatic focus and viewing angle adjustment functions.

[0045] Specifically, the annular integrated chip 18 is installed on the periphery of the five zoom imaging optical paths 17.1, with a total height of 6 mm, an inner ring radius of 17 mm, and an outer ring radius of 22 mm. It is made of four MEMS micro sensors, namely, thermal imaging sensor, gas sensor, Raman spectrum sensor, and temperature and humidity sensor, which are stacked vertically and compactly. At the same time, through-silicon via technology (TSV) is combined to achieve inter-layer interconnection, and the periphery is connected to the micro-ultrasound probe and the micro-fiber probe through a flexible printed circuit. TSV technology can establish high-density, low-latency electrical connections between the layers of the annular integrated chip 18, while greatly reducing the horizontal space occupation.

[0046] Specifically, the stacking diagram of the ring-shaped integrated chip 18 is as follows: Figure 6 As shown, the specific configuration is as follows: a high-performance system on chip (SoC) 22 is configured on the top layer, integrating a signal processing unit, an analog-to-digital converter and a communication module to realize real-time data processing and transmission; a thermal imaging sensor layer 23, a gas sensor layer 24, a Raman spectrum sensor layer 25 and a temperature and humidity sensor layer 26 are sequentially configured on the lower layer; the arrow represents the direction of data transmission, and the data is transmitted to the main body of the surgical exoscope.

[0047] The 3D display device includes a polarized 3D main screen 13, a polarized 3D secondary screen 14, and a head-mounted display device (HMD) 15; the polarized 3D display main screen 13 and the secondary screen 14 are connected to the box 3 through the display rotating bracket 6.1, and can be rotated on a fixed axis to facilitate adjustment according to the operator's orientation.

[0048] The head mounted display device 15 includes VR glasses and AR glasses. The size of the polarized 3D display main screen 13 in the embodiment is 24 inches, and the size of the polarized 3D display secondary screen 14 is 12 inches.

[0049] The auxiliary control device includes a pressure-sensitive foot pedal.

[0050] Specifically, the pressure-sensitive foot pedal is composed of a base 19.1, a pedal 20.1, and a fixing block 21.1, and there are two left and right ones. Figure 5 As shown. Based on the pressure sensor and the tilt sensor, the position and height of the intelligent multi-light path imaging system are controlled. The specific implementation method is as follows: step on the front of the pedal 20.1 to move forward; step on the rear of the pedal 20.1 to move backward; step on the left side of the pedal 20.1 to move left; step on the right side of the pedal 20.1 to move right; step on the center to adjust the height; step on both feet to zoom in or out; control the moving speed by tilting the feet; step on the fixed block 21.1 to fix the position of the intelligent multi-light path imaging system 11; lift the fixed block 21.1 to release the fixation; the operation screen 5 can help the operator customize different function combinations.

[0051] The front end of the box 3 is provided with a mechanical arm control system, an image processing host, and a light source control system, and the rear end of the box 3 is provided with a drawer 14 for storing medical equipment, etc. The outside of the box 3 is provided with an operation screen 5 and heat dissipation holes 4.

[0052] The robotic arm control system can provide motion control, posture adjustment, automatic positioning, balance control, safety functions, braking functions, sensor feedback, and integration functions.

[0053] Specifically, the image processing host is mainly used to process the image information obtained by the intelligent multi-optical path imaging system 11 and output it in the output mode required by the user. The output mode includes a naked eye 3D mode and an HMD mode. The naked eye 3D mode is suitable for the polarized 3D main screen 13 and the auxiliary screen 14, and the HMD mode is suitable for the head-mounted display device 15, such as Figure 4 As shown, in this embodiment, the image processing host performs simultaneous output of naked eye 3D mode and HMD mode.

[0054] The image processing host can provide an intraoperative navigation function. The specific implementation method is: using pre-acquired medical imaging data (CT, MRI, etc.) to build a three-dimensional model of the patient's anatomical structure, aligning the real-time position and posture of the exoscopic scope with the three-dimensional model, achieving real-time positioning of the exoscopic scope in the patient's body, displaying the current position and field of view of the exoscopic scope on the three-dimensional model, and providing intraoperative navigation for doctors.

[0055] The image processing host can provide a path planning function. The specific implementation method is: according to the surgical goal and anatomical structure, the optimal path for the intelligent multi-light path imaging system 11 to reach the target part is automatically generated, combined with the navigation system, it is guided to move along the planned path, avoiding key blood vessels and nerve tissues, and the path optimization can be performed considering factors such as the safety of the surgical operation and the operation time.

[0056] Specifically, the image processing host can also comprehensively utilize the brightness compensation model based on deep learning, multi-scale analysis and edge preservation technology to solve the problem of decreased brightness of the surgical field caused by the occlusion of the shadowless lamp during surgery. The specific implementation method is: training a convolutional neural network (CNN) specifically for surgical field brightness compensation, using a large amount of occluded and non-occluded surgical field image data for learning, so that it can automatically estimate and compensate for local brightness; applying the CNN model to surgical field images of multiple scales, and using multi-scale fusion technology to obtain clearer and more natural enhancement results; at the same time, we introduced an edge-aware loss function in the CNN model, and applied an edge-preserving filter in the post-processing stage to ensure that the enhanced image can retain key structural details and avoid image distortion or artifacts.

[0057] The light source control system can provide light source brightness adjustment, spectrum adjustment, light source stability control, and uniform lighting detection functions.

[0058] Specifically, the spectrum adjustment function of the light source control system is used for multimodal imaging; the multimodal imaging includes white light imaging, ICG fluorescence imaging, and narrow-band imaging. The intelligent multi-light path imaging system 11 in this embodiment integrates five zoom imaging light paths 17.1. When the use of the 3D display device is reduced, the zoom imaging light path 17.1 can be reserved for multimodal imaging.

[0059] Specifically, the light source control system is connected to the zoom imaging optical path 17.1 via an optical fiber. In this embodiment, each of the five imaging optical paths 17.1 is provided with an optical fiber connected to the light source control system for light source transmission to meet the needs of multi-modal imaging.

[0060] The operation screen 5 is used to start and shut down the exterior mirror system, manually adjust the pneumatic mechanical arm when necessary, set the output mode of the image processing host, and control the imaging mode.

[0061] Specifically, the five zoom imaging optical paths 17.1 integrated in the intelligent multi-optical path imaging system 11 are used to simultaneously perform multi-viewpoint 3D imaging and multi-modal imaging. Figure 4 A multi-viewpoint 3D imaging method of the intelligent multi-optical path imaging system 11 of the embodiment is illustrated. The 3D scene information collected by the five 3D display devices has different viewing angles. Any two zoom imaging optical paths 15.1 in the intelligent multi-optical path imaging system 11 are selected. The two optical paths can respectively simulate the imaging systems of the left and right eyes of the human body, and simultaneously collect two sets of images of the same surgical field. After the two sets of images are processed by the image processing host for three-dimensional synthesis, they are output to the polarized 3D main screen 13 and the auxiliary screen 14 in the naked eye 3D mode, or output to the head-mounted display device 15 in the HMD mode. The selection of viewing angle depends on the position of the surgeon. Preferably, the adjacent zoom imaging optical paths 17.1 shown in the figure are output to one 3D display device. In this case, it can be ensured that four surgeons perform surgery through the head-mounted display device 15 and one surgeon performs surgery through the polarized 3D main screen 13. When performing multimodal imaging, the use of 2 to 4 head mounted display devices 15 can be reduced accordingly, and the vacated 1 to 3 zoom imaging optical paths 17 . 1 can be used for imaging of other modalities respectively.

[0062] Specifically, the 3D scene information obtained after processing by the image processing host and the information transmission method between the 3D display device, the pressure-sensitive foot pedal and the robotic arm control system is 5G communication wireless transmission. Because the transmitted information involves the privacy data of multiple parties, a specific medical channel will be selected for encrypted transmission during transmission, while ensuring high frame rate and high-quality transmission.

[0063] Specifically, the intelligent focusing controller 12 adopts a rotary button design, and the specific implementation method is: the button can be pressed and rotated, pressing is used to switch between automatic focus and manual focus mode, and rotating is used for manual focus, accompanied by sound feedback to inform the operator of the focusing status.

[0064] Specifically, the thermal imaging sensor is used to help the surgeon perform tissue perfusion assessment, tumor or inflammation detection, etc. Its specific implementation method is: it works by detecting infrared radiation emitted from the surface of an object, converting these radiations into electrical signals and processing them into temperature distribution images. The results are superimposed in real time on the polarized 3D main screen 13 or presented separately on the polarized 3D secondary screen 14.

[0065] Specifically, the gas sensor is an electrochemical sensor used to detect changes in gas composition in the surgical area, which may help to detect complications early and can perform tissue metabolism monitoring, hypoxia detection, infection or necrosis indication.

[0066] Specifically, the Raman spectroscopy sensor can provide the operator with real-time tissue composition analysis capabilities in a non-invasive manner. Its specific implementation method is as follows: a miniature fiber optic probe is connected to an optical fiber to emit near-infrared light to the tissue and collect scattered light from the tissue at the same time. The small high-sensitivity spectrometer on the sensor analyzes the scattered light collected from the tissue, and the sensor data processing unit analyzes the spectral data in real time. The results are superimposed on the polarized 3D main screen 13 or presented separately on the polarized 3D secondary screen 14 in real time.

[0067] Specifically, the temperature and humidity sensor is used to monitor the temperature and humidity of the operating environment.

[0068] Specifically, the micro-ultrasound probe is a miniaturized ultrasonic imaging device that can provide the surgeon with real-time deep tissue structure information, assist in surgical navigation, and improve its accuracy.

[0069] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. An intelligent surgical exoscopic system, characterized in that: It comprises a work trolley, a pneumatic mechanical arm, an intelligent multi-optical path imaging system arranged at the end of the pneumatic mechanical arm, a 3D display device and an auxiliary control device; the work trolley comprises a box body, a display rotating bracket, a four-star foot base and a universal caster, wherein the display rotating bracket and the four-star foot base are respectively installed on the top and bottom of the box body, and the universal caster is cooperatively installed at the four top corners of the four-star foot base; the pneumatic mechanical arm comprises a pneumatic mechanical arm, a pneumatic mechanical arm and a pneumatic camera arm, wherein the lower end of the pneumatic mechanical arm is installed on the box body, the pneumatic mechanical arm has two sections, one of which is connected to the upper end of the pneumatic mechanical arm through a big arm spherical joint, the two sections and the other section are connected to the pneumatic camera arm through a small arm spherical joint, and the pneumatic camera arm is connected through a arm spherical joint. The 3D display device is connected to an intelligent multi-optical path imaging system; the intelligent multi-optical path imaging system includes multiple integrated zoom imaging optical paths, an intelligent focusing controller and a ring-shaped integrated chip; the ring-shaped integrated chip is installed on the periphery of the multiple zoom imaging optical paths, and integrates thermal imaging sensors, gas sensors, Raman spectrum sensors and temperature and humidity sensors, and is connected to the periphery with a miniature ultrasonic probe and a miniature optical fiber probe; the 3D display device includes a polarized 3D main screen, a polarized 3D secondary screen and a head-mounted display device; the polarized 3D main screen and the polarized 3D secondary screen are connected to the box through a display rotating bracket and can be rotated on a fixed axis; the head-mounted display device includes VR glasses and AR glasses; the auxiliary control device includes a pressure-sensitive foot pedal for controlling the orientation and height of the intelligent multi-optical path imaging system.

2. The intelligent surgical exoscopic system according to claim 1, characterized in that: A robotic arm control system, an image processing host and a light source control system are provided at the front end of the box, a drawer for storing medical equipment, etc. is provided at the rear end of the box, and an operation screen and heat dissipation holes are provided on the outside of the box. The operation screen is used to start and shut down the exterior mirror system, manually adjust the pneumatic robotic arm, set the output mode of the image processing host, and control the imaging mode; the heat dissipation holes are used to dissipate heat from the robotic arm control system, the image processing host and the light source control system; the robotic arm control system is used to provide motion control, posture adjustment, automatic positioning, balance control, safety functions, braking functions, sensor feedback and integration functions.

3. The intelligent surgical exoscopic system according to claim 2, characterized in that: The image processing host is used to process the image information obtained by the camera system and output it in the output mode required by the user. The output mode includes a naked-eye 3D mode and an HMD mode. The naked-eye 3D mode is suitable for a polarized 3D main screen and a polarized 3D secondary screen, and the HMD mode is suitable for a head-mounted display device.

4. The intelligent surgical exoscopic system according to claim 2, characterized in that: The image processing host provides intraoperative navigation function and path planning function, wherein the specific implementation method of the intraoperative navigation function is: using pre-acquired medical image data to build a three-dimensional model of the patient's anatomical structure, aligning the real-time position and posture of the exoscope with the three-dimensional model, realizing real-time positioning of the exoscope in the patient's body, displaying the current position and field of view of the exoscope on the three-dimensional model, and providing intraoperative navigation for the doctor; the specific implementation method of the path planning function is: according to the surgical objectives and anatomical structures, automatically generating the optimal path for the exoscope to reach the target site, combining with the navigation system, guiding the exoscope to move along the planned path, avoiding key blood vessels and nerve tissues.

5. The intelligent surgical exoscopic system according to claim 2, characterized in that: The image processing host also solves the problem of decreased brightness of the surgical field caused by the shadowless lamp being blocked during surgery. The specific implementation method is as follows: training a CNN model for surgical field brightness compensation; applying the CNN model to surgical field images of multiple scales, and using multi-scale fusion technology to obtain clearer and more natural enhancement results; at the same time, introducing an edge-aware loss function, and applying an edge-preserving filter in the post-processing stage to ensure that the enhanced image can retain key structural details and avoid image distortion or artifacts; The loss function is defined as: L = L_mse + λ × L_edge, Among them, L_mse is the mean square error loss function, which is used to measure the pixel-level difference between the enhanced image and the real image; L_edge is the edge loss function, which is used to measure the difference in edge information between the enhanced image and the real image; the Sobel operator or Canny operator is used to extract the image edge and calculate the difference in edge pixels; λ is the weight coefficient, which is used to balance the importance of the two loss functions; Among them, L_mse is defined as: L_mse=1 / N×Σ(I_e-I_gt) 2 , Among them, I_e is the enhanced image, I_gt is the real image, and N is the total number of pixels; L_edge is defined as: L_edge=1 / M×Σ(E_e-E_gt) 2 , Among them, E_e is the edge information of the enhanced image, E_gt is the edge information of the real image, and M is the total number of edge pixels.

6. The intelligent surgical exoscopic system according to claim 1, characterized in that: The light source control system provides light source brightness adjustment, spectrum adjustment, light source stability control, and uniform illumination detection functions; wherein the spectrum adjustment function of the light source control system is used for multimodal imaging; multimodal imaging includes white light imaging, ICG fluorescence imaging, and narrow-band imaging; the light source control system is connected to the zoom imaging optical path via an optical fiber, and there are multiple optical fibers for transmitting light of different wavelengths to the front end of the exterior mirror.

7. The intelligent surgical exoscopic system according to claim 2, characterized in that: The multiple zoom imaging optical paths integrated in the intelligent multi-optical path imaging system are used for multi-viewpoint 3D imaging and multi-modal imaging at the same time; the two imaging optical paths can respectively simulate the imaging systems of the left and right eyes of the human body, and simultaneously collect two sets of images of the same surgical field. After the two sets of images are processed by three-dimensional synthesis by the image processing host, they are output to the corresponding 3D display device; the 3D display devices each correspond to two imaging optical paths, and the remaining optical paths are used for multi-modal imaging; the multiple zoom imaging optical paths are distributed in a circular array around the central axis of the intelligent multi-optical path imaging system, meeting the optical zoom ratio of more than 10 times, and having automatic focus and viewing angle adjustment functions; the intelligent focus controller adopts a rotary button design, and the specific implementation method is as follows: the button can be pressed and rotated, pressing is used to switch between automatic focus and manual focus mode, and rotating is used for manual focus, accompanied by sound feedback to inform the operator of the focusing status.

8. The intelligent surgical exoscopic system according to claim 2, characterized in that: The thermal imaging sensor is used to help the surgeon perform tissue perfusion assessment, tumor or inflammation detection, and the results are superimposed on the polarized 3D main screen in real time or presented separately on the polarized 3D secondary screen; the gas sensor is used to detect changes in gas composition in the surgical area, which helps to detect complications early and can perform tissue metabolism monitoring, hypoxia detection, infection or necrosis indication; the Raman spectroscopy sensor provides the surgeon with real-time tissue composition analysis capabilities non-invasively, and the sensor is equipped with a small high-sensitivity spectrometer and a sensor data processing unit, and the results are superimposed on the polarized 3D main screen in real time or presented separately on the polarized 3D secondary screen; the temperature and humidity sensor is used to monitor the temperature and humidity of the surgical environment.

9. The intelligent surgical exoscopic system according to claim 1, characterized in that: The micro-ultrasound probe is a miniaturized ultrasonic imaging device that provides the surgeon with real-time deep tissue structure information, assists in surgical navigation, and improves its accuracy.

10. The intelligent surgical exoscopic system according to claim 1, characterized in that: The transmission method of the 3D scene information obtained after processing by the image processing host and the 3D display device, and the information transmission method of the pressure-sensitive foot pedal and the robotic arm control system are all 5G communication wireless transmission; the pressure-sensitive foot pedal is composed of a base, a pedal, and a fixed block. It is based on a pressure sensor and a tilt sensor. There are two left and right ones. The orientation and height of the intelligent multi-path imaging system are controlled by stepping on it in different ways. The fixed block is used to fix or release the position of the intelligent multi-path imaging system; the operation screen can help the operator customize different functional combinations of the pedal.

Images