Three-dimensional external-view mirror surgery system

Abstract

The application discloses a three-dimensional external-view mirror surgery system, and belongs to the technical field of medical instrument control, which comprises the following modules: an external-view microscope module, which is used for microscopic shooting and light illumination of a surgery area by using an external-view microscope; an image display module, which is used for output display of the content of the microscopic shooting; a human body shooting module, which is used for shooting of a doctor's face, eyeball and body posture; and a control and adjustment module, which is used for control and adjustment of the image display module according to the content of the shooting of the human body shooting module. The three-dimensional external-view mirror surgery system can realize automatic adjustment of image display, automatic movement of a display screen and improvement of the quality and efficiency of surgery.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical instrument control, and in particular relates to a three-dimensional external vision mirror surgery system. BACKGROUND

[0002] With the rapid development of medical imaging technology, the field of surgical operation gradually moves towards minimally invasive and precision. As an important auxiliary tool, the external vision mirror system plays an important role in minimally invasive surgery, open surgery, remote collaboration and other fields. The external vision mirror system transmits the image of the surgical area to the display in real time through the camera for the doctor to observe and operate, so that the doctor can perform precise surgical operation with as little exposure of the patient's internal tissues as possible, thereby reducing surgical trauma, shortening postoperative recovery time and improving surgical effect.

[0003] At present, there are many types of surgical external vision mirror systems available on the market, but there are still some problems that cannot be ignored. The external vision mirror is low in intelligence and convenience, the lens cannot be automatically zoomed, additional operation is required for zooming, which affects the surgical efficiency; the display screen position is fixed and difficult to adjust in time, when the operation time is long, the doctor will maintain a posture for a long time to adapt to the display screen, which will cause physical discomfort and may affect the operation quality. SUMMARY

[0004] To solve the above problems and technical defects, the present application adopts the following technical solution, a three-dimensional external vision mirror surgery system, comprising:

[0005] A microscopic external vision module is used for microscopic photography and light illumination of the surgical area using a microscopic external vision mirror.

[0006] An image display module is used for outputting and displaying the content of microscopic photography.

[0007] A human body photography module is used for photographing the doctor's face, eyeball and body posture.

[0008] A control and adjustment module is used for controlling and adjusting the image display module according to the content photographed by the human body photography module.

[0009] Preferably, the human body photography module comprises a face positioning module, an eyeball tracking module and a posture recognition module.

[0010] The face positioning module is used for determining the face orientation and position of the doctor.

[0011] The eyeball tracking module is used for tracking and positioning the eyeball of the doctor.

[0012] The posture recognition module is used for positioning the posture of the patient and the surgical area of the patient, and determining the body posture and body orientation of the doctor according to the positioning result.

[0013] Further, the control adjustment module comprises a three-dimensional adjustment module, a focal length adjustment module and a rotation adjustment module.

[0014] The three-dimensional adjustment module is configured to output a three-dimensional adjustment signal according to the face orientation and position of the doctor, and control the position of the image display module so that the position and orientation of the image display module match the face orientation and position of the doctor.

[0015] The focal length adjustment module is configured to determine the fixation point of the image displayed by the image display module according to the tracking and positioning result of the eyeball of the doctor, and output a focal length adjustment signal to control the image display module to perform aggregation processing on the fixation point.

[0016] The rotation adjustment module is configured to output a rotation adjustment signal according to the body posture and body orientation of the doctor, and perform rotation processing on the image displayed by the image display module so that the displayed image matches the body posture and body orientation of the doctor.

[0017] Further, the image display module comprises a three-dimensional adjustment support and a display screen, and the display screen is installed on the three-dimensional adjustment support and controlled by the three-dimensional adjustment signal.

[0018] Preferably, the image display module comprises an optical recognition module configured to integrate visible light and near-infrared band illumination, output band light matching the use environment, and simultaneously eliminate the reflection of surgical metal instruments by the built-in ring LED array.

[0019] Preferably, the device further comprises a voice control module configured to recognize the language of the doctor, convert the recognition result into a control signal, and adjust and control the image display module and the image display module.

[0020] Further, the control signal output by the voice control module has a higher priority than the adjustment signal output by the gesture recognition module.

[0021] A three-dimensional exoscope surgical device, comprising a service processor and a distributed memory, the service processor is connected to the memory, the distributed memory stores a service self-management program configured to store machine-readable instructions, the service processor executes the service self-management program, and the instructions are executed by the processor to implement a three-dimensional exoscope surgical system as described above.

[0022] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the content of a three-dimensional exoscope surgical system as described above when executing the program.

[0023] A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the content of a three-dimensional external view mirror surgery system as described above.

[0024] Compared with the prior art, the beneficial effects of the present application are:

[0025] The present application monitors and collects information of the doctor's face, eyeball, body posture and orientation in real time through the human body shooting module, and transmits the collected information to the control and adjustment module, the three-dimensional adjustment module, the focal length adjustment module and the rotation adjustment module respectively. The three-dimensional adjustment module, the focal length adjustment module and the rotation adjustment module process the received information respectively to generate corresponding adjustment signals. The three-dimensional adjustment signal controls the three-dimensional adjustment support of the image display module to drive the display screen to adjust the position and orientation. The focal length adjustment signal controls the microscopic external view module to focus on the gaze point of the surgery area. The rotation adjustment signal controls the image display module to rotate the display image. Through the continuous cooperative work of each module, the automatic adjustment of the image display during the surgery is realized, the automatic movement of the display screen is realized, and the quality and efficiency of the surgery are improved. BRIEF DESCRIPTION OF DRAWINGS

[0026] In the drawings:

[0027] Figure 1 The figure is a schematic diagram of the system structure of the embodiment of the present application;

[0028] Figure 2 The figure is a schematic diagram of the equipment structure of the embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0030] Embodiment 1

[0031] As Figure 1 shown, a three-dimensional external view mirror surgery system includes:

[0032] A microscopic external view module is used for microscopic shooting and light illumination of the surgery area using a microscopic external view mirror.

[0033] The microscopic external view module includes an optical recognition module for integrating visible light and near-infrared band illumination, outputting band light illumination matched with the use environment, and simultaneously eliminating the reflection of surgical metal instruments by built-in ring LED arrays.

[0034] The microscope module integrates a high-resolution microscope lens and a shadowless operating lamp, and performs real-time three-dimensional microscopic photography and illumination on the surgical area. The microscope module is internally provided with an electric zoom lens and supports remote control of focal length adjustment.

[0035] The image display module is configured to output and display the content of the microscopic photography.

[0036] The image display module comprises a three-dimensional adjustment support and a display screen. The display screen is mounted on the three-dimensional adjustment support and is controlled and adjusted by the three-dimensional adjustment signal.

[0037] The three-dimensional adjustment support adopts a six-degree-of-freedom mechanical arm structure, and the driving end is connected to the display screen. The three-dimensional adjustment support has a multi-degree-of-freedom adjustment capability, and is internally provided with a driving component adapted to the control and adjustment module, which can receive the three-dimensional adjustment signal sent by the control and adjustment module.

[0038] Under the action of the three-dimensional adjustment signal, the three-dimensional adjustment support can realize translational adjustment along the X, Y and Z axes and rotational adjustment around the corresponding axes, thereby changing the position and orientation of the display screen, so that the position and orientation of the display screen are matched with the face orientation and position of the doctor, and the doctor can comfortably observe the image.

[0039] The display screen adopts a high-refresh-rate medical OLED display screen, supports 4K / 3D image output, and has display resolution and color restoration degree meeting the high-precision requirement of surgical observation.

[0040] The human body photography module is configured to photograph the face, eyeballs and body posture of the doctor, capture the face, eyeballs and body posture data of the doctor in real time, and locate the surgical area of the patient.

[0041] The human body photography module comprises a face positioning module, an eyeball tracking module and a posture recognition module.

[0042] The face positioning module is configured to determine the face orientation and position of the doctor. The face positioning module adopts a binocular depth camera, determines the spatial coordinates and orientation angle of the face of the doctor through feature point recognition, adopts image recognition technology, continuously photographs the face of the doctor, extracts features from the photographed face image, obtains key feature points of the face, such as the corner of the eye, the tip of the nose and the corner of the mouth, determines the face orientation and specific position of the doctor in space based on the position change of the key feature points, and transmits the determination result to the three-dimensional adjustment module of the control and adjustment module in real time.

[0043] The eye-tracking module is used to track and locate the doctor's eyes. It is equipped with an infrared active light source and a high-speed camera to capture the doctor's eye movement trajectory in real time and calculate the coordinates of the gaze point. The acquired eye images are processed to extract feature information such as the pupil center and corneal reflection point. By analyzing the change trajectory of the feature information, the real-time tracking and positioning of the doctor's eyes can be achieved, and the position of the doctor's gaze point on the image displayed on the image display module can be determined. The gaze point information is then sent to the focus adjustment module of the control and adjustment module.

[0044] The posture recognition module is used to locate the patient's posture and surgical area. Based on the positioning results, it determines the doctor's body posture and orientation. Using a wide-angle camera combined with a skeletal key point detection algorithm, it identifies the doctor's body posture and the location of the patient's surgical area. It performs feature analysis on the patient's posture image to determine the patient's body position. It identifies and locates the surgical area image to clarify the spatial range of the surgical area. Combining the patient's posture and the positioning results of the surgical area, it calculates the doctor's body posture and orientation to adapt to the surgical operation through spatial position relationship calculation and transmits it to the rotation adjustment module of the control and adjustment module.

[0045] The control and adjustment module is used to control and adjust the image display module according to the content captured by the human body shooting module.

[0046] The control and adjustment module includes: a three-dimensional adjustment module, a focal length adjustment module, and a rotation adjustment module;

[0047] The three-dimensional adjustment module is used to output three-dimensional adjustment signals based on the doctor's facial orientation and position to control the position of the image display module, so that the position and orientation of the image display module match the doctor's facial orientation and position.

[0048] It receives facial positioning data, calculates the coordinates and rotation angles that the display screen needs to move, and drives the three-dimensional adjustment bracket to make the screen face the doctor's line of sight.

[0049] According to the preset position matching algorithm, the doctor's facial position and orientation parameters are converted into corresponding three-dimensional adjustment signals. The three-dimensional adjustment signals are transmitted to the three-dimensional adjustment bracket of the image display module, which controls the three-dimensional adjustment bracket to drive the display screen to perform translation and rotation movements until the position and orientation of the display screen reach the best matching state with the doctor's facial orientation and position, ensuring that the doctor can clearly observe the screen without deliberately adjusting his head posture.

[0050] The focus adjustment module is used to determine the doctor's gaze point on the image displayed on the image display module based on the doctor's eye tracking and positioning results. It outputs a focus adjustment signal to control the microscopic external vision module to focus the gaze point.

[0051] Based on eye-tracking data, the position of the doctor's gaze point on the display screen is mapped, and the target point corresponding to the surgical area is calculated in reverse. The electric zoom lens of the microscopic external vision module is then controlled to automatically focus on that point.

[0052] Based on the coordinates of the gaze point in the image and the imaging parameters of the external microscope module, a focus adjustment signal is generated and transmitted to the external microscope module to control the optical components inside the microscope head to adjust their positions. This allows the microscope head to precisely focus on the position corresponding to the gaze point in the surgical area, enhancing the image clarity of that area and enabling automatic zooming of the lens.

[0053] The rotation adjustment module is used to control the output rotation adjustment signal according to the doctor's body posture and orientation, and to rotate the image displayed by the image display module so that the displayed image matches the doctor's body posture and orientation.

[0054] Based on posture recognition data, the displayed image is rotated in real time using an affine transformation to ensure that the image orientation is consistent with the doctor's perspective.

[0055] The rotation adjustment signal is transmitted to the image display module to control the display screen to adjust the rotation angle of the displayed surgical image, so that the orientation of the displayed image matches the doctor's body posture and orientation, avoiding visual discomfort caused by the doctor's body posture not matching the image orientation, and improving the coordination of the surgical operation.

[0056] It also includes a voice control module, which is used to recognize the doctor's speech and convert the recognition results into control signals to adjust and control the image display module and the microscopic external vision module.

[0057] The control signals output by the voice control module have a higher priority than the adjustment signals output by the gesture recognition module.

[0058] Example 2

[0059] like Figure 2 As shown, a three-dimensional exoscopic surgical device includes a service processor and a distributed memory. The service processor is connected to the memory, and the distributed memory stores a service self-management program configured to store machine-readable instructions. The service processor executes the service self-management program, and the instructions, when executed by the processor, implement the three-dimensional exoscopic surgical system as described in Embodiment 1.

[0060] Example 3

[0061] From a hardware perspective, this application provides an embodiment of an electronic device comprising all or part of a three-dimensional exoscopic surgical system. The electronic device includes a service processor and a distributed memory. The service processor is connected to the memory. The distributed memory stores a service self-management program configured to store machine-readable instructions. The service processor executes the service self-management program. When the instructions are executed by the processor, they implement the three-dimensional exoscopic surgical system as described above.

[0062] From a hardware perspective, in order to effectively improve the flexibility, versatility, and efficiency of data acquisition, this application provides embodiments of electronic devices comprising all or part of a three-dimensional external endoscopic surgical system. The electronic devices specifically include the following components:

[0063] The system includes a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the core business system of the three-dimensional external endoscopic surgery system, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., but this embodiment is not limited to these.

[0064] In this embodiment, the logic controller can be implemented with reference to the embodiment of the three-dimensional external endoscopic surgical system, the content of which is incorporated herein, and repeated parts will not be described again.

[0065] It is understood that the user terminal may include smartphones, tablet electronic devices, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc., wherein the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0066] In practical applications, some parts of the three-dimensional exoscopic surgical system can be performed on the electronic device side as described above, or all operations can be completed in the client device. The choice depends on the processing power of the client device and the limitations of the user's usage scenario, and this application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0067] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster composed of multiple servers, or a server structure of a distributed device.

[0068] Example 4

[0069] Embodiments of this application also provide a computer-readable storage medium capable of implementing the three-dimensional exoscopic surgical system with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all the contents of the three-dimensional exoscopic surgical system with a server or client as the execution subject in the above embodiments.

[0070] The embodiments of this application may be provided as methods, apparatus, or computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A three-dimensional, external-viewing, surgical system, characterized by, The application relates to a surgical microscope system, which comprises the following modules: a micro-external view module for micro-photographing and light illumination of a surgical area by using a micro-external view mirror; an image display module for outputting and displaying the micro-photographed content; a human body photographing module for photographing the face, eyeball and body posture of a doctor; the human body photographing module comprises a face positioning module, an eyeball tracking module and a posture recognition module; the face positioning module is used for determining the face orientation and position of the doctor, adopts a binocular depth camera, determines the spatial coordinates and orientation angle of the face of the doctor through feature point recognition, adopts an image recognition technology, continuously photographs the face of the doctor, extracts features of the photographed face image, obtains key feature points of the face, determines the face orientation and specific position of the doctor in space based on the position changes of the key feature points, and transmits the determination result to a three-dimensional adjusting module in a control adjusting module in real time; the eyeball tracking module is used for tracking and positioning the eyeball of the doctor, is equipped with an infrared active light source and a high-speed camera, captures the eyeball movement track of the doctor in real time and calculates the gaze point coordinates, processes the collected eyeball image, extracts feature information of the eyeball, realizes real-time tracking and positioning of the eyeball of the doctor by analyzing the change track of the feature information, determines the gaze point position of the doctor on the image displayed by the image display module, and sends the gaze point information to a focal length adjusting module of the control adjusting module; the posture recognition module is used for positioning the posture of the patient and the surgical area of the patient, determines the body posture and body orientation of the doctor according to the positioning result, recognizes the body posture of the doctor and the position of the surgical area of the patient by combining a wide-angle camera with a skeleton key point detection algorithm, analyzes the feature of the posture image of the patient, determines the body placing state of the patient, recognizes and positions the surgical area image, clearly defines the position range of the surgical area in space, determines the body posture and body orientation of the doctor presented for adapting to the surgical operation by combining the positioning results of the posture of the patient and the surgical area and through space position relationship calculation, and transmits the body posture and body orientation to a rotating adjusting module of the control adjusting module; the control adjusting module is used for controlling and adjusting the image display module according to the content photographed by the human body photographing module; the control adjusting module comprises a three-dimensional adjusting module, a focal length adjusting module and a rotating adjusting module; the three-dimensional adjusting module is used for outputting a three-dimensional adjusting signal according to the face orientation and position of the doctor, controlling the position of the image display module, and matching the position and orientation of the image display module with the face orientation and position of the doctor; the focal length adjusting module is used for determining the gaze point of the doctor on the image displayed by the image display module according to the tracking and positioning result of the eyeball of the doctor, and outputting a focal length adjusting signal to control the micro-external view module to gather and process the gaze point; the rotating adjusting module is used for controlling and outputting a rotating adjusting signal according to the body posture and body orientation of the doctor, rotating the image displayed by the image display module, and matching the displayed image with the body posture and body orientation of the doctor.

2. The three-dimensional periscope surgical system of claim 1, wherein, The image display module comprises a three-dimensional adjusting support and a display screen, the display screen is installed on the three-dimensional adjusting support, and the three-dimensional adjusting support is controlled and adjusted by the three-dimensional adjusting signal.

3. The three-dimensional periscope surgical system of claim 1, wherein, The microscopic external viewing module comprises an optical identification module, which is used for integrating visible light and near-infrared band illumination, outputting band light illumination matched with the use environment, and eliminating reflection of surgical metal instruments by means of the built-in annular LED array.

4. The three-dimensional periscope surgical system of claim 1, wherein, The voice control module is further included, which is used for recognizing the language of the doctor, converting the recognition result into a control signal, and adjusting and controlling the image display module and the microscopic external viewing module.

5. The three-dimensional periscope surgical system of claim 4, wherein, The control signal output by the voice control module has a higher priority than the adjustment signal output by the gesture recognition module.

6. A three-dimensional, externally visible, surgical device, characterized by The device comprises a service processor and a distributed memory, the service processor is connected to the memory, the distributed memory stores a service self-management program configured to store machine-readable instructions, the service processor executes the service self-management program, and the instructions are executed by the processor to implement the three-dimensional external viewing mirror surgical system of claim 1.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the content of the three-dimensional external viewing mirror surgical system of claim 1.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the content of the three-dimensional external viewing mirror surgical system of claim 1.

Images