Operation training system and method based on multi-dimensional data perception fusion

Abstract

The invention belongs to the technical field of medical training, and discloses an operation training system and method based on multi-dimensional data perception fusion, and the system comprises an image collection module, a six-degree-of-freedom operation mechanism, a somatosensory and tactile feedback glove, an operation training robot, a simulation heart model, and a control system. The image acquisition module is in signal connection with the control system and is used for acquiring a 3D high-definition image and instrument space coordinates of an operation area in real time; and the six-degree-of-freedom operating mechanism is in signal connection with the control system and is used for converting an operation instruction of a doctor into a control signal of the operation training robot and feeding back force sense information. According to the scheme, a high-simulation operation training environment is provided through the six-degree-of-freedom teleoperation mechanism, the somatosensory and tactile sensing gloves, the operation training robot and the simulation heart model made of the multi-dimensional sensing electronic skin, multi-dimensional recording and feedback of operation operation are achieved, and the operation training efficiency is improved by combining a transfer learning method. And accurate recording and reproduction of the operation process of the qualified doctor are realized.

Description

Technical Field

[0001] This invention relates to the field of medical training technology, and more specifically, to a surgical training system and method based on multi-dimensional data perception fusion. Background Technology

[0002] Traditional cardiac surgery training relies mainly on on-site guidance from senior physicians and limited simulation training equipment, lacking multi-dimensional data perception and feedback mechanisms.

[0003] In recent years, virtual reality (VR) and robotics technologies have been gradually applied to surgical training, but the following technical bottlenecks still exist:

[0004] 1. Lack of tactile feedback: Existing virtual training systems mostly rely on visual simulation, lacking the realism of force-tactile interaction. The few devices that integrate tactile feedback suffer from low accuracy and high latency, making it difficult to simulate the mechanical properties of heart tissue.

[0005] 2. Limited data dimensions: Conventional training systems only record operational trajectories or single mechanical parameters, failing to achieve real-time fusion of multimodal data, resulting in a one-sided operational evaluation system.

[0006] 3. Lack of intelligent guidance: The system usually only provides basic operation repetitive training and cannot provide personalized guidance based on expert experience, resulting in poor training effect for interns. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing a surgical training system and method based on multi-dimensional data perception fusion.

[0008] To solve the above problems, the present invention adopts the following technical solution:

[0009] A surgical training system based on multi-dimensional data perception fusion includes an image acquisition module, a six-degree-of-freedom manipulator, a somatosensory and tactile feedback glove, a surgical training robot, a simulated heart model, and a control system.

[0010] The image acquisition module is connected to the control system signal and is used to acquire 3D high-definition images of the surgical area and the spatial coordinates of the instruments in real time.

[0011] The six-degree-of-freedom operating mechanism is connected to the control system signal, and is used to convert the doctor's operating instructions into control signals for the surgical training robot and to provide feedback force information.

[0012] The somatosensory and tactile feedback gloves are used to collect the doctor's hand movement data and tactile feedback signals, and the somatosensory and tactile feedback gloves are connected to the control system;

[0013] The surgical training robot is connected to the control system. The surgical training robot is controlled by a six-degree-of-freedom operating mechanism and is used to simulate surgical operations and interact with a simulated heart model.

[0014] The simulated heart model is made of multidimensional sensing electronic skin, which is used to sense the contact position of the scalpel, the applied force, and pain signals.

[0015] The control system integrates data acquisition, force feedback synchronization, and operation evaluation functions to compare and analyze the operation procedures of interns and senior doctors and generate improvement suggestions.

[0016] As a further aspect of the present invention, it also includes an operation console, on which the six-degree-of-freedom operating mechanism is mounted, the control system is integrated within the operation console, and a three-dimensional display device is mounted on the operation console via a bracket.

[0017] As a further aspect of the present invention: the image acquisition module includes a binocular stereo camera and an optical tracking sensor, used to simultaneously acquire 3D images of the surgical area and the spatial coordinates of the instruments.

[0018] As a further aspect of the present invention: the somatosensory and tactile feedback glove includes electronic skin, a finger curvature sensor, and a fingertip vibration motor. The electronic skin is used to sense force-controlled behavior, the curvature sensor collects hand movement data, and the vibration motor converts the force information of the surgical training robot into vibration frequency feedback to the user.

[0019] As a further aspect of the present invention: the six-degree-of-freedom operating mechanism includes a force feedback device, which dynamically adjusts the operating resistance based on the sensing signals from the simulated heart model.

[0020] As a further aspect of the present invention: the electronic skin is a flexible piezoresistive array that covers the palm and fingertip areas for high-precision measurement of contact pressure distribution.

[0021] As a further aspect of the present invention: the simulated heart model incorporates multimodal sensors, including pressure sensors, deformation sensors, and temperature sensors, to simulate the mechanical and physiological characteristics of heart tissue.

[0022] As a further aspect of the present invention: the control system includes:

[0023] The central processing unit (CPU) is the core scheduling hub of the system, coordinating data interaction and instruction transmission between multiple modules.

[0024] The multimodal data acquisition module collects visual data from the image acquisition module, motion trajectories of the six-degree-of-freedom manipulator and surgical training robot, force information from the somatosensory and tactile feedback gloves, and the movement trajectory and cutting force of the scalpel on the simulated heart model.

[0025] The force feedback synchronous control module dynamically adjusts the drive parameters of the surgical training robot based on the mechanical feedback signal from the simulated heart model.

[0026] The comparative analysis module is used to compare the operating procedures of interns with those of senior doctors.

[0027] The operation assessment module generates a scoring report and provides improvement suggestions by comparing the deviations between the operation trajectories of interns and experts;

[0028] The data storage module saves the operation records of senior physicians and interns, including instrument movement trajectories, force intensity curves, and operation timestamps. It provides a historical data retrieval interface for the comparative analysis module and supports the dynamic loading of training records from different stages for longitudinal performance evaluation. Every operation performed by interns can be instantly compared with expert data, forming a real-time iteration of "operation-feedback-optimization".

[0029] The remote communication module is used to transmit high-definition video streams and synchronous control commands to remote terminals in real time.

[0030] This invention also provides a surgical training method based on multi-dimensional data perception fusion, comprising the following steps:

[0031] S1. Senior doctors wear haptic and tactile feedback gloves and control the surgical training robot through a six-degree-of-freedom operating mechanism to perform surgical demonstrations on a simulated heart model. The system records the operation process.

[0032] S2. Interns receive surgical training through the system, which records their procedures in real time.

[0033] S3. Compare and analyze the operating procedures of interns with those of senior doctors;

[0034] S4. Based on the comparison results, provide improvement suggestions to enhance the surgical level.

[0035] Compared with the prior art, the advantages of this invention are:

[0036] This solution provides a highly realistic surgical training environment using a six-degree-of-freedom teleoperation mechanism, motion-sensing and tactile gloves, a surgical training robot, and a simulated heart model created with multi-dimensional sensing electronic skin. This enables multi-dimensional recording and feedback of surgical procedures. Combined with transfer learning methods, it achieves accurate recording and reproduction of the surgical process performed by experienced physicians. Through comparative analysis, personalized improvement suggestions are provided to interns, enhancing the effectiveness of their surgical skills training. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the structure of the present invention;

[0038] Figure 2 This is a system diagram of the present invention;

[0039] Figure 3 This is a system diagram of the control system of the present invention;

[0040] Figure 4 This is a flowchart of the present invention.

[0041] Explanation of the labels in the diagram:

[0042] 1. Image acquisition module; 11. Binocular stereo camera; 12. Optical tracking sensor; 2. Six-degree-of-freedom manipulator; 21. Force feedback device; 3. Somatosensory and tactile feedback glove; 31. Electronic skin; 32. Bending sensor; 33. Vibration motor; 4. Surgical training robot; 5. Simulated heart model; 51. Pressure sensor; 52. Deformation sensor; 53. Temperature sensor; 6. Control system; 61. Central processing unit; 62. Multimodal data acquisition module; 63. Force feedback synchronization control module; 64. Comparative analysis module; 65. Operation evaluation module; 66. Data storage module; 67. Remote communication module; 7. Operation console; 8. 3D display device. Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] Please see Figure 1-3The surgical training system based on multi-dimensional data perception fusion includes an image acquisition module 1, a six-degree-of-freedom manipulator 2, a somatosensory and tactile feedback glove 3, a surgical training robot 4, a simulated heart model 5, a control system 6, and an operation console 7. The six-degree-of-freedom manipulator 2 is mounted on the operation console 7, the control system 6 is integrated within the operation console 7, and a three-dimensional display device 8 is mounted on the operation console 7 via a bracket.

[0046] The image acquisition module 1 is connected to the control system 6 via signals to acquire real-time 3D high-definition images of the surgical area and the spatial coordinates of the instruments. The image acquisition module 1 includes a binocular stereo camera 11 and an optical tracking sensor 12, used to simultaneously acquire 3D images of the surgical area and the spatial coordinates of the instruments. The binocular stereo camera 11 and the optical tracking sensor 12 achieve sub-millimeter-level spatial positioning, ensuring the visualization accuracy of the surgical area, and forming a stereoscopic visual closed loop in conjunction with the 3D display device 8.

[0047] The six-degree-of-freedom (DOF) manipulator 2 is signal-connected to the control system 6, used to convert the doctor's operating commands into control signals for the surgical training robot 4 and provide force feedback information. The six-DOF manipulator 2 includes a force feedback device 21, which dynamically adjusts the operating resistance based on the sensor signals from the simulated heart model 5. The six-DOF manipulator 2 employs a parallel / series hybrid mechanism to convert the doctor's hand movements into the spatial pose of the robot's end effector in real time, ensuring 1:1 motion reproduction. Based on multimodal sensor data from the simulated heart model 5, the force feedback device 21 achieves continuous resistance adjustment from 0-20N through an electric servo motor or magnetorheological damper, simulating real surgical resistance.

[0048] The somatosensory and tactile feedback glove 3 is used to collect the doctor's hand movement data and tactile feedback signals. The somatosensory and tactile feedback glove 3 is connected to the control system 6. The somatosensory and tactile feedback glove 4 includes an electronic skin 31, a finger flexure sensor 32, and a fingertip vibration motor 33. The electronic skin 31 senses force-controlled behavior, the flexure sensor 32 collects hand movement data, and the vibration motor 33 converts the force information from the surgical training robot 4 into vibration frequency feedback to the user. The electronic skin 31 is a flexible piezoresistive array covering the palm and fingertip areas, used for high-precision measurement of contact pressure distribution. Through pressure distribution pattern recognition, it distinguishes six typical surgical gestures such as pinching, gripping, and pushing, preventing misoperation. The finger flexure sensor 32 (based on fiber optics or strain gauges) monitors the hand joint angles, and combined with an IMU, realizes the six-DOF pose calculation of the hand.

[0049] The surgical training robot 4 is connected to the control system 6. The surgical training robot 4 is controlled by the six-degree-of-freedom manipulator 2 and is used to simulate surgical operations and interact with the simulated heart model 5. The six-degree-of-freedom manipulator 2 (master end) and the surgical training robot 4 (slave end) adopt position-force hybrid control. The surgeon's movements at the master end are mapped to the slave end instruments through kinematic calculations to achieve sub-millimeter level operational accuracy.

[0050] The simulated heart model 5 is made of multi-dimensional sensing electronic skin, used to sense the contact position, applied force, and pain signals of the scalpel. The simulated heart model 5 incorporates multimodal sensors, including a pressure sensor 51, a deformation sensor 52, and a temperature sensor 53, simulating the mechanical and physiological characteristics of heart tissue. The pressure sensor 51 monitors the contact pressure of the scalpel / instrument in real time, avoiding excessive force that could cause tissue damage. The deformation sensor 52 detects the degree of local deformation of the heart model, reflecting the mechanical invasiveness of the surgical procedure. The temperature sensor 53 simulates the thermal effect of an electrosurgical / laser, monitoring the range of heat diffusion to avoid burning surrounding tissues. Combined with pressure / temperature data thresholds, visual / tactile warnings (such as a flashing red LED + vibration feedback) are triggered to reinforce awareness of proper operating procedures.

[0051] The control system 6 integrates data acquisition, force feedback synchronization, and operation evaluation functions to compare and analyze the operation procedures of interns and senior doctors and generate improvement suggestions.

[0052] like Figure 3 As shown, specifically, the control system 6 includes a central processing unit 61, a multimodal data acquisition module 62, a force feedback synchronization control module 63, a comparison and analysis module 64, an operation evaluation module 65, a data storage module 66, and a remote communication module 67.

[0053] The central processing unit 61 serves as the core scheduling hub of the system, coordinating data interaction and command transmission between multiple modules. The multimodal data acquisition module 62 acquires visual data from the image acquisition module 1, the motion trajectories of the six-degree-of-freedom manipulator 2 and the surgical training robot 4, force information from the somatosensory and tactile feedback glove 3, and the movement trajectory and cutting force of the scalpel on the simulated heart model 5. The force feedback synchronization control module 63 dynamically adjusts the drive parameters of the surgical training robot 4 based on the mechanical feedback signals from the simulated heart model 5. The comparative analysis module 64 compares the operational procedures of interns with those of senior physicians. The operation evaluation module 65 generates a scoring report and provides improvement suggestions by comparing the deviations in the operational trajectories of interns and experts. The data storage module 66 saves the operation records of senior physicians and interns, including instrument motion trajectories, force intensity curves, and operation timestamps, providing a historical data retrieval interface for the comparative analysis module and supporting the dynamic loading of training records from different stages for longitudinal performance evaluation. The remote communication module 67 transmits high-definition video streams and synchronized control commands to remote terminals in real time, supporting remote teaching and expert guidance.

[0054] This embodiment integrates multi-dimensional sensory data, including vision (3D high-definition imaging), force perception (six degrees of freedom force feedback), and touch (electronic skin), to construct an immersive training scenario that closely resembles a real surgical environment. This enables "eye-hand-force" coordination training, improving operational coordination. The six-degree-of-freedom manipulator 2 provides sub-millimeter-level positioning accuracy, and the multi-dimensional sensing electronic skin can detect contact force changes at the 0.1N level. A quantitative evaluation index system is established, including multiple core parameters such as operational force, trajectory accuracy, and time taken.

[0055] Example 2

[0056] Based on the above embodiments, such as Figure 4 As shown, this invention provides a surgical training method based on multi-dimensional data perception fusion, comprising the following steps:

[0057] S1. Senior doctors wear somatosensory and tactile feedback gloves 3 and control the surgical training robot 4 on the simulated heart model 5 through the six-degree-of-freedom manipulator 2 to perform surgical demonstrations. The system records the operation process (the motion trajectory of the six-degree-of-freedom manipulator 2, the motion trajectory of the surgical training robot 4, the movement trajectory of the scalpel on the simulated heart model 5 and the cutting force).

[0058] S2. Interns receive surgical training through the system, which records their procedures in real time.

[0059] S3. Compare and analyze the operating procedures of interns with those of senior doctors;

[0060] S4. Based on the comparison results, provide improvement suggestions to enhance the surgical level.

[0061] This embodiment utilizes transfer learning to analyze the operational data of senior physicians, generating reproducible standard procedures. Based on intelligent comparative analysis of the senior physician database, it generates operational deviation heatmaps in real time, providing targeted improvement suggestions and training programs. It supports operation playback and keyframe analysis, significantly improving the efficiency and safety of cardiac surgery training and providing an intelligent solution for medical education.

[0062] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. A surgical training system based on multi-dimensional data perception fusion, characterized in that, It includes an image acquisition module (1), a six-degree-of-freedom operating mechanism (2), a somatosensory and tactile feedback glove (3), a surgical training robot (4), a simulated heart model (5), and a control system (6); The image acquisition module (1) is connected to the control system (6) by signal, and is used to acquire 3D high-definition images of the surgical area and instrument spatial coordinates in real time; The six-degree-of-freedom operating mechanism (2) is connected to the control system (6) by signal, and is used to convert the doctor's operating instructions into control signals of the surgical training robot (4) and to provide feedback force information; The somatosensory and tactile feedback glove (3) is used to collect the doctor's hand movement data and tactile feedback signals. The somatosensory and tactile feedback glove (3) is connected to the control system (6). The surgical training robot (4) is connected to the control system (6). The surgical training robot (4) is controlled by a six-degree-of-freedom operating mechanism (2) and is used to simulate surgical operations and interact with a simulated heart model (5). The simulated heart model (5) is made of multidimensional sensing electronic skin, which is used to sense the contact position of the scalpel, the applied force and pain signals; The control system (6) integrates data acquisition, force feedback synchronization and operation evaluation functions, which are used to compare and analyze the operation process of interns and senior doctors and generate improvement suggestions.

2. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: It also includes an operation console (7), the six-degree-of-freedom operation mechanism (2) is mounted on the operation console (7), the control system (6) is integrated in the operation console (7), and a three-dimensional display device (8) is mounted on the operation console (7) via a bracket.

3. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: The image acquisition module (1) includes a binocular stereo camera (11) and an optical tracking sensor (12) for synchronously acquiring 3D images of the surgical area and the spatial coordinates of the instruments.

4. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: The somatosensory and tactile feedback glove (4) includes an electronic skin (31), a finger bending sensor (32), and a fingertip vibration motor (33). The electronic skin (31) is used to sense force control behavior, the bending sensor (32) collects hand movement data, and the vibration motor (33) converts the force information of the surgical training robot (4) into vibration frequency feedback to the user.

5. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: The six-degree-of-freedom operating mechanism (2) includes a force feedback device (21), which dynamically adjusts the operating resistance according to the sensing signal of the simulated heart model (5).

6. The surgical training system based on multi-dimensional data perception fusion according to claim 3, characterized in that: The electronic skin (31) is a flexible piezoresistive array that covers the palm and fingertip areas and is used for high-precision measurement of contact pressure distribution.

7. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: The simulated heart model (5) has built-in multimodal sensors, including a pressure sensor (51), a deformation sensor (52) and a temperature sensor (53), to simulate the mechanical and physiological characteristics of heart tissue.

8. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: The control system (6) includes: The central processing unit (61) is the core scheduling hub of the system, coordinating data interaction and instruction transmission between multiple modules; The multimodal data acquisition module (62) acquires visual data from the image acquisition module (1), motion trajectories of the six-degree-of-freedom operating mechanism (2) and the surgical training robot (4), force information from the somatosensory and tactile feedback glove (3), and the movement trajectory and cutting force of the scalpel on the simulated heart model (5). The force feedback synchronous control module (63) dynamically adjusts the driving parameters of the surgical training robot (4) based on the mechanical feedback signal of the simulated heart model (5); The comparative analysis module (64) is used to compare the operating procedures of interns with those of senior doctors; The operation assessment module (65) generates a scoring report by comparing the deviations between the operation trajectories of interns and experts, and provides improvement suggestions; The data storage module (66) saves the operation records of senior doctors and interns, including the movement trajectory of the instrument, the force intensity curve and the operation timestamp, and provides a historical data call interface for the comparative analysis module, supporting the dynamic loading of training records at different stages for longitudinal performance evaluation.

9. The surgical training system based on multi-dimensional data perception fusion according to claim 1, characterized in that: The control system (6) further includes a remote communication module (67), which is used to transmit high-definition video streams and synchronous control commands to a remote terminal in real time.

10. The surgical training method based on multi-dimensional data perception fusion according to any one of claims 1-8, characterized in that: Includes the following steps: S1. Senior doctors put on haptic and tactile feedback gloves (3) and control the surgical training robot (4) through the six-degree-of-freedom operating mechanism (2) to perform surgical demonstrations on a simulated heart model (5). The system records the operation process. S2. Interns receive surgical training through the system, which records their procedures in real time. S3. Compare and analyze the operating procedures of interns with those of senior doctors; S4. Based on the comparison results, provide improvement suggestions to enhance the surgical level.

Images