Virtual preoperative simulation device and method

Abstract

The invention relates to a virtual preoperative simulation device and method. The virtual preoperative simulation device comprises a hand tactile feedback device, an operating handle and a control device. The operating handle is provided with a laser transmitter, and the laser transmitter is used for transmitting laser to a virtual organ in a virtual operation scene, so that the laser intersects with the virtual organ to simulate an operation; the hand tactile feedback device is worn on the hand, holds the operating handle and is used for detecting the pressure applied to the operating handle by the hand; the control device is connected to the hand tactile feedback device and the laser transmitter and used for converting the pressure detected by the hand tactile feedback device into the intensity of laser and controlling the laser transmitter to correspondingly adjust the intensity of the laser to be displayed in the virtual operation scene. A doctor who operates an operation can obtain a visual feeling of the operation strength by observing the intensity of the laser, and more operation feedback is provided for the doctor, so that virtual preoperative simulation can be carried out more truly, and disconnection between the virtual preoperative simulation and a real operation process is avoided.

Description

Technical Field

[0001] The present invention relates to the field of medical virtual reality technology, and in particular to a virtual preoperative simulation device and method. Background Art

[0002] For medical surgeries with high precision and high risks, doctors can plan surgical steps in advance, identify problems that may be encountered during surgery, and develop response strategies. This helps improve the precision and safety of surgery, optimize the surgical process, improve surgical quality, and reduce the occurrence of intraoperative complications.

[0003] The prior art discloses a head-mounted AR display preoperative simulation method, comprising the following steps: S1. Displaying a medical surgery scene through an augmented reality AR device; S2. Obtaining the relative position relationship between a virtual object and a real object in the medical surgery scene; S3. When the relative position relationship between the virtual object and the real object meets a specified condition, adjusting the transparency of a specified part of the virtual object so that the transparency of the specified part is a transparency other than completely opaque; S4. Displaying the real object and the virtual object obtained after the transparency adjustment. The head-mounted AR display preoperative simulation method improves the authenticity of the preoperative simulation, and indirectly improves the accuracy of the user's operation when performing actual operations later.

[0004] It has the following technical problems: During virtual surgical operations, it can only adjust the visibility of virtual objects and cannot provide doctors with more operational feedback. Doctors find it difficult to get an intuitive sense of the force of the surgery and cannot perform virtual preoperative simulation more realistically, which can easily lead to a disconnect between virtual preoperative simulation and the actual surgical process. Summary of the invention

[0005] In view of the problems existing in the prior art, the purpose of the present invention is to provide a virtual preoperative simulation device and method, which can perform virtual preoperative simulation more realistically and avoid the disconnection between virtual preoperative simulation and the actual surgical process.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: A virtual preoperative simulation device, comprising a hand tactile feedback device, an operating handle and a control device; The operating handle is provided with a laser transmitter, which is used to emit laser to the virtual organ in the virtual surgical scene so that the laser intersects with the virtual organ to simulate the surgical operation; The hand tactile feedback device is worn on the hand and holds the operating handle to detect the pressure applied by the hand to the operating handle; The control device is respectively connected to the hand tactile feedback device and the laser emitter, and is used to convert the pressure detected by the hand tactile feedback device into the strength of the laser, and control the laser emitter to adjust the laser strength accordingly to display it in the virtual surgical scene.

[0007] Furthermore, a plurality of sub-millimeter first feedback points are provided on the inner side of the hand tactile feedback device, and the first feedback points are used to detect the first contact points between the fingers and the hand tactile feedback device, and to obtain the pressure applied by the hand to the operating handle by detecting the pressure applied by the fingers on the first contact points.

[0008] Furthermore, the first feedback point is provided with an ultrasonic imaging device and an ultrasonic positioning device. The ultrasonic imaging device is used to continuously collect composite plane wave data of hundreds of milliseconds using ultrafast ultrasonic imaging technology, quickly scan the finger area, and obtain an ultrasonic image of the finger area during a single transmission process; the ultrasonic positioning device performs beam synthesis and demodulation on the collected radio frequency data based on the ultrasonic image of the finger area to extract the signal and calibrate and locate the signal to determine the first fitting point.

[0009] Furthermore, a plurality of sub-millimeter level second feedback points are provided on the outer side of the operating handle, and the second feedback points are used to detect the second fitting points between the operating handle and the hand tactile feedback device to obtain the posture and contact points of holding the operating handle.

[0010] Furthermore, a linear electric pulse conduction device is provided on the inner side of the hand tactile feedback device, and the linear electric pulse conduction device is provided with electric pulse conduction contacts. The linear electric pulse conduction device is connected to the control device, wherein the control device sends a signal to the linear electric pulse conduction device according to the pressure applied by the hand to the operating handle and the posture and contact point of holding the operating handle. The linear electric pulse conduction device generates an electric signal in response, and the electric pulse conduction contacts transmit the electric signal to the fingers to stimulate nerve endings.

[0011] A virtual preoperative simulation method comprises the following steps: The hand tactile feedback device is worn on the hand and the operating handle is held. The laser emitter of the operating handle emits laser to the virtual organ in the virtual surgical scene, so that the laser intersects with the virtual organ to simulate the surgical operation; The hand tactile feedback device detects the pressure applied by the hand to the operating handle; The control device converts the pressure detected by the hand tactile feedback device into the strength of the laser, and controls the laser transmitter to adjust the laser strength accordingly to display it in the virtual surgical scene.

[0012] Furthermore, the method for detecting the pressure applied by the hand to the operating handle is to detect the first contact point between the finger and the hand tactile feedback device through the first feedback point of the hand tactile feedback device, and obtain the pressure applied by the hand to the operating handle by detecting the pressure applied by the finger on the first contact point.

[0013] Furthermore, the method for determining the first fitting point includes using ultrafast ultrasonic imaging technology to continuously collect composite plane wave data for hundreds of milliseconds, quickly scanning the finger area, and obtaining an ultrasonic image of the finger area during a single transmission process; based on the ultrasonic image of the finger area, beamforming and demodulating the collected radio frequency data to extract the signal and calibrate and locate the signal.

[0014] Furthermore, when the hand tactile feedback device holds the operating handle, the second contact point between the operating handle and the hand tactile feedback device is detected through the second feedback point to obtain the posture and contact point of holding the operating handle. The control device controls the first contact point to stimulate the finger according to the pressure applied to the operating handle by the hand and the posture and contact point of holding the operating handle.

[0015] Furthermore, controlling the first contact to provide stimulation feedback to the finger includes sending a signal to the linear electric pulse conduction device, the linear electric pulse conduction device generates an electric signal in response, and the electric pulse conduction contact transmits the electric signal to the finger to stimulate nerve endings.

[0016] In general, the present invention has the following advantages: The pressure applied by the hand to the operating handle is detected by the hand tactile feedback device worn on the hand and holding the operating handle. The pressure detected by the hand tactile feedback device is converted into the strength of the laser, and then the laser with adjusted strength is displayed in the virtual surgical scene, so that the doctor performing the operation can get an intuitive feeling of the surgical force by observing the strength of the laser. At the same time, setting the first feedback point on the hand tactile feedback device and the second feedback point on the operating handle can provide more accurate tactile feedback and provide doctors with more operational feedback, so that virtual preoperative simulation can be performed more realistically, avoiding the disconnection between virtual preoperative simulation and the real surgical process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the principle of an embodiment of the present invention.

[0018] Figure 2 Schematic diagram of the structure of a hand tactile feedback device according to an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of the structure of the operating handle of an embodiment of the present invention.

[0020] Figure 4 Simulate a flow chart for the surgery.

[0021] In the figure: 1-hand tactile feedback device; 11-first feedback point; 2-operating handle; 21-second feedback point; 22-laser transmitter. DETAILED DESCRIPTION

[0022] The present invention will be described in further detail below.

[0023] like Figure 1-Figure 3 As shown, a virtual preoperative simulation device includes a hand tactile feedback device 1, an operating handle 2 and a control device; The operating handle 2 is provided with a laser emitter 22, and the laser emitter 22 is used to emit laser light to the virtual organ in the virtual surgical scene, so that the laser light intersects with the virtual organ to simulate the surgical operation; The hand tactile feedback device 1 is worn on the hand and holds the operating handle 2, and is used to detect the pressure applied by the hand to the operating handle 2; The control device is connected to the hand tactile feedback device 1 and the laser emitter 22 respectively, and is used to convert the pressure detected by the hand tactile feedback device 1 into the strength of the laser, and control the laser emitter 22 to adjust the laser strength accordingly and display it in the virtual surgery scene. The stronger the laser, the greater the energy of the surgical operation, and the greater the strength and depth of the surgical operation on the virtual organ, and vice versa.

[0024] The hand tactile feedback device 1 is a structure similar to a finger sleeve, and a plurality of sub-millimeter first feedback points 11 are arranged inside the hand tactile feedback device 1. The first feedback point 11 is used to detect the first contact point between the finger and the hand tactile feedback device 1, and to obtain the pressure applied by the hand to the operating handle 2 by detecting the pressure applied by the finger on the first contact point. By arranging a plurality of sub-millimeter first feedback points 11 inside the hand tactile feedback device 1, the first contact point between the finger and the hand tactile feedback device 1 can be accurately captured, and the tactile reception range is narrowed to a partial area of ​​the finger, which reduces energy consumption and improves the accuracy of simulated tactile.

[0025] The first feedback point 11 is provided with an ultrasonic imaging device and an ultrasonic positioning device. The ultrasonic imaging device is used to continuously collect composite plane wave data of hundreds of milliseconds using ultrafast ultrasonic imaging technology, quickly scan the finger area, and obtain an ultrasonic image of the finger area in a single transmission process; the ultrasonic positioning device performs beam synthesis and demodulation on the collected radio frequency data based on the ultrasonic image of the finger area to extract the signal and calibrate and locate the signal to determine the first fitting point. Through ultrasonic imaging and ultrasonic positioning, the first fitting point between the finger and the hand tactile feedback device 1 can be accurately obtained, providing more accurate feedback for the doctor's delicate surgical operations.

[0026] The outer side of the operating handle 2 is provided with a plurality of sub-millimeter-level second feedback points 21, which are used to detect the second fitting point between the operating handle 2 and the hand tactile feedback device 1, so as to obtain the posture and contact point of the doctor when holding the operating handle 2. The first feedback point 11 and the second feedback point 21 are both connected to the control device, and the data obtained by the second feedback point 21 will be transmitted to the control device together with the pressure data detected by the first feedback point 11, and the finger will be stimulated and fed back by sending a signal to the first feedback point 11 of the hand tactile feedback device 1.

[0027] Different doctors have different finger contacts and grip contacts on the operating handle 2. The present invention sets contact feedback on the inner side of the hand tactile feedback device 1 and the outer side of the operating handle 2 respectively. The double contact feedback can increase the accuracy of the feedback, obtain more accurate positioning, and better simulate the doctor's delicate surgical operations.

[0028] In this embodiment, a linear electric pulse conduction device is provided on the inner side of the hand tactile feedback device 1, and the linear electric pulse conduction device is spirally arranged along the hand tactile feedback device 1. The linear electric pulse conduction device is provided with sub-millimeter electric pulse conduction contacts, and the linear electric pulse conduction device is connected to the control device, wherein when the hand tactile feedback device 1 holds the operating handle 2, the control device sends a signal to the linear electric pulse conduction device according to the detection data of the first feedback point 11 and the second feedback point 21, and the linear electric pulse conduction device generates an electric signal accordingly, and the electric pulse conduction contact transmits the electric signal to the finger to stimulate the nerve endings, thereby simulating the tactile feeling. The greater the pressure applied by the hand to the operating handle 2, the greater the strength of the electric signal, which is mainly reflected in the resistance and tension of different human tissues in the surgical operation. The doctor can have different tactile experiences, which enhances the realism of the surgical simulation.

[0029] like Figure 4 As shown, a virtual preoperative simulation method comprises the following steps: The doctor wears the hand tactile feedback device 1 on his fingers and holds the operating handle 2, and wears the head-mounted display on his head, wherein the head-mounted display adopts the head display of the existing technology; Different surgical instruments can be switched according to the needs of the operation, and the way to switch surgical instruments can be achieved through different gripping postures. For example, if the doctor's thumb, index finger and middle finger are pinched together three times in a row, the handheld surgical instrument will be switched, and different surgical instruments can be seen in the head-mounted display.

[0030] Start a virtual surgery scene (an existing development engine such as Unity3D may be used), and use the laser emitter 22 of the operating handle 2 to emit laser light to the virtual organ in the virtual surgery scene, so that the laser light intersects with the virtual organ to simulate the surgery operation; The hand tactile feedback device 1 detects the pressure applied by the hand to the operating handle 2; The control device converts the pressure detected by the hand tactile feedback device 1 into the intensity of the laser, and controls the laser emitter 22 to adjust the laser intensity accordingly and display it in the virtual surgical scene.

[0031] Determine the dynamic shape of the virtual organ three-dimensional model after the simulated cutting operation and present it in the virtual surgical scene.

[0032] The entry point of the simulated surgical operation is determined by the intersection between the laser emitted by the operating handle 2 and the virtual organ, and the simulated surgical operation path is obtained by the extension direction of the laser. The real-time simulation of the present invention is more accurate and clear, which is convenient for doctors to obtain feedback information during the surgical operation.

[0033] The pressure applied by the hand to the operating handle 2 is detected by detecting the first contact point between the finger and the hand tactile feedback device 1 through the first feedback point 11 of the hand tactile feedback device 1, and obtaining the pressure applied by the hand to the operating handle 2 by detecting the pressure applied by the finger on the first contact point. By detecting the first contact point between the finger and the hand tactile feedback device 1, the detection site can be narrowed down to a partial area of ​​the finger, or even refined from surface detection to point detection, so as to more accurately feedback the doctor's delicate operation.

[0034] Specifically, the method for determining the first fitting point includes using ultrafast ultrasonic imaging technology to continuously collect composite plane wave data for hundreds of milliseconds, quickly scanning the finger area, and obtaining an ultrasonic image of the finger area during a single transmission process; based on the ultrasonic image of the finger area, the collected radio frequency data is beamformed and demodulated to extract the signal and calibrate and locate the signal to determine the first fitting point.

[0035] After obtaining the pressure applied by the hand to the operating handle 2, the control device sends a signal to the linear electric pulse conduction device, the linear electric pulse conduction device generates an electric signal in response, and the electric pulse conduction contact transmits the electric signal to the finger to stimulate the nerve endings.

[0036] The present invention utilizes a combination of sound waves and lasers to transmit tactile signals to increase the timeliness of feedback. The transmission of sound wave signals is more accurate, and the transmission of laser signals is faster. These two technologies are respectively applied to different signal transmission stages in the surgical simulation process to give full play to their greatest advantages. Sound waves are used to locate the contact points of the hand, and the contact positioning is more accurate. The laser is used to emit laser rays that interact with the virtual organs in the virtual scene to obtain specific position information of the surgical operation, capture relative position information more quickly, reduce delays, and improve the accuracy and timeliness of tactile feedback. It is low in cost and high in efficiency, meets the requirements of the modern medical field, enhances the doctor's sense of reality and operating experience in virtual preoperative simulation, and makes it easier to apply the preoperative simulation operation process to real surgery.

[0037] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A virtual preoperative simulation device, characterized in that: It includes a hand tactile feedback device, an operating handle and a control device; The operating handle is provided with a laser transmitter, which is used to emit laser to the virtual organ in the virtual surgical scene so that the laser intersects with the virtual organ to simulate the surgical operation; The hand tactile feedback device is worn on the hand and holds the operating handle to detect the pressure applied by the hand to the operating handle; The control device is respectively connected to the hand tactile feedback device and the laser emitter, and is used to convert the pressure detected by the hand tactile feedback device into the strength of the laser, and control the laser emitter to adjust the laser strength accordingly to display it in the virtual surgical scene.

2. A virtual preoperative simulation device according to claim 1, characterized in that: There are multiple sub-millimeter first feedback points on the inner side of the hand tactile feedback device. The first feedback point is used to detect the first contact point between the finger and the hand tactile feedback device, and obtain the pressure applied by the hand to the operating handle by detecting the pressure applied by the finger on the first contact point.

3. A virtual preoperative simulation device according to claim 2, characterized in that: The first feedback point is provided with an ultrasonic imaging device and an ultrasonic positioning device. The ultrasonic imaging device is used to continuously collect composite plane wave data of hundreds of milliseconds using ultrafast ultrasonic imaging technology, quickly scan the finger area, and obtain an ultrasonic image of the finger area during a single transmission process; the ultrasonic positioning device performs beam synthesis and demodulation on the collected radio frequency data based on the ultrasonic image of the finger area to extract the signal and calibrate and locate the signal to determine the first fitting point.

4. A virtual preoperative simulation device according to claim 2, characterized in that: The outside of the operating handle is provided with multiple sub-millimeter level second feedback points, which are used to detect the second fitting points between the operating handle and the hand tactile feedback device to obtain the posture and contact points of holding the operating handle.

5. A virtual preoperative simulation device according to claim 4, characterized in that: A linear electric pulse conduction device is provided on the inner side of the hand tactile feedback device. The linear electric pulse conduction device is provided with electric pulse conduction contacts. The linear electric pulse conduction device is connected to the control device, wherein the control device sends a signal to the linear electric pulse conduction device according to the pressure applied by the hand to the operating handle and the posture and contact point of holding the operating handle. The linear electric pulse conduction device generates an electric signal accordingly, and the electric pulse conduction contacts transmit the electric signal to the fingers to stimulate the nerve endings.

6. A virtual preoperative simulation method, characterized in that: The virtual preoperative simulation device according to any one of claims 1 to 5 comprises the following steps: The hand tactile feedback device is worn on the hand and the operating handle is held. The laser emitter of the operating handle emits laser to the virtual organ in the virtual surgical scene, so that the laser intersects with the virtual organ to simulate the surgical operation; The hand tactile feedback device detects the pressure applied by the hand to the operating handle; The control device converts the pressure detected by the hand tactile feedback device into the strength of the laser, and controls the laser transmitter to adjust the laser strength accordingly to display it in the virtual surgical scene.

7. The virtual preoperative simulation method according to claim 6, characterized in that: The method for detecting the pressure applied by the hand to the operating handle is to detect the first contact point between the finger and the hand tactile feedback device through the first feedback point of the hand tactile feedback device, and obtain the pressure applied by the hand to the operating handle by detecting the pressure applied by the finger on the first contact point.

8. The virtual preoperative simulation method according to claim 7, characterized in that: The method for determining the first fitting point includes using ultrafast ultrasonic imaging technology to continuously collect composite plane wave data for hundreds of milliseconds, quickly scanning the finger area, and obtaining an ultrasonic image of the finger area during a single transmission process; based on the ultrasonic image of the finger area, beamforming and demodulating the collected radio frequency data to extract the signal and calibrate and locate the signal.

9. The virtual preoperative simulation method according to claim 8, characterized in that: When the hand tactile feedback device holds the operating handle, the second contact point between the operating handle and the hand tactile feedback device is detected through the second feedback point to obtain the posture and contact point of holding the operating handle. The control device controls the first contact point to stimulate and feedback the finger according to the pressure applied to the operating handle by the hand and the posture and contact point of holding the operating handle.

10. The virtual preoperative simulation method according to claim 9, characterized in that: Controlling the first contact to provide stimulation feedback to the finger includes sending a signal to the linear electric pulse conduction device, the linear electric pulse conduction device generates an electric signal in response, and the electric pulse conduction contact transmits the electric signal to the finger to stimulate the nerve endings.

Images