Mixed reality-based renal biopsy procedure navigation method and apparatus

Abstract

The application discloses a kind of renal puncture surgery navigation method and device based on mixed reality.Method comprising the following steps: creating the three-dimensional virtual abdominal model of patient abdomen and the three-dimensional virtual mechanical arm model of puncture mechanical arm;Printing three-dimensional puncture phantom;Build the virtual scene of three-dimensional virtual abdominal model and three-dimensional virtual mechanical arm model;Create the first position coordinate code of three-dimensional puncture phantom in virtual scene and the second position coordinate code of puncture mechanical arm in virtual scene;Mark the first position coordinate code coordinate in three-dimensional puncture phantom and the second position coordinate code in puncture mechanical arm;Scan first position coordinate code and second position coordinate code, three-dimensional virtual abdominal model and three-dimensional puncture phantom virtual superposition, three-dimensional virtual mechanical arm model and puncture mechanical arm virtual superposition, control the movement trajectory of puncture mechanical arm in virtual superposition virtual scene by gesture interaction, puncture operation is carried out to three-dimensional puncture phantom, so can avoid hand-eye problem.

Description

Technical Field

[0001] This invention relates to a medical device, and more particularly to a method and apparatus for navigating kidney biopsy based on mixed reality. Background Technology

[0002] Renal biopsy allows for the examination of morphological changes in kidney tissue, making it an important method for renal pathological examination. Currently, the most common guidance methods used in clinical practice for renal biopsy are CT guidance and ultrasound guidance.

[0003] In the process of developing this invention, the applicant discovered that the guidance methods using CT guidance and ultrasound guidance have at least the following technical problems:

[0004] 1. CT-guided methods require CT scans to determine the relative position of the puncture needle and the lesion, therefore, CT-guided methods will expose patients to X-ray radiation;

[0005] 2. Although ultrasound-guided imaging can achieve real-time imaging, its real-time display effect on the puncture needle during clinical surgery is not excellent due to the patient's respiratory movements, and it is easy to find the puncture needle.

[0006] 3. Both of the above guidance methods require the doctor to constantly monitor the screen during surgery, which makes it impossible for the doctor to fully focus their vision on the patient. This can easily lead to hand-eye coordination problems during surgery. Summary of the Invention

[0007] This invention provides a method and device for navigating kidney biopsy surgery based on mixed reality, which solves the problem that doctors need to constantly monitor the screen during kidney biopsy surgery.

[0008] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0009] Firstly, a virtual reality-based navigation method for renal biopsy is provided, comprising the following steps: creating a three-dimensional virtual abdominal model of the patient's abdomen and a three-dimensional virtual robotic arm model of the biopsy robot; printing a three-dimensional biopsy phantom based on the three-dimensional virtual abdominal model; constructing a virtual scene of the three-dimensional virtual abdominal model and the three-dimensional virtual robotic arm model; creating a first position coordinate code for the three-dimensional biopsy phantom in the virtual scene and a second position coordinate code for the biopsy robot in the virtual scene; marking the first position coordinate code on the three-dimensional biopsy phantom and the second position coordinate code on the biopsy robot; scanning the first position coordinate code and the second position coordinate code, superimposing the three-dimensional virtual abdominal model and the three-dimensional biopsy phantom, and superimposing the three-dimensional virtual robotic arm model and the biopsy robot; and controlling the movement trajectory of the biopsy robot in the superimposed virtual scene through gesture interaction to perform the biopsy on the three-dimensional biopsy phantom.

[0010] Secondly, a virtual reality-based renal biopsy navigation device is provided, comprising a three-dimensional puncture phantom, a puncture robotic arm, and a head-mounted display. The three-dimensional puncture phantom is printed based on a three-dimensional virtual abdominal model of the patient, and a first position coordinate code is set on the phantom. The puncture robotic arm is set with a second position coordinate code. The head-mounted display is communicatively connected to the puncture robotic arm, and the head-mounted display has a virtual scene constructed based on the three-dimensional virtual abdominal model and the three-dimensional virtual robotic arm model. When the head-mounted display scans the first and second position coordinate codes, the three-dimensional virtual abdominal model and the three-dimensional puncture phantom are superimposed, and the three-dimensional virtual robotic arm model and the puncture robotic arm are superimposed. Through gesture interaction, the movement trajectory of the puncture robotic arm is controlled within the superimposed virtual scene to perform the puncture surgery on the three-dimensional puncture phantom.

[0011] In this embodiment of the invention, a virtual scene is constructed using virtual reality technology, creating a three-dimensional virtual abdominal model of the patient and a three-dimensional virtual robotic arm model of the puncture robot. The three-dimensional virtual abdominal model is then printed as a three-dimensional puncture phantom. After the three-dimensional virtual abdominal model and the three-dimensional puncture phantom are superimposed, and the three-dimensional virtual robotic arm model and the puncture robot are superimposed, the movement trajectory of the puncture robot is controlled via gesture interaction within the superimposed virtual scene to perform the puncture surgery on the three-dimensional puncture phantom. This method of performing kidney puncture surgery eliminates the need for the surgeon to monitor the screen during the procedure, thus avoiding hand-eye coordination issues and the problems of radiation exposure and poor real-time display of the puncture needle associated with CT-guided methods. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0013] Figure 1 This is a flowchart of the steps of the virtual reality-based renal biopsy surgical navigation method according to the first embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of a three-dimensional puncture phantom according to the first embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of the puncture robotic arm according to the first embodiment of the present invention;

[0016] Figure 4 This is a schematic diagram of a user wearing a head-mounted display according to the first embodiment of the present invention;

[0017] Figure 5This is a system diagram of the virtual reality-based renal biopsy surgical navigation device according to the first and second embodiments of the present invention;

[0018] Figure 6 This is a schematic diagram of the use of the virtual reality-based kidney biopsy navigation device according to the second embodiment of the present invention.

[0019] The following explanation is based on the accompanying diagram:

[0020] 100: Virtual Reality-Based Kidney Biopsy Navigation Device; 1: Three-Dimensional Puncture Phantom; 11: Abdominal Main Phantom; 12: Left Kidney Phantom; 13: Right Kidney Phantom; 101: First Position Coordinate Code; 102: Second Position Coordinate Code; 2: Puncture Robotic Arm; 21: Robotic Arm Main Body; 22: Puncture Components; 211: Base; 221: Puncture Needle; 222: Needle Holder; 3: Head-Mounted Display. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Furthermore, the terms "first," "second," etc., used herein are not intended to specifically refer to order or sequence, nor are they intended to limit the invention; they are merely used to distinguish components or operations described using the same technical terms.

[0023] Please see Figures 1 to 3 , Figure 1 This is a flowchart of the steps of the virtual reality-based renal biopsy surgical navigation method according to the first embodiment of the present invention. Figure 2 This is a schematic diagram of the three-dimensional puncture phantom according to the first embodiment of the present invention. Figure 3This is a schematic diagram of the puncture robotic arm according to the first embodiment of the present invention. As shown in the figure, the virtual reality-based renal puncture surgery navigation method S in this embodiment is mainly applicable to controlling the puncture robotic arm 2 to perform renal puncture surgery on the three-dimensional puncture phantom 1. The virtual reality-based renal puncture surgery navigation method S includes the following steps S1 to S6. When the virtual reality-based renal puncture surgery navigation method S is used for renal puncture surgery, in step S1, a three-dimensional virtual abdominal model of the patient's abdomen and a three-dimensional virtual robotic arm model of the puncture robotic arm 2 are first created; then, in step S2, the three-dimensional puncture phantom 1 is printed based on the three-dimensional virtual abdominal model; then, in step S3, a virtual scene of the three-dimensional virtual abdominal model and the three-dimensional virtual robotic arm model is built; then, in step S4, a first position coordinate code 101 of the three-dimensional puncture phantom 1 and a second position coordinate code 102 of the puncture robotic arm 2 in the virtual scene are created; then, in step S5, the coordinates of the first position coordinate code 101 on the three-dimensional puncture phantom 1 and the second position coordinate code 102 on the puncture robotic arm 2 are marked.

[0024] As described above, in step S6, the first position coordinate code 101 and the second position coordinate code 102 are scanned. The three-dimensional virtual abdominal model and the three-dimensional puncture phantom 1 are superimposed, and the three-dimensional virtual robotic arm model and the puncture robotic arm 2 are superimposed. Through gesture interaction, the movement trajectory of the puncture robotic arm 2 is controlled in the superimposed virtual scene to perform the puncture surgery on the three-dimensional puncture phantom 1. Performing kidney puncture surgery in this way allows the doctor to avoid monitoring the screen during the procedure, thus avoiding hand-eye coordination problems, radiation exposure to the patient due to CT guidance, and poor real-time display of the puncture needle 221.

[0025] The following is a detailed explanation. (See also: [link to relevant documentation]) Figure 1 As shown, in step S1, when creating a three-dimensional virtual abdominal model of the patient's abdomen, CT image data of the patient's abdomen is imported into Mimics software. The three-dimensional virtual model is reconstructed using Mimics software, and the tissues of the patient's abdomen are delineated. These tissues include, but are not limited to, bones, kidneys, and blood vessels, ultimately resulting in a three-dimensional virtual abdominal model in STL format. Next, in step S2, based on the STL format three-dimensional virtual abdominal model, a three-dimensional puncture phantom 1 is obtained using silicone material through 3D printing. (See attached image.) Figure 2 As shown, in this embodiment, the three-dimensional puncture phantom 1 includes an abdominal main phantom 11 and a left kidney phantom 12 and a right kidney phantom 13 located inside the abdominal main phantom 11.

[0026] See also Figures 1 to 3In step S3, the 3D virtual abdominal model and the 3D virtual robotic arm model are imported into Unity software to create a boot screen, patient case information, an interactive interface, and a gesture interaction program to control the movement of the puncture robotic arm 2. Specifically, the 3D virtual abdominal model and the 3D virtual robotic arm model are imported into 3ds Max software, and the format is converted to FBX format. Then, the FBX format 3D virtual abdominal model and 3D virtual robotic arm model are imported into Unity software. C# scripts are written in Unity software to create the boot screen, patient case information, and interactive interface. At the same time, interactive functions to control the movement of the virtual model of the puncture robotic arm 2 are written in conjunction with the 3D puncture phantom 1 and the 3D virtual robotic arm model. This completes the development of the renal puncture surgery navigation software. Then, the puncture robotic arm 2 is connected to Unity software via socket communication, thus completing the construction of the virtual scene.

[0027] In step S4, the first position coordinates of the 3D puncture phantom 1 in the virtual scene are obtained in Unity software, converted into a first position coordinate code 101, and printed. Similarly, the second position coordinates of the puncture robotic arm 2 in the virtual scene are obtained in Unity software, converted into a second position coordinate code 102, and printed. In this embodiment, the first position coordinate code 101 and the second position coordinate code 102 are corresponding QR codes. Next, in step S5, the printed first position coordinate code 101 is affixed to the corresponding position of the 3D puncture phantom 1, and the printed second position coordinate code 102 is affixed to the corresponding position of the puncture robotic arm 2.

[0028] Please refer to the following: Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of a user wearing a head-mounted display according to the first embodiment of the present invention. Figure 5 This is a system diagram of the virtual reality-based renal biopsy surgical navigation device according to the first embodiment of the present invention. As shown in the figure, in step S6, the application of the virtual scene built in step 5 is packaged and deployed to the head-mounted display 3 using Visual Studio software. The user enters the boot interface by wearing the head-mounted display 3, and enters the renal biopsy surgical navigation software through gesture interaction. After seeing the patient's medical record information, the user enters the virtual scene according to the information on the interactive interface and voice prompts. The user scans the first position coordinate code 101 of the three-dimensional puncture phantom 1 and the second position coordinate code 102 on the puncture robotic arm 2 through the head-mounted display 3. The user can see the effect of the three-dimensional virtual abdominal model superimposed on the three-dimensional puncture phantom 1 and the three-dimensional virtual robotic arm model superimposed on the puncture robotic arm 2 in the head-mounted display 3.

[0029] As described above, the 3D virtual robotic arm model is currently in its initial position. The user selects the 3D virtual robotic arm model in the virtual scene interface and clicks on any location within the workspace of the puncture robotic arm 2. The user can then see the needle tip at the end of the 3D virtual robotic arm model move to the designated position, and the puncture robotic arm 2 simultaneously moves to the corresponding position in the real scene. This allows the user to control the movement trajectory of the puncture robotic arm 2 within a virtual scene that overlays reality through gesture interaction, performing a puncture on the 3D puncture phantom 1. When the user manipulates the 3D virtual robotic arm model to move the puncture needle 221 of the puncture robotic arm 2 to the insertion point on the surface of the 3D puncture phantom 1, the user can continue to manipulate the 3D virtual robotic arm model to control the puncture robotic arm 2 to perform punctures within the 3D puncture phantom 1 according to the planned path. When the puncture needle 221 reaches the kidney lesion, the user can manipulate the 3D virtual robotic arm model to control the puncture robotic arm 2 to move out of the body and reset / return to its initial position, ending the puncture procedure.

[0030] Please see Figure 5 and Figure 6 The figures show a system diagram and a usage diagram of the virtual reality-based renal biopsy navigation device according to the second embodiment of the present invention. As shown in the figures, the virtual reality-based renal biopsy navigation device 100 of this embodiment is mainly applicable to renal biopsy surgery. The virtual reality-based renal biopsy navigation device 100 of this embodiment includes a three-dimensional biopsy phantom 1, a biopsy robotic arm 2, and a head-mounted display 3. The three-dimensional biopsy phantom 1 is used to simulate the abdominal structure of the human body. It is printed based on a three-dimensional virtual abdominal model of the patient's abdomen. Specifically, the CT image data of the patient's abdomen can be imported into Mimics software, and a three-dimensional virtual abdominal model in STL format can be obtained through three-dimensional reconstruction. Based on the three-dimensional virtual abdominal model, the three-dimensional biopsy phantom 1 is printed using silicone material. In this embodiment, the three-dimensional biopsy phantom 1 includes an abdominal main body phantom 11 and a left kidney phantom 12 and a right kidney phantom 13 located inside the abdominal main body phantom 11. The three-dimensional biopsy phantom 1 is provided with a first position coordinate code 101, such as... Figure 6 As shown, the first position coordinate code 101 is attached to the surface of the abdominal main body model 11. The first position coordinate code 101 is the position coordinate of the three-dimensional puncture model 1 in the virtual scene. In this embodiment, the first position coordinate code 101 is a QR code.

[0031] The puncture robotic arm 2 is used to perform puncture surgery on the left kidney phantom 12 and right kidney phantom 13 of the three-dimensional puncture phantom 1. In this embodiment, the puncture robotic arm 2 includes a robotic arm body 21, a puncture assembly 22, and a control module (not shown in the figure). The puncture assembly 22 is connected to the robotic arm body 21. The puncture assembly 22 includes a puncture needle 221 and a needle holder 222 for fixing the puncture needle 221. The needle holder 222 is disposed on the robotic arm body 21. The robotic arm body 21 moves the puncture needle 221 through the needle holder 222 and performs the puncture action by moving the needle holder 222 downward. The control module is electrically connected to the robotic arm body 21. The control module controls the puncture needle 221 to perform the puncture surgery action by controlling the robotic arm body 21. In this embodiment, the robotic arm body 21 can use a three-axis robotic arm, but the present invention is not limited to this. The puncture robotic arm 2 is provided with a second position coordinate code 102, such as... Figure 6 As shown, the second position coordinate code 102 is attached to the base 211 of the robotic arm body 21. The second position coordinate code 102 is the position coordinate of the puncture robotic arm 2 in the virtual scene. In this embodiment, the second position coordinate code 102 is a QR code.

[0032] The head-mounted display 3 is communicatively connected to the puncture robotic arm 2. In this embodiment, the head-mounted display 3 establishes a communication connection with the control module of the puncture robotic arm 2. Specifically, the head-mounted display 3 can establish a communication connection with the puncture robotic arm 2 via socket communication. The head-mounted display 3 has a virtual scene, which is built based on a 3D virtual abdominal model and a 3D virtual robotic arm model of the puncture robotic arm 2. In this embodiment, the virtual scene is built in Unity software and packaged and deployed within the head-mounted display 3. The virtual scene includes a 3D virtual abdominal model, a 3D virtual robotic arm model, a boot screen, patient medical information, an interactive interface, and a gesture interaction program for controlling the movement of the puncture robotic arm 2. The head-mounted display 3 can use a HoloLens 2 headset.

[0033] See also Figure 6 As shown, in this embodiment, the virtual reality-based renal biopsy navigation device 100, when used for renal biopsy, involves the doctor wearing a head-mounted display 3. The doctor scans the first position coordinate code 101 and the second position coordinate code 102 using the head-mounted display 3, causing the three-dimensional virtual abdominal model to be superimposed on the three-dimensional puncture phantom 1, and the three-dimensional virtual robotic arm model to be superimposed on the puncture machine. Then, through gesture interaction, the doctor controls the movement trajectory of the puncture robotic arm 2 within the superimposed virtual scene to perform the puncture surgery on the three-dimensional puncture phantom 1. Using this navigation device for renal biopsy eliminates the need for the doctor to monitor the screen during the procedure, thus avoiding hand-eye coordination issues, radiation exposure to the patient due to CT guidance, and poor real-time display of the puncture needle 221.

[0034] In summary, this invention provides a method and device for kidney biopsy navigation based on mixed reality. This invention utilizes virtual reality technology to construct a virtual scene containing a 3D virtual abdominal model of the patient and a 3D virtual robotic arm model of the puncture robot. The 3D virtual abdominal model is then printed as a 3D puncture phantom. After the 3D virtual abdominal model and the 3D puncture phantom are superimposed in a virtual-real context, and the 3D virtual robotic arm model and the puncture robot model are superimposed in a virtual-real context, the movement trajectory of the puncture robot is controlled via gesture interaction within the superimposed virtual scene to perform the puncture surgery on the 3D puncture phantom. This method of performing kidney biopsy eliminates the need for the surgeon to monitor a display screen during the procedure, thus avoiding hand-eye coordination issues and the problems associated with CT-guided procedures, such as radiation exposure and poor real-time needle display.

[0035] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0036] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A virtual reality-based method for navigating kidney biopsy surgery, characterized in that, Includes the following steps: The process involves creating a three-dimensional virtual abdominal model of the patient's abdomen and a three-dimensional virtual robotic arm model of the puncture robotic arm. The steps for creating the three-dimensional virtual abdominal model of the patient's abdomen include: importing CT image data of the patient's abdomen into Mimics software; reconstructing the three-dimensional virtual model and outlining the tissues of the patient's abdomen to obtain the three-dimensional virtual abdominal model. Based on the aforementioned three-dimensional virtual abdominal model, a three-dimensional puncture phantom was printed. The steps of constructing the virtual scene of the three-dimensional virtual abdomen model and the three-dimensional virtual robotic arm model include: importing the three-dimensional virtual abdomen model and the three-dimensional virtual robotic arm model into Unity software; creating a boot screen, patient case information, an interactive interface, and a gesture interaction program to control the movement of the puncture robotic arm in the Unity software; The steps of creating the first position coordinate code of the three-dimensional puncture phantom in the virtual scene and the second position coordinate code of the puncture robotic arm in the virtual scene include: obtaining the first position coordinates of the three-dimensional puncture phantom in the virtual scene, converting the first position coordinates into a first position coordinate code, and printing it; and obtaining the second position coordinates of the puncture robotic arm in the virtual scene, converting the second position coordinates into a second position coordinate code, and printing it. Mark the first position coordinate code on the three-dimensional puncture phantom and the second position coordinate code on the puncture robot arm; Scan the first position coordinate code and the second position coordinate code, and the three-dimensional virtual abdominal model and the three-dimensional puncture phantom are superimposed, and the three-dimensional virtual robotic arm model and the puncture robotic arm are superimposed. Through gesture interaction, the movement trajectory of the puncture robotic arm is controlled in the virtual scene superimposed with the real and virtual to perform puncture surgery on the three-dimensional puncture phantom.

2. A renal biopsy navigation device applied to the virtual reality-based renal biopsy navigation method as described in claim 1, characterized in that, include: The three-dimensional puncture phantom is printed based on a three-dimensional virtual abdominal model of the patient's abdomen, and the three-dimensional puncture phantom is provided with a first position coordinate code. The puncture robotic arm is equipped with a second position coordinate code; A head-mounted display is communicatively connected to the puncture robotic arm. The head-mounted display has a virtual scene, which is constructed based on the three-dimensional virtual abdominal model and the three-dimensional virtual robotic arm model of the puncture robotic arm. When the head-mounted display scans the first position coordinate code and the second position coordinate code, the three-dimensional virtual abdominal model and the three-dimensional puncture phantom are superimposed, and the three-dimensional virtual robotic arm model and the puncture machine are superimposed. Through gesture interaction, the movement trajectory of the puncture robotic arm is controlled in the virtual scene superimposed with the real and virtual to perform puncture surgery on the three-dimensional puncture phantom.

3. The virtual reality-based renal biopsy navigation device according to claim 2, characterized in that, The three-dimensional puncture phantom was printed using silicone material.

4. The virtual reality-based renal biopsy navigation device according to claim 2, characterized in that, The puncture robotic arm includes: Robotic arm body; A puncture assembly is connected to the main body of the robotic arm. The puncture assembly includes a puncture needle and a needle holder for fixing the puncture needle. The needle holder is disposed on the main body of the robotic arm. The control module is electrically connected to the main body of the robotic arm, and the head-mounted display is communicatively connected to the control module.

5. The virtual reality-based renal biopsy navigation device according to claim 4, characterized in that, The head-mounted display establishes a communication connection with the control module via socket communication.

6. The virtual reality-based renal biopsy navigation device according to claim 2, characterized in that, The virtual scene was built using Unity software and then packaged and deployed within the head-mounted display.

7. The virtual reality-based renal biopsy navigation device according to claim 6, characterized in that, The virtual scene includes the three-dimensional virtual abdominal model, the three-dimensional virtual robotic arm model, as well as the startup interface, patient case information, an interactive interface, and a gesture interaction program to control the movement of the puncture robotic arm.

Images