Ultrasonic subcutaneous puncture guiding system and method based on mixed reality

Abstract

The invention relates to an ultrasonic subcutaneous puncture guiding system and method based on mixed reality. The system comprises a medical image scanning device, an ultrasonic device, a virtual reality processing module, a virtual reality display superposition module and a mixed reality device. Medical image scanning equipment scans a two-dimensional image of the puncture part of the patient and converts the two-dimensional image into a three-dimensional image; the ultrasonic equipment collects a real-time ultrasonic image; the mixed reality equipment collects and transmits a three-dimensional human body image; the virtual reality processing module carries out 3D conversion processing on the real-time ultrasonic image to obtain a virtual image, the virtual reality processing module carries out edge recognition on the three-dimensional image, the virtual image and the three-dimensional human body image, and the relative positions of the three-dimensional image, the virtual image and the three-dimensional human body image are determined and fused; and the virtual reality display superposition module superposes the fused image on the three-dimensional human body image of the patient in the view of the operator. According to the invention, the complexity of equipment and the operation table preparation time are reduced, the operation is simple, and the learning difficulty is low.

Description

Technical Field

[0001] This invention relates to the field of puncture needle guidance technology, and in particular to an ultrasonic subcutaneous puncture guidance system and method based on mixed reality. Background Technology

[0002] In traditional clinical practice, percutaneous puncture mainly relies on real-time ultrasound guidance. However, in actual operation, the puncture needle is very small, resulting in low clarity of the ultrasound image, making it difficult to distinguish the needle tip and the needle shaft. The small ultrasound scanning range leads to a narrow field of view, making it impossible to accurately distinguish the local anatomical location in the human body based solely on the ultrasound image. Furthermore, the narrow ultrasound scanning range and the small puncture needle make it impossible to determine the spatial relationship between the ultrasound image and the puncture needle. The ultrasound image and the patient are not in the same field of view, requiring the doctor to simultaneously view the ultrasound diagnostic instrument screen and locate the puncture site and needle on the patient during the puncture procedure. This makes it impossible for the doctor to focus on only one field of view, further increasing the difficulty of the puncture procedure. Moreover, the need to move the ultrasound diagnostic instrument back and forth during the procedure as needed brings many inconveniences to the operation.

[0003] Patent application number 201811514968.0 discloses a "puncture positioning technique assisted by marker-based mixed reality technology." This technique, to achieve lung nodule puncture positioning using mixed reality technology, requires placing at least three markers made of bone-like material on the chest, with the three markers forming a plane, and marking the placement locations with color. However, this method cannot eliminate displacement differences caused by dynamic changes such as breathing and heartbeat, and requires additional spatial positioning technology and equipment, increasing costs, making the instrument more complex, and increasing preparation time.

[0004] Patent application number 202211486906.X discloses "A method for displaying ultrasound images based on mixed reality technology." This technical solution also uses feature markers with obvious external characteristics that are not similar to markers in nature or operating rooms. The feature markers are scanned into ".od" files, which are then converted into Unity package files supported by Unity using Vuforia. An MR environment is built around these markers and packaged into an APPX file for mixed reality glasses to recognize. However, this process is complex and requires professional personnel, making the procedure difficult to learn and hindering the widespread application of the equipment. Summary of the Invention

[0005] To reduce the difficulty, complexity, and learning curve of percutaneous puncture surgery, as well as to decrease surgical costs and preparation time, this invention proposes a mixed reality-based ultrasound-guided subcutaneous puncture system and method.

[0006] In a first aspect, this application provides a mixed reality-based ultrasound subcutaneous puncture guidance system, including a medical imaging scanning device, an ultrasound device, a virtual reality processing module, a virtual reality display overlay module, and a mixed reality device;

[0007] The medical imaging scanning device is used to scan two-dimensional images of the patient's puncture site;

[0008] The medical imaging scanning device performs three-dimensional reconstruction on the two-dimensional image to generate a three-dimensional image;

[0009] The medical imaging scanning device sends the three-dimensional image to the virtual reality processing module;

[0010] The ultrasound device is used to acquire real-time ultrasound images of the patient's puncture site and transmit them to the virtual reality processing module.

[0011] The mixed reality device is used to acquire three-dimensional human images of the patient in the surgeon's field of vision and transmit them to the virtual reality processing module;

[0012] The virtual reality processing module is used to perform 3D conversion processing on the real-time ultrasound image to obtain a virtual image with the same proportion as the puncture area. The virtual reality processing module performs edge recognition on the three-dimensional image, the virtual image and the three-dimensional human image of the patient in the surgeon's field of vision, determines the relative position of the three-dimensional image, the virtual image and the three-dimensional human image of the patient in the surgeon's field of vision and fuses them, and transmits the fused image to the virtual reality display overlay module.

[0013] The virtual reality display overlay module is used to overlay the fused image onto the patient's three-dimensional human image in the surgeon's field of vision and transmit it to the mixed reality device for the surgeon to view.

[0014] Preferably, the medical imaging scanning device is a CT, MRI, or PET-CT.

[0015] Preferably, the mixed reality device is a VR or MR visual device.

[0016] Preferably, the patient puncture site includes the chest, abdomen, and spine.

[0017] Secondly, this application provides a mixed reality-based ultrasound subcutaneous puncture guidance method, employing the aforementioned mixed reality-based ultrasound subcutaneous puncture guidance system, comprising the following steps:

[0018] S1. A two-dimensional image of the patient's puncture site is scanned by a medical imaging scanning device;

[0019] S2. The medical imaging scanning equipment performs three-dimensional reconstruction on two-dimensional images to generate three-dimensional images and transmits the three-dimensional images to the virtual reality processing module.

[0020] S3. The ultrasound equipment acquires real-time ultrasound images of the puncture site and transmits them to the virtual reality processing module.

[0021] S4. The mixed reality device acquires a three-dimensional human image of the patient in the surgeon's field of vision and transmits it to the virtual reality processing module;

[0022] S5. The virtual reality processing module performs 3D conversion processing on the real-time ultrasound image, converting it into a virtual image with the same proportion as the puncture area. It also performs edge recognition on the three-dimensional image, the virtual image, and the three-dimensional human image of the patient in the surgeon's field of vision, determines the relative positions of the three-dimensional image, the virtual image, and the three-dimensional human image of the patient in the surgeon's field of vision, and fuses them. The fused image is then transmitted to the virtual reality display overlay module.

[0023] S6. The virtual reality display overlay module overlays the fused image onto the patient's three-dimensional human image in the surgeon's field of vision.

[0024] S7. The virtual reality display overlay module transmits the overlaid image to the mixed reality device for the operator to view.

[0025] Preferably, in step S3, the real-time ultrasound image during puncture includes images of the puncture site and the puncture needle.

[0026] Preferably, in step S5, the relative position of the puncture site and the puncture needle is also displayed through a virtual image of equal scale.

[0027] The technical solution of this invention is based on the fusion of two-dimensional images from real-time ultrasound and dynamic images from three-dimensional models based on CT, MRI, or PET-CT, which can simultaneously display the needle tip, needle shaft, and perspective view of the puncture site. Furthermore, because the ultrasound equipment acquires images in real time and fuses them with the three-dimensional model in real time using mixed reality technology, image displacement caused by the patient's heartbeat or breathing is avoided. Additionally, no additional spatial markers or spatial positioning equipment are required, reducing equipment complexity and surgical preparation time. The surgeon can complete the entire puncture process alone, making it simple and easy to learn with almost no learning curve. This invention has excellent market application and promotion prospects. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is an architectural diagram of an embodiment of a mixed reality-based ultrasound subcutaneous puncture guidance system;

[0030] Figure 2 This is a flowchart illustrating an embodiment of an ultrasound-guided subcutaneous puncture method based on mixed reality.

[0031] In the above figures, the figure numbers indicate the following:

[0032] 1. Medical imaging scanning equipment;

[0033] 2. Ultrasonic equipment;

[0034] 3. Virtual Reality Processing Module;

[0035] 4. Virtual reality display overlay module;

[0036] 5. Mixed Reality Devices. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example

[0039] In one specific embodiment, such as Figure 1 and Figure 2As shown, a mixed reality-based ultrasound-guided subcutaneous puncture system includes a medical imaging scanning device 1, an ultrasound device 2, a virtual reality processing module 3, a virtual reality display overlay module 4, and a mixed reality device 5. The medical imaging scanning device 1 scans two-dimensional images of the patient's puncture site; the medical imaging scanning device 1 performs three-dimensional reconstruction of the two-dimensional images to generate three-dimensional images; the medical imaging scanning device 1 sends the three-dimensional images to the virtual reality processing module 3; the ultrasound device 2 acquires real-time ultrasound images of the patient's puncture site and transmits them to the virtual reality processing module 3; the mixed reality device 5 acquires and transmits three-dimensional human images of the patient within the operator's field of vision. The virtual reality processing module 3 is used to perform 3D conversion processing on the real-time ultrasound image to obtain a virtual image with the same proportion as the puncture area. The virtual reality processing module 3 performs edge recognition on the three-dimensional image, the virtual image and the three-dimensional human image of the patient in the surgeon's field of vision, determines the relative position of the three-dimensional image, the virtual image and the three-dimensional human image of the patient in the surgeon's field of vision and fuses them, and transmits the fused image to the virtual reality display overlay module 4. The virtual reality display overlay module 4 is used to overlay the fused image on the three-dimensional human image of the patient in the surgeon's field of vision and transmit it to the mixed reality device 5 for the surgeon to view.

[0040] In this embodiment, the medical imaging scanning device 1 is a PET-CT.

[0041] In this embodiment, the mixed reality device 5 is a VR device.

[0042] In this embodiment, the patient's puncture site is the chest.

[0043] Based on the aforementioned mixed reality-based ultrasound subcutaneous puncture guidance system, this embodiment of the invention also provides a mixed reality-based ultrasound subcutaneous puncture guidance method, the steps of which are as follows:

[0044] S1. A two-dimensional image of the patient's puncture site is scanned by a medical imaging scanning device;

[0045] S2. The medical imaging scanning equipment performs three-dimensional reconstruction on two-dimensional images to generate three-dimensional images and transmits the three-dimensional images to the virtual reality processing module.

[0046] S3. The ultrasound equipment acquires real-time ultrasound images of the puncture site and transmits them to the virtual reality processing module.

[0047] S4. The mixed reality device acquires a three-dimensional human image of the patient in the surgeon's field of vision and transmits it to the virtual reality processing module;

[0048] S5. The virtual reality processing module performs 3D conversion processing on the real-time ultrasound image, converting it into a virtual image with the same proportion as the puncture area. It also performs edge recognition on the three-dimensional image, the virtual image, and the three-dimensional human image of the patient in the surgeon's field of vision, determines the relative positions of the three-dimensional image, the virtual image, and the three-dimensional human image of the patient in the surgeon's field of vision, and fuses them. The fused image is then transmitted to the virtual reality display overlay module.

[0049] S6. The virtual reality display overlay module overlays the fused image onto the patient's three-dimensional human image in the surgeon's field of vision.

[0050] S7. The virtual reality display overlay module transmits the overlaid image to the mixed reality device for the operator to view.

[0051] In this embodiment, in step S3, the real-time ultrasound image during puncture includes images of the puncture site and the puncture needle.

[0052] In this embodiment, in step S5, the relative position of the puncture site and the puncture needle is also displayed by a virtual image of equal scale.

[0053] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A mixed reality based ultrasound subcutaneous puncture guiding system, characterized in that, This includes medical imaging scanning equipment, ultrasound equipment, virtual reality processing modules, virtual reality display overlay modules, and mixed reality equipment; The medical imaging scanning device is used to scan a two-dimensional image of the patient's puncture site. The medical imaging scanning device performs three-dimensional reconstruction on the two-dimensional image to generate a three-dimensional image. The medical imaging scanning device sends the three-dimensional image to the virtual reality processing module. The ultrasound device is used to acquire real-time ultrasound images of the patient's puncture site and transmit them to the virtual reality processing module. The mixed reality device is used to acquire three-dimensional human images of the patient in the surgeon's field of vision and transmit them to the virtual reality processing module; The virtual reality processing module is used to perform 3D conversion processing on the real-time ultrasound image to obtain a virtual image with the same proportion as the puncture area. The virtual reality processing module performs edge recognition on the three-dimensional image, the virtual image and the three-dimensional human image of the patient in the surgeon's field of vision, determines the relative position of the three-dimensional image, the virtual image and the three-dimensional human image of the patient in the surgeon's field of vision and fuses them, and transmits the fused image to the virtual reality display overlay module. The virtual reality display overlay module is used to overlay the fused image onto the patient's three-dimensional human image in the surgeon's field of vision and transmit the overlaid real-time image to the mixed reality device for the surgeon to view.

2. The mixed reality based ultrasound subdermal puncture guidance system of claim 1, wherein, The medical imaging scanning equipment is CT, MRI or PET-CT.

3. The mixed reality based ultrasound subdermal puncture guidance system of claim 2, wherein, The mixed reality device is a VR or MR visual device.

4. The mixed reality based ultrasound subdermal puncture guidance system of claim 3, wherein, The patient's puncture sites include the chest, abdomen, and spine.

5. A mixed reality based ultrasound subcutaneous puncture guiding method, characterized by, The ultrasonic subcutaneous puncture guidance system based on mixed reality, as described in any one of claims 1 to 4, comprises the following steps: S1. A two-dimensional image of the patient's puncture site is scanned by a medical imaging scanning device; S2. The medical imaging scanning equipment performs three-dimensional reconstruction on two-dimensional images to generate three-dimensional images and transmits the three-dimensional images to the virtual reality processing module. S3. The ultrasound equipment acquires real-time ultrasound images of the puncture site and transmits them to the virtual reality processing module. S4. The mixed reality device acquires a three-dimensional human image of the patient in the surgeon's field of vision and transmits it to the virtual reality processing module; S5. The virtual reality processing module performs 3D conversion processing on the real-time ultrasound image, converting it into a virtual image with the same proportion as the puncture area. It also performs edge recognition on the three-dimensional image, the virtual image, and the three-dimensional human image of the patient in the surgeon's field of vision, determines the relative positions of the three-dimensional image, the virtual image, and the three-dimensional human image of the patient in the surgeon's field of vision, and fuses them. The fused image is then transmitted to the virtual reality display overlay module. S6. The virtual reality display overlay module overlays the fused image onto the patient's three-dimensional human image in the surgeon's field of vision. S7. The virtual reality display overlay module transmits the overlaid image to the mixed reality device for the operator to view.

6. The mixed reality based ultrasound subdermal puncture guidance method of claim 5, wherein, In step S3, the real-time ultrasound image during puncture includes images of the puncture site and the puncture needle.

7. The mixed reality based ultrasound subdermal puncture guidance method of claim 6, wherein, In step S5, the relative position of the puncture site and the puncture needle is also displayed through a virtual image of equal scale.

Images