Augmented reality-assisted system and system for interventional therapy

Abstract

The present invention provides an augmented reality (AR)-assisted system for interventional therapy, comprising: a plurality of positioning sheets attached to the body surface of a patient; an interventional therapy imaging device used for collecting sequential 2D images of a lesion area of the patient that comprise the positioning sheets; a 3D model reconstruction module used for generating, on the basis of a preoperative sequential 2D image, a corresponding 3D virtual model, obtaining positional information of an interventional instrument inside the body of the patient on the basis of an intraoperative sequential 2D image, and adding the positional information into the 3D virtual model; an image processing module used for carrying out virtual-real fusion on the 3D virtual model and the body of the patient on the basis of the plurality of positioning sheets to generate a corresponding virtual-real fusion image; and AR glasses used for displaying the virtual-real fusion image in real time, a plurality of virtual positioning sheets in the virtual-real fusion image respectively coinciding with the plurality of positioning sheets on the body surface of the patient in position. The present invention also provides a navigation method in interventional therapy. According to the present invention, the frequency of intraoperative image acquisition can be reduced, improving the surgical safety and reducing the dependence on the surgeon.

Description

Augmented reality assistance system and method for interventional therapy Technical Field

[0001] The present invention relates to the technical field of interventional therapy equipment, and in particular to an augmented reality auxiliary system and method for interventional therapy. Background Art

[0002] During interventional procedures, stents and other interventional devices are typically implanted into blood vessels. Due to vascular hypoplasia, tortuosity, and elevated origins, the paths these devices must follow are complex. Accurately determining the device's position within the body is crucial, particularly at vascular bifurcations, as failure to do so increases the risk of vascular damage.

[0003] Current interventional procedures primarily rely on DSA (Digital Subtract Angiography), which relies on highly penetrating X-rays to obtain 2D black-and-white Dicom (Digital Imaging and Communications in Medicine) images, combined with contrast agents to highlight the location of blood vessels. This technology not only requires a large amount of contrast agent, but also requires multiple X-rays as the interventional device navigates within the vessel, inevitably exposing the patient and physician to significant ionizing radiation. Furthermore, the procedure relies heavily on the physician's experience to determine the course and bifurcation of the vessels, which increases the time and complexity of the procedure.

[0004] On the other hand, mainstream interventional medical device manufacturers maintain a monopoly over their DSA, CTA (CT Angiography), and MRA (Magnetic Resonance Angiography) equipment, with limited API access. This leads to issues such as delays in reading Dicom (Digital Imaging and Communications in Medicine) images and poor compatibility with older equipment, hindering the implementation of surgeries. Technical issues

[0005] The purpose of the present invention is to provide an augmented reality auxiliary system and method for interventional therapy, which can generate a corresponding 3D virtual model based on a pre-collected 2D sequence image of the lesion area, and display a virtual-real fusion image that integrates the 3D virtual model and the patient's body through AR (Augmented Reality) glasses, while adding the position information of the interventional instrument collected during the operation to the virtual-real fusion image. The present invention can reduce the number of times 2D sequence images are collected during surgery, reduce the radiation dose of the interventional therapy imaging equipment and the amount of contrast agent used during surgery, shorten the operation time, and greatly reduce the dependence on the operator's experience. The present invention can also quickly and in real time update the virtual-real fusion image based on the changes in the operator's viewpoint to meet practical needs. Solution

[0006] In order to achieve the above objectives, the present invention provides an augmented reality assistance system for interventional therapy, comprising:

[0007] Multiple positioning sheets are attached to the patient's body surface;

[0008] An interventional therapy imaging device, used for acquiring a sequence of 2D images of the patient's lesion area including the positioning film;

[0009] a 3D model reconstruction module, which generates a corresponding 3D virtual model based on the serial 2D images acquired before the operation, obtains position information of the interventional device in the patient's body based on the serial 2D images acquired during the operation, and adds the position information to the 3D virtual model;

[0010] An image processing module, which performs virtual-real fusion of the 3D virtual model and the patient's body based on the multiple positioning slices to generate a corresponding virtual-real fusion image;

[0011] AR glasses are used to display the virtual-reality fusion image in real time; multiple virtual positioning pieces in the virtual-reality fusion image respectively coincide with multiple positioning pieces on the patient's body surface.

[0012] Optionally, the image processing module includes a calculation unit, which is configured to calculate the coordinates of each point on the 3D virtual model in a new AR coordinate system based on the change of the operator's viewpoint; the image processing module updates the virtual-reality fusion image based on the coordinates in the new AR coordinate system; the AR coordinate system is based on the points in the patient's body. is the origin, the point Located directly above the center of the operating table, the Z axis of the AR coordinate system passes through the point and the surgeon's perspective;

[0013] Calculating the coordinates of each point on the 3D virtual model in the new AR coordinate system includes:

[0014] Obtaining a corresponding coordinate change matrix based on the change of the operator's viewpoint; calculating the coordinates of each positioning piece in the new 3D virtual model in the new AR coordinate system based on the coordinate change matrix;

[0015] The 3D virtual model is divided into multiple 3D areas corresponding to multiple positioning pieces, and the distance between each point on the 3D area and the corresponding positioning piece is less than a set threshold; according to the coordinates of the positioning piece in the new AR coordinate system, the coordinates of each point on the 3D area corresponding to the positioning piece in the new AR coordinate system are calculated through translation operation.

[0016] Optionally, when the operator moves, his viewpoint changes from the first viewpoint Become a second viewpoint hour:

[0017] Let the first vector The starting point is , point to , ; Let the second vector The starting point is point to , ; First vector With the second vector The angle between ; At the point A sphere with a center of 1 and a radius of 1 On the surface, there exists a unit vector , Perpendicular to the first vector With the second vector The plane of composition;

[0018] The coordinate change matrix includes a first change matrix ;

[0019] The positions of the rth positioning piece in the first AR coordinate system and the second AR coordinate system are respectively recorded as is the total number of positioning pieces; the first AR coordinate system and the second AR coordinate system correspond to the first viewpoint , Second Viewpoint ;

[0020] is the vector in the first AR coordinate system, and the vector in the second AR coordinate system , express The inverse of

[0021] Through translation operation, based on The coordinates of each point on the 3D area corresponding to the rth positioning piece in the second AR coordinate system are calculated.

[0022] Optionally, the operator rotates the angle around a virtual rotation axis The operator's perspective changes from the first perspective Become a third point of view ; Let the first vector The starting point is , point to , ; The rotation axis passes through the point and the first viewpoint ;

[0023] The positions of the rth positioning piece in the plurality of positioning pieces in the first AR coordinate system and the third AR coordinate system are respectively recorded as is the total number of positioning pieces; the first AR coordinate system and the third AR coordinate system correspond to the first viewpoint , Third Viewpoint ;

[0024] The coordinate change matrix includes a second change matrix ;

[0025] The position of the rth positioning piece in the third AR coordinate system is recorded as ; is the vector in the first AR coordinate system, and the vector in the third AR coordinate system , express The inverse of

[0026] Through translation operation, based on The coordinates of each point on the 3D area corresponding to the rth positioning slice in the third AR coordinate system are calculated.

[0027] Optionally, the 3D virtual model includes highlighted surgical information, wherein the surgical information includes any one or more of: an insertion position of an interventional device, a vascular path to a lesion;

[0028] The 3D model reconstruction module includes an image segmentation unit for segmenting the region corresponding to each tissue from the 2D image; the 3D model reconstruction module also displays or does not display the image of the corresponding non-vascular tissue in the 3D virtual model based on the received instructions.

[0029] Optionally, the interventional therapy imaging device includes: any one of a digital subtraction angiography device, a CT device, and a magnetic resonance angiography device.

[0030] Optionally, the positioning pieces are made of metal, and the surfaces of the plurality of positioning pieces are respectively provided with different patterns.

[0031] Optionally, the augmented reality assistance system for interventional therapy further includes an image display unit for displaying the 2D image acquired during the operation in real time.

[0032] The present invention further provides a navigation method for interventional therapy, which is implemented using the augmented reality assistance system for interventional therapy as described in the present invention. The method comprises the steps of:

[0033] S1. Fixing multiple positioning sheets on the skin surface of the patient's lesion area; collecting serial 2D images of the patient's lesion area before surgery to generate a corresponding 3D virtual model;

[0034] S2. Displaying a virtual-real fusion image including the 3D virtual model through AR glasses; overlapping multiple virtual positioning pieces in the virtual-real fusion image with multiple positioning pieces on the patient's skin surface; and inserting an interventional device.

[0035] S3. Excite the head of the interventional instrument and collect a sequence of 2D images of the patient's lesion area to obtain the position information of the interventional instrument in the patient's body; display the position information of the interventional instrument in the virtual-real fusion image; advance the interventional instrument; repeat step S3 until the interventional instrument reaches the lesion area.

[0036] Optionally, the navigation method during interventional therapy further comprises the steps of: calculating the coordinates of each point on the 3D virtual model in a new AR coordinate system based on a change in the operator's viewpoint, so as to update the virtual-reality fusion image;

[0037] Calculating the coordinates of each point on the 3D virtual model in the new AR coordinate system includes:

[0038] Obtaining a corresponding coordinate change matrix based on the change of the operator's viewpoint; obtaining the coordinates of each positioning piece in the 3D virtual model in the new AR coordinate system based on the coordinate change matrix;

[0039] The 3D virtual model is divided into multiple 3D areas corresponding to the multiple positioning pieces; based on the coordinates of the positioning pieces in the new AR coordinate system, the coordinates of each point on the 3D area corresponding to the positioning piece in the new AR coordinate system are calculated through translation operation. Beneficial effects

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1) The augmented reality-assisted system and method for interventional therapy of the present invention reconstructs a corresponding 3D virtual model based on 2D sequential images of the patient's lesion area collected preoperatively. The system then displays a fused virtual-real image of the 3D virtual model and the patient's body through AR glasses. The position information of the interventional instrument collected during the operation is also incorporated into the fused virtual-real image. The 3D virtual model segments the intraoperative blood vessels and other non-vascular tissues, and highlights the vascular path to the lesion. This intuitively visualizes the path of the interventional instrument within the blood vessels for the operator in the surgical area, eliminating the need for the operator to switch back and forth between the surgical area and the display screen (used to display 2D sequential images). This significantly reduces the time required to determine and correct errors in vascular bifurcations during surgery, reduces reliance on the operator and observation room assistants, reduces the number of surgical staff, and improves the success rate of the procedure.

[0042] 2) This invention can reduce the number of 2D image sequences acquired during surgery, significantly lowering the radiation exposure of interventional imaging equipment, improving surgical safety and minimizing harm to patients and surgeons. This invention reduces the interaction costs between doctors and medical equipment, ultimately reducing surgical time and costs.

[0043] 3) The present invention requires no or only a small amount of contrast agent, thereby reducing the safety risks caused by the patient's adverse reaction to the contrast agent.

[0044] 4) This invention can perform noise reduction and smoothing on preoperative 2D image sequences, enabling the presentation of a high-definition 3D virtual model during surgery without sacrificing intraoperative time. This method can rapidly calculate the coordinates of the 3D virtual model in the new AR coordinate system, adapting to the surgeon's viewpoint, and updating the fused virtual and real image in real time. This method offers fast computation speed, low overhead, and accurate results, effectively ensuring surgical outcomes.

[0045] 5) The augmented reality-assisted system of this invention is compatible with existing interventional imaging equipment, eliminating the need for modification and reducing the cost of interventional procedures. This invention can also be used in medical education, where intuitive 3D virtual models help improve teaching efficiency and reduce the learning costs of medical students. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1 is a schematic diagram of an augmented reality assistance system for interventional therapy according to the present invention;

[0047] FIG2 is a schematic diagram of the operator's viewpoint, positioning film, and interventional instrument in Example 1 of the present invention;

[0048] FIG3 is a schematic diagram of a 3D virtual model displayed on the surface of a pig body;

[0049] FIG4 is a schematic diagram showing changes in coordinates of a positioning plate when the operator changes from a first viewpoint to a second viewpoint in the first embodiment of the present invention;

[0050] FIG5 is a schematic diagram showing changes in coordinates of a positioning plate when the operator changes from a first viewpoint to a third viewpoint in the first embodiment of the present invention;

[0051] FIG6 is a flow chart of a navigation method during interventional therapy according to an embodiment of the present invention. Best Mode for Carrying Out the Invention

[0052] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are greatly simplified and not to exact ratios, and are intended solely to facilitate and clearly illustrate the embodiments of the present invention.

[0053] Current interventional procedures primarily rely on DSA, combined with contrast agents to highlight vascular locations. This not only requires a large amount of contrast agent, but also requires multiple X-rays as the interventional device navigates within the vessel, inevitably exposing both the patient and the physician to significant ionizing radiation. Furthermore, the procedure relies on the physician's experience to determine the course and bifurcation of the vessels. Misjudgments can increase the risk of vascular injury, leading to increased surgical time and complexity.

[0054] Augmented reality glasses or helmets overlay virtual objects in the user's direct field of view through transparent glass. Unlike virtual reality (VR), where the user's vision is blocked, users in augmented reality can observe both the real and virtual worlds simultaneously.

[0055] This invention applies augmented reality technology to interventional surgery. Pre-collected 2D sequential images of the lesion area are used to generate a corresponding 3D virtual model. AR glasses are then used to display a fused image that combines the 3D virtual model with the patient's body. The positional information of the interventional device acquired during surgery is incorporated into this fused image, and the fused image is rapidly updated in real time based on the surgeon's viewpoint. The surgeon can use AR glasses to visually visualize the vascular pathways and the location of the interventional device on the human body, without having to switch back and forth between the surgical area and the display (which displays the 2D sequential images). This invention reduces the number of 2D sequential images acquired during surgery, lowering the radiation exposure and contrast agent usage of the interventional imaging equipment. It significantly shortens surgical time and reduces reliance on surgeon experience, thereby improving surgical safety and success rates. The augmented reality-assisted system is compatible with existing interventional imaging equipment, eliminating the need for modification and reducing the cost of interventional surgery. The invention can also be used in medical education. The intuitive 3D virtual model helps improve teaching efficiency and reduce the learning costs of medical students.

[0056] Example 1

[0057] The present invention provides an augmented reality auxiliary system for interventional therapy, as shown in FIG1 , comprising: a plurality of positioning films 10 , an interventional therapy imaging device 20 , a 3D model reconstruction module 30 , an image processing module 40 , and AR glasses 50 .

[0058] Multiple positioning sheets 10 are attached to the patient's body surface. In this embodiment, the positioning sheets 10 are made of metal, and each sheet has a different pattern on its surface. In a preferred embodiment, there are at least three positioning sheets 10, with the spacing between adjacent positioning sheets 10 being 3 to 7 cm. Figure 2 shows six positioning sheets 10, which is for illustrative purposes only.

[0059] The interventional imaging device 20 is used to acquire a series of 2D images of the patient's lesion area, including the positioning film 10. The interventional imaging device 20 can be a digital subtraction angiography (DSA), CT, or magnetic resonance angiography device. In this embodiment, the interventional imaging device 20 is a DSA. Because the positioning film 10 is made of metal, it exhibits high brightness in 2D images, making it easily distinguishable from human tissue. It should be noted that the positioning film 10 is affixed to the patient's body surface during preoperative image acquisition and is not removed until after the procedure is complete.

[0060] The 3D model reconstruction module 30 generates a corresponding 3D virtual model based on the serial 2D images collected before the operation. Figure 3 is a schematic diagram of the 3D virtual model mapped on the surface of the pig body in the experiment. The 3D model reconstruction module 30 can also perform noise reduction, smoothing and other processing on the preoperative 2D serial images, greatly improving the clarity of the model. The 3D model reconstruction module 30 also obtains the position information of the interventional device 70 in the patient's body based on the serial 2D images collected during the operation, and adds the position information to the 3D virtual model. In this embodiment, the 3D virtual model also includes highlighted surgical information, such as: the insertion position of the interventional device 70, the vascular path to the lesion, etc.

[0061] As shown in Figure 1, the 3D model reconstruction module 30 in this embodiment includes an image segmentation unit 31, which is used to segment the regions corresponding to various tissues from the 2D image. The 3D model reconstruction module 30 also displays or hides images of corresponding non-vascular tissues in the 3D virtual model based on received instructions. These instructions can be voice commands from the surgeon, such as "Show the heart region," "Hide the liver region," etc. This allows the surgeon to focus on the surgical area during surgery, avoiding distractions from other information.

[0062] Based on the multiple positioning films 10, the image processing module 40 fuses the 3D virtual model and the patient's body into a virtual-real image, generating a corresponding fused image. This fused image is then displayed in real time to the operator via the AR glasses 50. The multiple virtual positioning films 10 in the fused image overlap with the multiple positioning films 10 on the patient's body surface. As can be easily understood, the positioning films 10 accurately anchor the 3D virtual model to the human body.

[0063] The 3D virtual model allows the surgeon to intuitively visualize the vascular path to the lesion within the surgical area, allowing the surgeon to insert the interventional instrument 70 based on the virtual-real fusion image of the surgical area. The surgeon can also activate the DSA scan during surgery by pressing a foot pedal. The 3D model reconstruction module 30 obtains the position of the interventional instrument 70 within the human body based on the DSA images acquired during surgery, and displays this position information in real time within the 3D virtual model. This eliminates the need for the surgeon's viewpoint to switch back and forth between the surgical area and the display screen (used to display 2D sequential images). This significantly reduces the time required to judge and correct errors at vascular bifurcations during surgery. The ability to clearly and intuitively observe the 3D virtual model reduces reliance on the surgeon's experience, reduces the number of surgical staff, and improves the success rate of surgery.

[0064] In existing technology, the interventional device 70 may need to capture a new DSA image after traveling a short distance, resulting in significant radiation exposure for both the patient and the surgeon. The present invention reduces the number of intraoperative image captures, significantly lowering the radiation exposure of the interventional imaging device 20, improving surgical safety and minimizing harm to both the patient and the surgeon. This invention reduces the interaction costs between the surgeon and the medical device, ultimately reducing surgical time and costs. The present invention eliminates or requires only a small amount of contrast agent, minimizing safety risks associated with adverse patient reactions to contrast agents.

[0065] As shown in FIG1 , the image processing module 40 in this embodiment includes a calculation unit 41. The calculation unit 41 is configured to calculate the coordinates of each point on the 3D virtual model in a new AR coordinate system (corresponding to the operator's viewpoint) based on the change of the operator's viewpoint. The image processing module 40 updates the virtual-reality fusion image based on the coordinates in the new AR coordinate system. The AR coordinate system is based on the points in the patient's body. is the origin (in the vertical direction, point The same distance from the operating table surface and the exposed surface of the human body. as a central point in the human body), point Located directly above the center of the operating table, corresponding to the Z-axis passing point of the AR coordinate system and the surgeon's viewpoint. As the surgeon moves around the operating table. It is regarded as a fixed point in the AR coordinate system to reduce the error of coordinate calculation.

[0066] Calculating the coordinates of each point on the 3D virtual model in the new AR coordinate system includes the following steps:

[0067] H1. Obtain a corresponding coordinate change matrix based on the change of the operator's viewpoint; calculate the coordinates of each positioning piece 10 in the new 3D virtual model in the new AR coordinate system based on the coordinate change matrix.

[0068] H2. The 3D virtual model is divided into multiple 3D areas corresponding to multiple positioning pieces 10, and the distance between each point on the 3D area and the corresponding positioning piece 10 is less than the set threshold; according to the coordinates of the positioning piece 10 in the new AR coordinate system, the coordinates of each point on the 3D area corresponding to the positioning piece 10 in the new AR coordinate system are calculated through translation operation.

[0069] Changes in the AR device's imaging caused by changes in perspective due to movement and head rotation are achieved by applying matrix transformations such as rotation and stretching to the vector between the viewpoint and a few anchor points (usually the two points with the shortest sum of the absolute distances between the interventional device and the viewpoint, with a third point selected every 10 frames for calibration). Because the patient's original modeling data in Euclidean space does not undergo any deformation, the vector used for translation operations on each point on the model is the difference between the updated and unupdated coordinates of the selected anchor point. This vector can be used to update the real-time position of all other locations using only vector addition and subtraction, rather than matrix multiplication.

[0070] The traditional calculation method on the left uses a dot matrix to calculate the coordinates of each point on the 3D virtual model. The difficulty lies in the high frame rate of current DSA images (usually exceeding 10 frames per second), and the amount of data calculation required is too large. At the current frame rate, according to the traditional calculation method, the calculation speed is bound to be too slow, and the virtual-reality fusion image cannot be updated in real time as the operator's perspective changes. In the present invention, the coordinates of the positioning film 10 are calculated through matrix multiplication, and the coordinates of other points on the 3D area corresponding to the positioning film 10 are updated through translation operations (addition and subtraction operations). Since matrix multiplication does not need to be performed on each point on the 3D virtual model, the amount of calculation is greatly reduced, ensuring that the virtual-reality fusion image is updated in real time as the operator's perspective changes. In addition, by calculating the coordinates of each point by region, it is ensured that the 3D virtual model observed by the operator accurately reflects the actual tissue in the patient's body.

[0071] The operator's viewpoint changes include two situations:

[0072] Case 1: As shown in Figure 4, when the operator moves, his viewpoint changes from the first viewpoint to the Become a second viewpoint .

[0073] Let the first vector The starting point is , point to , ; Let the second vector The starting point is point to , ; First vector With the second vector The angle between . At the point A sphere with a center of 1 and a radius of 1 On the surface, there exists a unit vector (Not shown in the figure, The calculation method of is prior art and will not be described here). Perpendicular to the first vector With the second vector The composed plane.

[0074] The coordinate change matrix contains the first change matrix .

[0075] The positions of the rth positioning piece in the plurality of positioning pieces 10 in the first AR coordinate system and the second AR coordinate system are respectively recorded as is the total number of positioning pieces 10. The first AR coordinate system and the second AR coordinate system correspond to the first viewpoint , Second Viewpoint .

[0076] is the vector in the first AR coordinate system, and the vector in the second AR coordinate system , express The inverse of.

[0077] Through translation operation, based on The coordinates of each point on the 3D area corresponding to the rth positioning slice in the second AR coordinate system are calculated.

[0078] Case 2: As shown in Figure 5, the operator rotates around a virtual rotation axis by an angle , the rotation axis passes through the point and the first viewpoint The operator's perspective changes from the first perspective Become a third point of view (Not shown in the figure).

[0079] The positions of the rth positioning piece in the plurality of positioning pieces 10 in the third AR coordinate system are respectively recorded as The third AR coordinate system corresponds to the third viewpoint .

[0080] The coordinate change matrix contains the second change matrix

[0081] Vector in the third AR coordinate system , express The inverse of.

[0082] Through translation operation, based on The coordinates of each point on the 3D area corresponding to the rth positioning slice in the third AR coordinate system are calculated.

[0083] The augmented reality assistance system in this embodiment further includes an image display unit 60 for displaying 2D images acquired during surgery in real time, thereby providing surgical information to surgical assistants who are not wearing AR glasses 50.

[0084] The present invention further provides a navigation method for interventional therapy, which is implemented using the augmented reality assistance system for interventional therapy as described in the present invention. As shown in FIG6 , the method comprises the steps of:

[0085] S1. Fixing a plurality of positioning sheets 10 on the skin surface of the patient's lesion area; collecting a sequence of 2D images of the patient's lesion area before surgery to generate a corresponding 3D virtual model;

[0086] S2. Displaying a virtual-real fusion image including a 3D virtual model through the AR glasses 50; overlapping the multiple virtual positioning pieces 10 in the virtual-real fusion image with the multiple positioning pieces 10 on the patient's skin surface; inserting the interventional instrument 70;

[0087] S3. Excite the interventional instrument head and collect serial 2D images of the patient's lesion area to obtain the position information of the interventional instrument 70 in the patient's body; display the position information of the interventional instrument 70 in the virtual-real fusion image; advance the interventional instrument 70; repeat step S3 until the interventional instrument 70 reaches the lesion area.

[0088] When the operator moves, step S3 further includes: calculating the coordinates of each point on the 3D virtual model in the new AR coordinate system based on the change of the operator's viewpoint to update the virtual and real fusion image;

[0089] Calculating the coordinates of each point on the 3D virtual model in the new AR coordinate system includes:

[0090] Based on the change of the operator's viewpoint, a corresponding coordinate change matrix is ​​obtained; based on the coordinate change matrix, the coordinates of each positioning piece 10 in the 3D virtual model in the new AR coordinate system are obtained;

[0091] The 3D virtual model is divided into multiple 3D areas corresponding to multiple positioning pieces 10; according to the coordinates of the positioning piece 10 in the new AR coordinate system, the coordinates of each point on the 3D area corresponding to the positioning piece 10 in the new AR coordinate system are calculated through translation operation.

[0092] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An augmented reality assisted system for interventional therapy, characterized in that: Include: Multiple positioning sheets are attached to the patient's body surface; An interventional therapy imaging device, used for acquiring a sequence of 2D images of the patient's lesion area including the positioning film; A 3D model reconstruction module generates a corresponding 3D virtual model based on the sequence of 2D images collected before the operation, obtains the position information of the interventional device in the patient's body based on the sequence of 2D images collected during the operation, and adds the position information to the 3D virtual model; An image processing module, which performs virtual-real fusion of the 3D virtual model and the patient's body based on the multiple positioning films to generate a corresponding virtual-real fusion image; The AR glasses are used to display the virtual-real fusion image in real time; the multiple virtual positioning pieces in the virtual-real fusion image are respectively overlapped with the multiple positioning pieces on the patient's body surface.

2. The augmented reality assistance system for interventional therapy according to claim 1, characterized in that: The image processing module includes a calculation unit, which is configured to calculate the coordinates of each point on the 3D virtual model in a new AR coordinate system based on the change of the operator's viewpoint; the image processing module updates the virtual-reality fusion image based on the coordinates in the new AR coordinate system; the AR coordinate system is based on the points in the patient's body. is the origin, the point Located directly above the center of the operating table, the Z axis of the AR coordinate system passes through the point and the surgeon’s point of view; The calculation of the coordinates of each point on the 3D virtual model in the new AR coordinate system includes: Based on the change of the operator's viewpoint, a corresponding coordinate change matrix is ​​obtained; based on the coordinate change matrix, the coordinates of each positioning piece in the new 3D virtual model in the new AR coordinate system are calculated; The 3D virtual model is divided into multiple 3D areas corresponding to multiple positioning pieces, and the distance between each point on the 3D area and the corresponding positioning piece is less than a set threshold; according to the coordinates of the positioning piece in the new AR coordinate system, the coordinates of each point on the 3D area corresponding to the positioning piece in the new AR coordinate system are calculated through translation operation.

3. The augmented reality assistance system for interventional therapy according to claim 2, characterized in that: When the operator moves, his viewpoint changes from the first viewpoint Become a second viewpoint hour: Let the first vector The starting point is ， Oriented ， ; Let the second vector The starting point is Oriented ， ; The first vector With the second vector The angle between ; At the point A sphere with a center of 1 and a radius of 1 There exists a unit vector ， Perpendicular to the first vector With the second vector The plane formed; The coordinate change matrix includes a first change matrix ； The positions of the rth positioning piece among the plurality of positioning pieces in the first AR coordinate system and the second AR coordinate system are respectively recorded as is the total number of positioning pieces; the first AR coordinate system and the second AR coordinate system correspond to the first viewpoint , Second Viewpoint ； is the vector in the first AR coordinate system, and the vector in the second AR coordinate system ， express The inverse of Through translation operation, based on The coordinates of each point on the 3D area corresponding to the rth positioning piece in the second AR coordinate system are calculated.

4. The augmented reality assistance system for interventional therapy according to claim 2, characterized in that: The operator rotates around a virtual rotation axis , the operator's viewpoint changes from the first viewpoint Become a third point of view ; Let the first vector The starting point is , point to , ; The rotation axis passes through the point and the first viewpoint ; The positions of the rth positioning piece among the plurality of positioning pieces in the first AR coordinate system and the third AR coordinate system are respectively recorded as is the total number of positioning pieces; the first AR coordinate system and the third AR coordinate system correspond to the first viewpoint , Third Viewpoint ； The coordinate change matrix includes a second change matrix ； The position of the rth positioning piece in the third AR coordinate system is recorded as ； is the vector in the first AR coordinate system, and the vector in the third AR coordinate system ， express The inverse of Through translation operation, based on The coordinates of each point on the 3D area corresponding to the rth positioning piece in the third AR coordinate system are calculated.

5. The augmented reality assistance system for interventional therapy according to claim 1, characterized in that: The 3D virtual model includes highlighted surgical information, wherein the surgical information includes: any one or more of the insertion position of the interventional instrument and the vascular path to the lesion; The 3D model reconstruction module includes an image segmentation unit for segmenting the area corresponding to each tissue from the 2D image; the 3D model reconstruction module also displays or does not display the image of the corresponding non-vascular tissue in the 3D virtual model based on the received instructions.

6. The augmented reality assistance system for interventional therapy according to claim 1, characterized in that: Interventional therapy imaging equipment includes: any one of digital subtraction angiography equipment, CT equipment, and magnetic resonance angiography equipment.

7. The augmented reality assistance system for interventional therapy according to claim 1, characterized in that: The positioning pieces are made of metal, and different patterns are arranged on the surfaces of the plurality of positioning pieces.

8. The augmented reality assistance system for interventional therapy according to claim 1, characterized in that: It also includes an image display unit for displaying the 2D image acquired during the operation in real time.

9. A navigation method in interventional therapy, implemented by using the augmented reality auxiliary system for interventional therapy as claimed in any one of claims 1 to 8, characterized in that: Contains steps: S1. Fixing a plurality of positioning sheets on the skin surface of the patient's lesion area; collecting a sequence of 2D images of the patient's lesion area before surgery to generate a corresponding 3D virtual model; S2, displaying a virtual-real fusion image including the 3D virtual model through AR glasses; overlapping the positions of multiple virtual positioning pieces in the virtual-real fusion image with the positions of multiple positioning pieces on the patient's skin surface respectively; inserting an interventional instrument; S3, stimulate the head of the interventional instrument and collect a sequence of 2D images of the patient's lesion area to obtain the position information of the interventional instrument in the patient's body; display the position information of the interventional instrument in the virtual-real fusion image; determine the direction and distance of the interventional instrument, and advance the interventional instrument; repeat step S3 until the interventional instrument reaches the lesion area.

10. The navigation method during interventional therapy according to claim 9, characterized in that: The method further comprises the steps of: calculating the coordinates of each point on the 3D virtual model in a new AR coordinate system based on the change of the operator's viewpoint, so as to update the virtual-real fusion image; The calculation of the coordinates of each point on the 3D virtual model in the new AR coordinate system includes: Based on the change of the operator's viewpoint, a corresponding coordinate change matrix is ​​obtained; based on the coordinate change matrix, the coordinates of each positioning piece in the 3D virtual model in the new AR coordinate system are obtained; The 3D virtual model is divided into a plurality of 3D regions corresponding to the plurality of positioning pieces; and the coordinates of each point on the 3D region corresponding to the positioning piece in the new AR coordinate system are calculated through translation operation according to the coordinates of the positioning piece in the new AR coordinate system.

Images