Laparoscope augmented reality surgical navigation method, equipment and device

In laparoscopic augmented reality navigation technology, the vascular branches are marked using three-dimensional models and surface intersecting operations, which solves the problem of intuition of blood vessel display and improves the accuracy of surgical navigation.

CN119014979AActive Publication Date: 2024-11-26QINGDAO HISENSE MEDICAL EQUIP

Patent Information

Application Number
CN202310613885.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26
Estimated Expiration
2043-05-26

AI Technical Summary

Technical Problem

In the existing laparoscopic augmented reality navigation technology, the blood vessel display is not intuitive enough, resulting in low accuracy in vascular judgment in the internal lesions and in the safe resection area of ​​the lesions.

Method used

Based on the registered visceral three-dimensional model, the vertex coordinate set of lesions and blood vessel surface grid grid is obtained, and the intersecting surface of the blood vessel and the target area is determined through surface intersecting operations or distance operations, the blood vessel branches to be marked and displayed on the laparoscopic image. .

Benefits of technology

It improves the intuitive display of blood vessels in the target area, provides more accurate navigation assistance functions, and enhances the accuracy of the surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119014979A_ABST
    Figure CN119014979A_ABST
Patent Text Reader

Abstract

The invention discloses a laparoscope augmented reality surgical navigation method, equipment and device, and the method comprises the steps: firstly obtaining a focus surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set based on a registered viscera three-dimensional model, then carrying out the surface intersection operation or distance operation based on the obtained coordinate sets, and finally obtaining a target viscera three-dimensional model. After it is determined that an intersecting surface exists between the blood vessel and the target area, a blood vessel branch to be marked is determined according to an operation result, the blood vessel branch is correspondingly marked and then displayed on a laparoscopic image of the current view, and the target area can be the area where the focus is located and can also be a safe resection area where the focus is resected. The blood vessels in the target area can be marked and displayed on the laparoscope image, so that the blood vessels in the target area are displayed more visually, a navigation auxiliary function is provided for doctors in the operation process, and the accuracy of augmented reality operation navigation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical application technology, and in particular to a laparoscopic augmented reality surgery navigation method, equipment and device. Background Art

[0002] Laparoscopic liver resection has the advantages of small incision, less pain, quick recovery and small scar, and has become the current development trend of surgery.

[0003] At present, in laparoscopic augmented reality navigation technology, it is impossible to specially display the blood vessels in a specific area. For example, for the blood vessels inside the lesion, or for the blood vessels in the safe resection area of ​​the lesion, whether there are blood vessels in the lesion or in the safe resection area of ​​the lesion can only be judged based on the doctor's experience. This judgment method has low accuracy and the blood vessel display is not intuitive enough. Summary of the invention

[0004] The present invention provides a laparoscopic augmented reality surgery navigation method, equipment and device, which are used to solve the problem that blood vessel display is not intuitive enough in the laparoscopic augmented reality surgery navigation technology in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a laparoscopic augmented reality surgery navigation method, the method comprising:

[0006] Based on the registered visceral three-dimensional model, a mesh vertex coordinate set of the lesion surface and a mesh vertex coordinate set of the blood vessel surface are obtained;

[0007] Based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the target area to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set;

[0008] The blood vessel branches to be marked are determined according to the calculation results, and the blood vessel branches to be marked are marked accordingly and then displayed on the laparoscopic image of the current view.

[0009] In a possible implementation, the safety surface mesh vertex coordinate set is calculated in the following manner:

[0010] For each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculating the outer normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangular face to which the lesion surface mesh vertex coordinate belongs; multiplying the outer normal coordinate and a first preset threshold, adding the product to the lesion surface mesh vertex coordinate to obtain the safety surface mesh vertex coordinate;

[0011] The obtained safety surface mesh vertex coordinates are used as the safety surface mesh vertex coordinate set.

[0012] In a possible implementation, performing a surface intersection operation on the blood vessel and the target region based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersection surface between the blood vessel and the target region includes:

[0013] Based on the coordinates of each blood vessel surface mesh vertex in the blood vessel surface mesh vertex coordinate set and the coordinates of each target surface mesh vertex in the target surface mesh vertex coordinate set, using an intersection operation in a Boolean operation to perform a surface intersection operation on the blood vessel and the diseased target area;

[0014] If there is an operation result, it is determined that there is an intersection surface between the blood vessel and the target area.

[0015] In a possible implementation, performing a distance operation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersecting surface between the blood vessel and the target area includes:

[0016] For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculating a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set; and determining a first minimum distance among the calculated first distances;

[0017] Taking the maximum value among the determined first minimum distances as the first distance calculation result;

[0018] If the first distance calculation result is less than or equal to a second preset threshold, it is determined that there is an intersection surface between the blood vessel and the target area.

[0019] In a possible implementation, performing a distance operation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersecting surface between the blood vessel and the target area includes:

[0020] Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain the centerline coordinates of the blood vessel centerline;

[0021] For each center line coordinate: calculating a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, and determining a second minimum distance among the calculated second distances;

[0022] The maximum value among the determined second minimum distances is taken as the second distance calculation result;

[0023] If the second operation result is less than or equal to a third preset threshold, it is determined that there is an intersecting surface between the blood vessel and the target area.

[0024] In a possible implementation, determining the blood vessel branch to be marked according to the calculation result includes:

[0025] The center line coordinate corresponding to the minimum value in the determined second minimum distance is used as the target center line coordinate;

[0026] Determine a centerline branch corresponding to the target centerline coordinates according to the target centerline coordinates and adjacent neighborhoods of points on the centerline;

[0027] The blood vessel corresponding to the centerline branch is used as the blood vessel branch to be marked.

[0028] In a possible implementation, determining the centerline branch where the target centerline coordinates are located according to the target centerline coordinates and the adjacent neighborhood of the point on the centerline includes:

[0029] On the center line, determining a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate;

[0030] In a first direction, taking the first center line coordinate as a starting point, sequentially determine first numbers of adjacent center line coordinates; and in a second direction, taking the second center line coordinate as a starting point, sequentially determine second numbers of adjacent center line coordinates;

[0031] If the first number is greater than a fourth preset threshold, and the second number is greater than the fourth preset threshold, the center line between the center line coordinates corresponding to the first number and the center line coordinates corresponding to the second number is used as the center line branch.

[0032] In a possible implementation, the method further includes:

[0033] Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, performing a surface intersection operation or a distance operation on the blood vessel and the lesion to determine that there is no intersection surface between the blood vessel and the lesion;

[0034] Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the lesion safe resection area to determine that there is an intersection surface between the blood vessel and the lesion safe switching area.

[0035] In a second aspect, an embodiment of the present application provides a laparoscopic augmented reality fusion display device, the device comprising a processor and a display screen;

[0036] The processor is used to obtain a lesion surface mesh vertex coordinate set and a blood vessel surface mesh vertex coordinate set based on the registered visceral three-dimensional model; based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, perform a surface intersection operation or a distance operation on the blood vessel and the target area to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set; determine a blood vessel branch to be marked based on the operation result, and mark the blood vessel branch to be marked accordingly;

[0037] The display screen is used for laparoscopic augmented reality fusion display of the marked vascular branches.

[0038] In a possible implementation, the processor calculates the safety surface mesh vertex coordinate set in the following manner:

[0039] For each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculating the outer normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangular face to which the lesion surface mesh vertex coordinate belongs; multiplying the outer normal coordinate and a first preset threshold, adding the product to the lesion surface mesh vertex coordinate to obtain the safety surface mesh vertex coordinate;

[0040] The obtained safety surface mesh vertex coordinates are used as the safety surface mesh vertex coordinate set.

[0041] In a possible implementation manner, the processor is specifically configured to:

[0042] Based on the coordinates of each blood vessel surface mesh vertex in the blood vessel surface mesh vertex coordinate set and the coordinates of each target surface mesh vertex in the target surface mesh vertex coordinate set, using an intersection operation in a Boolean operation to perform a surface intersection operation on the blood vessel and the diseased target area;

[0043] If there is an operation result, it is determined that there is an intersection surface between the blood vessel and the target area.

[0044] In a possible implementation manner, the processor is specifically configured to:

[0045] For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculating a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set; and determining a first minimum distance among the calculated first distances;

[0046] Taking the maximum value among the determined first minimum distances as the first distance calculation result;

[0047] If the first distance calculation result is less than or equal to a second preset threshold, it is determined that there is an intersection surface between the blood vessel and the target area.

[0048] In a possible implementation manner, the processor is specifically configured to:

[0049] Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain the centerline coordinates of the blood vessel centerline;

[0050] For each center line coordinate: calculating a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, and determining a second minimum distance among the calculated second distances;

[0051] The maximum value among the determined second minimum distances is taken as the second distance calculation result;

[0052] If the second operation result is less than or equal to a third preset threshold, it is determined that there is an intersecting surface between the blood vessel and the target area.

[0053] In a possible implementation manner, the processor is specifically configured to:

[0054] The center line coordinate corresponding to the minimum value in the determined second minimum distance is used as the target center line coordinate;

[0055] Determine a centerline branch corresponding to the target centerline coordinates according to the target centerline coordinates and the adjacent neighborhood of the point on the centerline;

[0056] The blood vessel corresponding to the centerline branch is used as the blood vessel branch to be marked.

[0057] In a possible implementation manner, the processor is specifically configured to:

[0058] On the center line, determining a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate;

[0059] In a first direction, taking the first center line coordinate as a starting point, sequentially determine first numbers of adjacent center line coordinates; and in a second direction, taking the second center line coordinate as a starting point, sequentially determine second numbers of adjacent center line coordinates;

[0060] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, the center line between the center line coordinates corresponding to the first number and the center line coordinates corresponding to the second number is used as the center line branch.

[0061] In a possible implementation, the processor is further configured to:

[0062] Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, performing a surface intersection operation or a distance operation on the blood vessel and the lesion to determine that there is no intersection surface between the blood vessel and the lesion;

[0063] Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the lesion safe resection area to determine that there is an intersection surface between the blood vessel and the lesion safe switching area.

[0064] In a third aspect, an embodiment of the present application provides a laparoscopic augmented reality fusion display device, comprising:

[0065] An acquisition module, used for acquiring a mesh vertex coordinate set of a lesion surface and a mesh vertex coordinate set of a blood vessel surface based on the registered visceral three-dimensional model;

[0066] A calculation module, for performing a surface intersection calculation or a distance calculation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, to determine that the blood vessel and the target area have an intersection surface, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set;

[0067] Marking display module: used to determine the vascular branches to be marked according to the calculation results, and display the vascular branches to be marked on the laparoscopic image of the current view after marking them accordingly.

[0068] In a possible implementation, the computing module is specifically used for:

[0069] For each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculating the outer normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangular face to which the lesion surface mesh vertex coordinate belongs; multiplying the outer normal coordinate and a first preset threshold, adding the product to the lesion surface mesh vertex coordinate to obtain the safety surface mesh vertex coordinate;

[0070] The obtained safety surface mesh vertex coordinates are used as the safety surface mesh vertex coordinate set.

[0071] In a possible implementation, the computing module is specifically used for:

[0072] Based on the coordinates of each blood vessel surface mesh vertex in the blood vessel surface mesh vertex coordinate set and the coordinates of each target surface mesh vertex in the target surface mesh vertex coordinate set, using an intersection operation in a Boolean operation to perform a surface intersection operation on the blood vessel and the diseased target area;

[0073] If there is an operation result, it is determined that there is an intersection surface between the blood vessel and the target area.

[0074] In a possible implementation, the computing module is specifically used for:

[0075] For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculating a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set; and determining a first minimum distance among the calculated first distances;

[0076] Taking the maximum value among the determined first minimum distances as the first distance calculation result;

[0077] If the first distance calculation result is less than or equal to a second preset threshold, it is determined that there is an intersection surface between the blood vessel and the target area.

[0078] In a possible implementation, the computing module is specifically used for:

[0079] Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain the centerline coordinates of the blood vessel centerline;

[0080] For each center line coordinate: calculating a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, and determining a second minimum distance among the calculated second distances;

[0081] The maximum value among the determined second minimum distances is taken as the second distance calculation result;

[0082] If the second operation result is less than or equal to a third preset threshold, it is determined that there is an intersecting surface between the blood vessel and the target area.

[0083] In a possible implementation, the mark display module is specifically used for:

[0084] The center line coordinate corresponding to the minimum value in the determined second minimum distance is used as the target center line coordinate;

[0085] Determine a centerline branch corresponding to the target centerline coordinates according to the target centerline coordinates and adjacent neighborhoods of points on the centerline;

[0086] The blood vessel corresponding to the centerline branch is used as the blood vessel branch to be marked.

[0087] In a possible implementation, the mark display module is specifically used for:

[0088] On the center line, determining a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate;

[0089] In a first direction, taking the first center line coordinate as a starting point, sequentially determine first numbers of adjacent center line coordinates; and in a second direction, taking the second center line coordinate as a starting point, sequentially determine second numbers of adjacent center line coordinates;

[0090] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, the center line between the center line coordinates corresponding to the first number and the center line coordinates corresponding to the second number is used as the center line branch.

[0091] In a possible implementation, the computing module is specifically used for:

[0092] Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, performing a surface intersection operation or a distance operation on the blood vessel and the lesion to determine that there is no intersection surface between the blood vessel and the lesion;

[0093] Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the lesion safe resection area to determine that there is an intersection surface between the blood vessel and the lesion safe switching area.

[0094] The beneficial effects of the present invention are as follows:

[0095] The laparoscopic augmented reality surgical navigation method, equipment and device provided in the embodiment of the present application first obtains the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set based on the registered visceral three-dimensional model, and then performs surface intersection operation or distance operation based on the obtained coordinate set. After determining that there is an intersecting surface between the blood vessel and the target area, the blood vessel branch to be marked is determined according to the operation result, and the blood vessel branch is marked accordingly and displayed on the laparoscopic image of the current view. The target area here can be the area where the lesion is located, or it can be the safe resection area for resecting the lesion. Due to the laparoscopic augmented reality surgical navigation method provided in the embodiment of the present application, the blood vessels in the target area can be marked and displayed on the laparoscopic image, so that the blood vessels in the target area are displayed more intuitively, providing navigation assistance functions for doctors during the operation, thereby improving the accuracy of augmented reality surgical navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0097] Figure 1 A schematic diagram of the structure of a laparoscopic augmented reality surgical navigation system provided in an embodiment of the present application;

[0098] Figure 2 A schematic diagram of a flow chart of a laparoscopic augmented reality surgery navigation method provided in an embodiment of the present application;

[0099] Figure 3 A schematic diagram of a three-dimensional liver model provided in an embodiment of the present application;

[0100] Figure 4 A schematic diagram of a blood vessel branch to be marked provided in an embodiment of the present application;

[0101] Figure 5 A schematic diagram of a process for determining a blood vessel branch to be marked provided in an embodiment of the present application;

[0102] Figure 6 A schematic flow chart of another method for determining a blood vessel branch to be marked provided in an embodiment of the present application;

[0103] Figure 7 A schematic diagram of the structure of a blood vessel branch provided in an embodiment of the present application;

[0104] Figure 8 A schematic flow chart of another method for determining a blood vessel branch to be marked provided in an embodiment of the present application;

[0105] Fig. 9 A schematic diagram of a determined centerline branch provided in an embodiment of the present application;

[0106] Fig.10 A schematic diagram of a determined blood vessel branch to be marked provided in an embodiment of the present application;

[0107] Fig.11 A complete flowchart of a laparoscopic augmented reality surgical navigation method provided in an embodiment of the present application;

[0108] Fig.12 A schematic diagram of the structure of a laparoscopic augmented reality surgical navigation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0109] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0110] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0111] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0112] In order to facilitate understanding of a laparoscopic augmented reality surgical navigation method, device and apparatus provided in the embodiments of the present application, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0113] (1) Laparoscopic augmented reality surgical navigation technology is the main form of laparoscopic augmented reality navigation. It is mainly implemented by reconstructing the preoperative visceral three-dimensional model from CT / MRI medical images, and then registering the preoperative visceral three-dimensional model with the intraoperative laparoscopic image through image registration technology to generate the intraoperative visceral three-dimensional model. Finally, the augmented reality display technology is used to enhance the display of the intraoperative scene and realize image navigation.

[0114] (2) Augmented Reality (AR) technology is a new technology that "seamlessly" integrates real-world information and virtual-world information. It uses computers and other scientific technologies to simulate and superimpose physical information (visual information, sound, taste, touch, etc.) that is difficult to experience within a certain time and space in the real world, and then applies virtual information to the real world and is perceived by human senses, thereby achieving a sensory experience beyond reality. The real environment and virtual objects are superimposed on the same screen or space in real time and exist simultaneously. Augmented reality technology includes new technologies and new means such as multimedia, three-dimensional modeling, real-time video display and control, multi-sensor integration, real-time tracking and registration, and scene fusion.

[0115] The following is an introduction to the application scenario of a laparoscopic augmented reality surgical navigation method provided by an embodiment of the present application in conjunction with the accompanying drawings. Figure 1 The laparoscopic augmented reality surgical navigation system shown includes a laparoscope 2 and a laparoscopic augmented reality surgical navigation device 3, wherein the laparoscopic augmented reality surgical navigation device 3 includes a processor 101 and a display screen 102, wherein:

[0116] The processor 101 is used to obtain a lesion surface mesh vertex coordinate set and a blood vessel surface mesh vertex coordinate set based on the registered visceral three-dimensional model; based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, perform a surface intersection operation or a distance operation on the blood vessel and the target area to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to the lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set; determine the blood vessel branch to be marked according to the operation result, and mark the blood vessel branch to be marked accordingly;

[0117] The display screen 102 is used to display the marked blood vessel branches on the laparoscopic image of the current view.

[0118] Figure 1In the figure, 1 represents an internal organ. A preoperative three-dimensional model of the internal organ 1 is obtained before the operation. During the operation on the internal organ 1, the laparoscope 2 collects relevant information of the internal organ 1 in real time. The preoperative three-dimensional model of the internal organ 1 and the relevant information collected during the operation are aligned to obtain the aligned three-dimensional model, which is then superimposed on the laparoscopic image of the current view through enhanced display technology.

[0119] The method provided in the embodiments of the present application is not limited to Figure 1 The application scenario shown can also be used in other possible application scenarios, which are not limited in the embodiments of the present application.

[0120] like Figure 2 As shown, the flowchart of the laparoscopic augmented reality surgery navigation method of the embodiment of the present application, the specific steps are as follows:

[0121] S201, based on the registered visceral three-dimensional model, obtaining a lesion surface mesh vertex coordinate set and a blood vessel surface mesh vertex coordinate set;

[0122] S202, based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, performing a surface intersection operation or a distance operation on the blood vessel and the target area to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set;

[0123] S203 , determining the vascular branches to be marked according to the calculation results, marking the vascular branches to be marked accordingly and displaying them on the laparoscopic image of the current view.

[0124] The laparoscopic augmented reality surgical navigation method provided in the embodiment of the present application first obtains the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set based on the registered visceral three-dimensional model, and then performs surface intersection operation or distance operation based on the obtained coordinate set. After determining that there is an intersecting surface between the blood vessel and the target area, the blood vessel branch to be marked is determined according to the operation result, and the blood vessel branch is marked accordingly and displayed on the laparoscopic image of the current view. The target area here can be the area where the lesion is located, or it can be the safe resection area for resecting the lesion. Since the laparoscopic augmented reality surgical navigation method provided in the embodiment of the present application can mark the blood vessels in the target area and display them on the laparoscopic image, the blood vessels in the target area can be displayed more intuitively, providing doctors with navigation assistance functions during the operation, thereby improving the accuracy of augmented reality surgical navigation.

[0125] The three-dimensional viscera model used in the embodiment of the present application is a registered three-dimensional viscera model, that is, the three-dimensional viscera model before surgery is registered with the relevant information collected in the book.

[0126] The use of the registered visceral three-dimensional model can improve the accuracy of augmented reality surgical navigation, complete the augmented reality display of the intraoperative field of view during the operation, and complete the real-time navigation of the surgical process.

[0127] In a specific implementation, the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set can be directly obtained from the registered visceral three-dimensional model, and the blood vessel surface mesh vertex coordinate set obtained in the embodiment of the present application is a blood vessel surface mesh vertex coordinate set of the same type, for example, the blood vessel surface mesh vertex coordinate set of the portal vein. If the visceral three-dimensional model is a liver three-dimensional model, the blood vessel surface mesh vertex coordinates can also be the blood vessel surface mesh vertex coordinates of the hepatic vein.

[0128] like Figure 3 FIG. 1 is a partial schematic diagram of a three-dimensional liver model, in which there are lesions (not shown in the figure), portal veins and hepatic veins inside the liver. Figure 3 It can be seen that the visceral 3D model is composed of multiple triangular patches, each vertex of which is a surface mesh vertex coordinate, and the surface mesh vertex coordinate is a 3D coordinate. The liver 3D model includes the liver surface mesh vertex coordinates, the portal vein surface mesh vertex coordinates, the hepatic vein surface mesh vertex coordinates, and the lesion surface mesh vertex coordinates.

[0129] In one embodiment, when the target area is a safe resection area, the safe surface mesh vertex coordinates corresponding to the safe resection area can be obtained according to the lesion surface mesh vertex coordinate set.

[0130] Specifically, for each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculate the external normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangle patch to which the lesion surface mesh vertex coordinate belongs; multiply the external normal coordinate and the first preset threshold value, add the product to the lesion surface mesh vertex coordinate to obtain the safe surface mesh vertex coordinate; and use the obtained safe surface mesh vertex coordinate as the safe surface mesh vertex coordinate set.

[0131] When calculating the external normal coordinates corresponding to the vertex coordinates of the lesion surface mesh according to the triangular facet to which the vertex coordinates of the lesion surface mesh belong, first determine the triangular facet to which the vertex coordinates of the lesion surface mesh belong, then calculate the surface normal vector corresponding to each triangular facet, and calculate the average value of the calculated surface normal vectors, and the obtained value is the external normal coordinate corresponding to the vertex coordinates of the lesion surface mesh.

[0132] For example, the triangular patches to which the vertex coordinate a of the lesion surface mesh belongs include triangular patch 1, triangular patch 2 and triangular patch 3, wherein the vertex coordinates of each triangular patch are known, and the surface normal vector 1 corresponding to triangular patch 1 is calculated according to the three vertex coordinates corresponding to triangular patch 1, and the surface normal vector 2 corresponding to triangular patch 2 is calculated according to the three vertex coordinates corresponding to triangular patch 2, and the surface normal vector 3 corresponding to triangular patch 3 is calculated according to the three vertex coordinates corresponding to triangular patch 3, and then the average value of the surface normal vector 1, the surface normal vector 2 and the surface normal vector 3 is calculated, and the result is the external normal vector corresponding to the vertex coordinate a of the lesion surface mesh.

[0133] It should be noted that, in a specific implementation, the three vertices of a triangle are usually marked in a clockwise order. If the coordinate of vertex 1 of the triangle is V1, the coordinate of vertex 2 is V2, and the coordinate of vertex 3 is V3, then the first direction d1 = V2-V1, the second direction d2 = V3-V2, and the surface normal vector corresponding to the triangle is d1*d2.

[0134] After obtaining the external normal vector, for each lesion surface mesh vertex coordinate, the external normal vector is multiplied by the lesion surface mesh vertex coordinate, and then the product and the lesion surface mesh vertex coordinate corresponding to the product are added to obtain the safety surface mesh vertex coordinate.

[0135] For example, the external normal vector is Xi, the vertex coordinate set of the lesion surface mesh is {c1, c2, c3}, and the obtained vertex coordinate set of the safe surface mesh is {c1+Xi*c1, c2+Xi*c2, c3+Xi*c3}.

[0136] The laparoscopic augmented reality surgical navigation method provided in the embodiment of the present application can mark blood vessels in two areas, the blood vessels within the lesion and the blood vessels within the safe resection area of ​​the lesion. The embodiment of the present application is described in detail below using the marking of blood vessels within the lesion as an example.

[0137] In a specific implementation, after obtaining the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set, a surface intersection operation or distance operation of the blood vessel and the lesion is performed based on the blood vessel surface mesh vertex coordinate set and the lesion surface mesh vertex coordinate set to determine whether there is an intersecting surface between the blood vessel and the lesion, that is, to determine whether the blood vessel has invaded the lesion. In other words, to determine whether there is a blood vessel in the lesion. If there is a blood vessel, it is necessary to determine the vascular branch of the blood vessel invading the lesion and mark the vascular branch accordingly, for example, marking it in red, and then displaying it on the laparoscopic image of the current view to remind the doctor that the vascular branch is in the lesion, and to pay attention to the blood vessel when resecting the lesion, and to perform a hemostasis operation.

[0138] Specifically, based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, it is determined whether there is an intersecting surface between the blood vessel and the lesion, which can be determined in the following three ways.

[0139] Method 1: Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, the intersection operation in Boolean operation is used to perform surface intersection operation on the blood vessel and the lesion. If there is no operation result, it is determined that there is no intersection surface between the blood vessel and the lesion; if there is an operation result, it is determined that there is an intersection surface between the blood vessel and the lesion.

[0140] In one embodiment, the blood vessel branches to be marked may be determined according to the calculation results, for example, Figure 4 As shown, the shaded part in the figure is the calculated intersection result, and this part of the blood vessel is marked in red.

[0141] In the embodiment of the present application, the intersection operation in the Boolean operation can be used to directly calculate the blood vessel branches to be marked without first determining whether there is an intersection, which can improve the calculation efficiency.

[0142] like Figure 5 As shown, it is a schematic diagram of a process of determining a blood vessel branch to be marked provided in an embodiment of the present application.

[0143] S501, based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, using the intersection operation in Boolean operation to perform surface intersection operation on the blood vessel and the lesion;

[0144] S502, determine whether there is a calculation result, if yes, execute S503, otherwise execute S507;

[0145] S503, determine that the lesion invades the blood vessel;

[0146] S504, determining the blood vessel branches to be marked according to the calculation results;

[0147] S505, marking the blood vessel branches in red;

[0148] S506, displaying the blood vessel branches marked in red on the laparoscopic image of the current view;

[0149] S507. Determine that the lesion has not invaded the blood vessels.

[0150] Method 2: for each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculate a first distance between the difference between the blood vessel surface mesh vertex coordinate and each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set; determine a first minimum distance among the calculated first distances; use the maximum value among the determined multiple first minimum distances as a first distance calculation result; if the first distance calculation result is less than or equal to a second preset threshold, determine that the blood vessel and the lesion have an intersecting surface; if the first distance calculation result is greater than the first preset threshold, determine that the blood vessel and the lesion do not have an intersecting surface.

[0151] For example, the vertex coordinate set of the lesion surface mesh is {A1, A2, A3}, and the vertex coordinate set of the blood vessel surface mesh is {B1, B2, B3}. The distance d11 between the three-dimensional coordinate A1 and the three-dimensional coordinate B1, the distance d12 between the three-dimensional coordinate A1 and the three-dimensional coordinate B2, and the distance d13 between the three-dimensional coordinate A1 and the three-dimensional coordinate B3 are calculated; the distance d21 between the three-dimensional coordinate A2 and the three-dimensional coordinate B1, the distance d22 between the three-dimensional coordinate A2 and the three-dimensional coordinate B2, and the distance d23 between the three-dimensional coordinate A2 and the three-dimensional coordinate B3 are calculated; the distance d31 between the three-dimensional coordinate A3 and the three-dimensional coordinate B1, the distance d32 between the three-dimensional coordinate A3 and the three-dimensional coordinate B2, and the distance d33 between the three-dimensional coordinate A1 and the three-dimensional coordinate B3 are calculated. The minimum value determined from the distances d11, d12 and d13 is d11, the minimum value determined from the distances d21, d22 and d23 is d21, the minimum value determined from the distances d31, d32 and d33 is d32, and then the maximum value d11 is selected from the distances d11, d21 and d32. Finally, d11 is compared with the first preset threshold. If d11 is less than or equal to the second preset threshold, it is determined that there is an intersecting surface between the blood vessel and the lesion. If d11 is greater than the second preset threshold, it is determined that there is no intersecting surface between the blood vessel and the lesion.

[0152] After determining that there is an intersecting surface between the blood vessel and the lesion, the intersection operation of Boolean operation is used. Based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set, the intersection operation of Boolean operation is used to perform surface intersection operation on the lesion and the blood vessel, and the blood vessel intersecting with the lesion is determined according to the operation result. For details, please refer to Figure 4 , I will not go into details here.

[0153] The method for determining the blood vessel branches to be marked provided in the second method first determines whether the lesion invades the blood vessel based on the distance. If it is determined that the lesion invades the blood vessel, the invaded blood vessel branch is determined based on the Boolean operation of the three-dimensional grid.

[0154] like Figure 6 , which is a flow chart of another method for determining a blood vessel branch to be marked provided in an embodiment of the present application.

[0155] S601, calculating the distance between each lesion surface mesh vertex coordinate and each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set;

[0156] S602, determining the minimum distance among the calculated distances;

[0157] S603, taking the maximum value among the determined multiple minimum distances as the first distance calculation result;

[0158] S604, determining whether the first distance calculation result is less than or equal to a first preset threshold, if so, executing S605, otherwise executing S610;

[0159] S605, determining that there is an intersecting surface between the blood vessel and the lesion;

[0160] S606, based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set, using the intersection operation in Boolean operation to perform surface intersection operation on the lesion and the blood vessel;

[0161] S607, determining the blood vessel branches to be marked according to the calculation results;

[0162] S608, marking the blood vessel branches in red;

[0163] S609, displaying the blood vessel branches marked in red on the laparoscopic image of the current view;

[0164] S610: Determine whether there is no intersecting surface between the blood vessel and the lesion.

[0165] Method three: based on the blood vessel surface mesh vertex coordinate set, use the mean curvature flow algorithm to obtain the centerline coordinates of the centerline of the blood vessel; for each centerline coordinate, calculate the second distance between the centerline coordinate and each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set; determine the second minimum distance in the calculated second distances; use the maximum value in the determined second minimum distances as the second distance operation result; if the second distance operation result is less than or equal to a third preset threshold, it is determined that the blood vessel and the lesion have an intersecting surface; if the second distance operation result is greater than the third preset threshold, it is determined that the blood vessel and the lesion do not have an intersecting surface.

[0166] The third preset threshold here may be the radius of the blood vessel.

[0167] The example of calculating the second distance operation result can refer to the example of calculating the distance of the first distance operation result, which will not be repeated here.

[0168] After determining that there is an intersecting surface between the blood vessel and the lesion in the above manner, the centerline coordinates corresponding to the minimum value in the determined second minimum distance are used as the target centerline coordinates, and then the centerline branch where the target centerline coordinates are located is determined based on the target centerline coordinates and the adjacent neighborhood of the point on the centerline; the blood vessel corresponding to the determined centerline branch is used as the blood vessel branch to be marked.

[0169] In the embodiment of the present application, the target centerline coordinates are the centerline coordinates that are closest to the lesion. Since it has been determined that the blood vessel and the lesion have an intersecting surface, the blood vessel corresponding to the centerline branch where the centerline coordinates closest to the lesion are located is the blood vessel that intersects with the lesion.

[0170] Specifically, to determine the center line branch where the target center line coordinates are located, the first center line coordinate and the second center line coordinate adjacent to the target center line coordinate can be determined on the center line; in the first direction, taking the first center line coordinate as the starting point, the first number of adjacent center line coordinates is determined in sequence, and in the second direction, taking the second center line coordinate as the starting point, the second number of adjacent center line coordinates is determined in sequence; if the first number is greater than a fourth preset threshold, and the second number is greater than the fourth preset threshold, the center line between the center line coordinate corresponding to the first number and the center line coordinate corresponding to the second number is used as the center line branch.

[0171] Bi Tu, such as Figure 7 As shown, the blood vessel includes four blood vessel branches, namely blood vessel branch L1, blood vessel branch L2, blood vessel branch L3, blood vessel branch L4 and blood vessel branch L5, wherein the centerline coordinate e is the target centerline coordinate, Figure 7It can be seen from the figure that the center line coordinates adjacent to the center line coordinate e are the center line coordinate e11 and the center line e21. With the center line e11 as the starting point, in the first direction, it is determined that the number of center line coordinates adjacent to the center line coordinate e11 is 2, namely, the center line coordinate e and the center line coordinate e12. Then, it is determined that the number of center line coordinates adjacent to the center line coordinate e12 is 2, namely, the center line coordinate e11 and the center line coordinate e13. Then, it is determined that the number of center line coordinates adjacent to the center line coordinate e13 is 3, namely, the center line coordinate e12, the center line coordinate e31 and the center line coordinate e41. The number 3 is greater than the fourth preset threshold 2, and the center point coordinate e13 is determined to be the first end of the blood vessel branch to be marked. point; in the second direction, determine that the number of center line coordinates adjacent to the center line coordinate e21 is 2, namely, the center line coordinate e and the center line coordinate e22, then determine that the number of center line coordinates adjacent to the center line coordinate e22 is 2, namely, the center line coordinate e21 and the center line coordinate e23, then determine that the number of center line coordinates adjacent to the center line coordinate e23 is 2, namely, the center line coordinate e22 and the center line coordinate e24, then determine that the number of center line coordinates adjacent to the center line coordinate e24 is 3, namely, the center line coordinate e23, the center line coordinate e51 and the center line coordinate e61, the number 3 is greater than the fourth preset threshold 2, then determine that the center point coordinate e24 is the second endpoint of the blood vessel branch to be marked. After determining the first endpoint and the second endpoint, the blood vessel between the first endpoint and the second endpoint is taken as the blood vessel branch to be marked, that is, Figure 7 The vascular branch L1.

[0172] like Figure 8 , which is a flow chart of another method for determining a blood vessel branch to be marked provided in an embodiment of the present application.

[0173] S801, based on the mesh vertex coordinate set on the blood vessel surface, using the mean curvature flow algorithm to obtain the centerline coordinates of the centerline of the blood vessel;

[0174] S802, calculating the distance between each center line coordinate and each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set;

[0175] S803, determining the minimum distance among the calculated distances;

[0176] S804, taking the maximum value among the determined multiple minimum distances as the second distance calculation result;

[0177] S805, determining whether the second distance calculation result is less than or equal to a third preset threshold, if so, executing S806, otherwise executing S810;

[0178] S806, determining that there is an intersecting surface between the blood vessel and the lesion;

[0179] S807, taking the center line coordinates corresponding to the minimum value in the determined minimum distance as the target center line coordinates;

[0180] S808, determining a centerline branch where the target centerline coordinates are located according to the target centerline coordinates and the adjacent neighborhood of the point on the centerline;

[0181] S809, taking the blood vessel corresponding to the centerline branch as the blood vessel branch to be marked;

[0182] S810: Determine that there is no intersecting surface between the blood vessel and the lesion.

[0183] In the embodiment of the present application, since the number of centerline coordinates is much smaller than the number of mesh vertex coordinates on the blood vessel surface, the centerline coordinates and the mesh vertex coordinates on the lesion surface are used to determine whether there is an intersecting surface between the blood vessel and the lesion, which can reduce the amount of calculation and improve the display efficiency of the blood vessel.

[0184] In one embodiment, after determining that there is an intersecting surface between the blood vessel and the lesion according to the centerline coordinates, the blood vessel branch to be marked can also be determined by Boolean operation. For details, refer to the method in the second method, and the repeated parts will not be repeated.

[0185] In one embodiment, after determining the distance between each lesion surface mesh vertex coordinate and each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set in method 2, the blood vessel branch to be marked can also be determined by means of centerline coordinates. For details, please refer to the method in method 3, and the repeated parts will not be repeated.

[0186] like Fig. 9 FIG. 1 is a schematic diagram of a determined centerline branch provided in an embodiment of the present application. Fig. 9 The centerline branch marked in black is the centerline branch corresponding to the blood vessel branch to be marked.

[0187] like Fig.10 As shown, a determined blood vessel branch to be marked is provided in an embodiment of the present application. After the blood vessel branch is marked with a corresponding color, it can be displayed in a laparoscopic image.

[0188] In a specific implementation, in order to facilitate doctors to view the distance between the blood vessels and the tumor surface, the laparoscopic augmented reality surgery navigation method provided in the embodiment of the present application can also mark the distance from the blood vessel branches located in the safe area to the tumor surface.

[0189] like Fig.11 As shown, it is a complete flow chart of a laparoscopic augmented reality surgery navigation method provided by an embodiment of the present application;

[0190] S1201, based on the registered visceral three-dimensional model, obtaining a lesion surface mesh vertex coordinate set and a blood vessel surface mesh vertex coordinate set;

[0191] S1202, performing a surface intersection operation or a distance operation on the blood vessel and the lesion based on the mesh vertex coordinate set of the blood vessel surface figure and the mesh vertex coordinate set of the lesion surface;

[0192] S1203, judging whether the blood vessel and the lesion have intersecting surfaces according to the calculation result, if so, executing S1204, otherwise, executing S1206;

[0193] S1204, determining a first blood vessel branch to be marked;

[0194] S1205, after marking the first blood vessel branch to be marked, display it on the laparoscopic image;

[0195] S1206, obtaining a safe surface mesh vertex coordinate set for removing the lesion according to the mesh vertex coordinates of each lesion surface;

[0196] S1207, performing a surface intersection operation or a distance operation on the blood vessel and the safety area according to the safety surface mesh vertex coordinate set and the blood vessel surface vertex coordinate set;

[0197] S1208, judging whether there is an intersecting surface between the blood vessel and the safety area according to the calculation result, if so, executing S1209, otherwise executing S1211;

[0198] S1209, determining a second blood vessel branch to be marked;

[0199] S1210, after marking the second vascular branch to be marked, display it on the laparoscopic image;

[0200] S1211. Determine that there are no blood vessels in the safe area.

[0201] Based on the same inventive concept, the principle of solving the problem by the laparoscopic augmented reality surgical navigation device provided in the embodiment of the present application is similar to the principle of solving the problem by the above method, and the repeated parts will not be repeated.

[0202] In one embodiment, the processor 101 calculates the safety surface mesh vertex coordinate set in the following manner:

[0203] For each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculating the outer normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangular face to which the lesion surface mesh vertex coordinate belongs; multiplying the outer normal coordinate and a first preset threshold, adding the product to the lesion surface mesh vertex coordinate to obtain the safety surface mesh vertex coordinate;

[0204] The obtained safety surface mesh vertex coordinates are used as the safety surface mesh vertex coordinate set.

[0205] In one embodiment, the processor 101 is specifically configured to:

[0206] Based on the coordinates of each blood vessel surface mesh vertex in the blood vessel surface mesh vertex coordinate set and the coordinates of each target surface mesh vertex in the target surface mesh vertex coordinate set, using an intersection operation in a Boolean operation to perform a surface intersection operation on the blood vessel and the diseased target area;

[0207] If there is an operation result, it is determined that there is an intersection surface between the blood vessel and the target area.

[0208] In one embodiment, the processor 101 is specifically configured to:

[0209] For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculating a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set; and determining a first minimum distance among the calculated first distances;

[0210] Taking the maximum value among the determined first minimum distances as the first distance calculation result;

[0211] If the first distance calculation result is less than or equal to a second preset threshold, it is determined that there is an intersection surface between the blood vessel and the target area.

[0212] In one embodiment, the processor 101 is specifically configured to:

[0213] Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain the centerline coordinates of the blood vessel centerline;

[0214] For each center line coordinate: calculating a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, and determining a second minimum distance among the calculated second distances;

[0215] The maximum value among the determined second minimum distances is taken as the second distance calculation result;

[0216] If the second operation result is less than or equal to a third preset threshold, it is determined that there is an intersecting surface between the blood vessel and the target area.

[0217] In one embodiment, the processor 101 is specifically configured to:

[0218] The center line coordinate corresponding to the minimum value in the determined second minimum distance is used as the target center line coordinate;

[0219] Determine a centerline branch corresponding to the target centerline coordinates according to the target centerline coordinates and adjacent neighborhoods of points on the centerline;

[0220] The blood vessel corresponding to the centerline branch is used as the blood vessel branch to be marked.

[0221] In one embodiment, the processor 101 is specifically configured to:

[0222] On the center line, determining a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate;

[0223] In a first direction, taking the first center line coordinate as a starting point, sequentially determine first numbers of adjacent center line coordinates; and in a second direction, taking the second center line coordinate as a starting point, sequentially determine second numbers of adjacent center line coordinates;

[0224] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, the center line between the center line coordinates corresponding to the first number and the center line coordinates corresponding to the second number is used as the center line branch.

[0225] In one embodiment, the processor 101 is further configured to:

[0226] Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, performing a surface intersection operation or a distance operation on the blood vessel and the lesion to determine that there is no intersection surface between the blood vessel and the lesion;

[0227] Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the lesion safe resection area to determine that there is an intersection surface between the blood vessel and the lesion safe switching area.

[0228] Based on the same inventive concept, the embodiment of the present application also provides a laparoscopic augmented reality surgical navigation device. The principle of solving the problem by the device is similar to the principle of solving the problem by the above method, and the repeated parts will not be repeated.

[0229] like Fig.12 As shown, a laparoscopic augmented reality surgical navigation device provided in an embodiment of the present application includes:

[0230] An acquisition module 1301 is used to acquire a lesion surface mesh vertex coordinate set and a blood vessel surface mesh vertex coordinate set based on the registered visceral three-dimensional model;

[0231] The operation module 1302 is used to perform a surface intersection operation or a distance operation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or the safety surface mesh vertex coordinate set corresponding to the lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set;

[0232] The marking display module 1303 is used to determine the vascular branches to be marked according to the calculation results, and mark the vascular branches to be marked accordingly and then display them on the laparoscopic image of the current view.

[0233] In one embodiment, the operation module 1302 is specifically used to:

[0234] For each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculating the outer normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangular face to which the lesion surface mesh vertex coordinate belongs; multiplying the outer normal coordinate and a first preset threshold, adding the product to the lesion surface mesh vertex coordinate to obtain the safety surface mesh vertex coordinate;

[0235] The obtained safety surface mesh vertex coordinates are used as the safety surface mesh vertex coordinate set.

[0236] In one embodiment, the operation module 1302 is specifically used to:

[0237] Based on the coordinates of each blood vessel surface mesh vertex in the blood vessel surface mesh vertex coordinate set and the coordinates of each target surface mesh vertex in the target surface mesh vertex coordinate set, using an intersection operation in a Boolean operation to perform a surface intersection operation on the blood vessel and the diseased target area;

[0238] If there is an operation result, it is determined that there is an intersection surface between the blood vessel and the target area.

[0239] In one embodiment, the operation module 1302 is specifically used to:

[0240] For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculating a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set; and determining a first minimum distance among the calculated first distances;

[0241] Taking the maximum value among the determined first minimum distances as the first distance calculation result;

[0242] If the first distance calculation result is less than or equal to a second preset threshold, it is determined that there is an intersection surface between the blood vessel and the target area.

[0243] In one embodiment, the operation module 1302 is specifically used to:

[0244] Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain the centerline coordinates of the blood vessel centerline;

[0245] For each center line coordinate: calculating a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, and determining a second minimum distance among the calculated second distances;

[0246] The maximum value among the determined second minimum distances is taken as the second distance calculation result;

[0247] If the second operation result is less than or equal to a third preset threshold, it is determined that there is an intersecting surface between the blood vessel and the target area.

[0248] In one embodiment, the mark display module 1303 is specifically used for:

[0249] The center line coordinate corresponding to the minimum value in the determined second minimum distance is used as the target center line coordinate;

[0250] Determine a centerline branch corresponding to the target centerline coordinates according to the target centerline coordinates and the adjacent neighborhood of the point on the centerline;

[0251] The blood vessel corresponding to the centerline branch is used as the blood vessel branch to be marked.

[0252] In one embodiment, the mark display module 1303 is specifically used for:

[0253] On the center line, determining a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate;

[0254] In a first direction, taking the first center line coordinate as a starting point, sequentially determine first numbers of adjacent center line coordinates; and in a second direction, taking the second center line coordinate as a starting point, sequentially determine second numbers of adjacent center line coordinates;

[0255] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, the center line between the center line coordinates corresponding to the first number and the center line coordinates corresponding to the second number is used as the center line branch.

[0256] In one embodiment, the operation module 1302 is specifically used to:

[0257] Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, performing a surface intersection operation or a distance operation on the blood vessel and the lesion to determine that there is no intersection surface between the blood vessel and the lesion;

[0258] Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the lesion safe resection area to determine that there is an intersection surface between the blood vessel and the lesion safe switching area.

[0259] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0260] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0261] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0262] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0263] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A laparoscopic augmented reality surgical navigation method, characterized in that: The method includes: Based on the registered visceral three-dimensional model, a mesh vertex coordinate set of the lesion surface and a mesh vertex coordinate set of the blood vessel surface are obtained; Based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the target area to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set; The blood vessel branches to be marked are determined according to the calculation results, and the blood vessel branches to be marked are marked accordingly and then displayed on the laparoscopic image of the current view.

2. The method according to claim 1, characterized in that The safe surface mesh vertex coordinate set is calculated by: For each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set: calculating the outer normal coordinate corresponding to the lesion surface mesh vertex coordinate according to the triangular face to which the lesion surface mesh vertex coordinate belongs; multiplying the outer normal coordinate and a first preset threshold, adding the product to the lesion surface mesh vertex coordinate to obtain the safety surface mesh vertex coordinate; The obtained safety surface mesh vertex coordinates are used as the safety surface mesh vertex coordinate set.

3. The method according to claim 1, characterized in that The performing of a surface intersection operation on the blood vessel and the target region based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersection surface between the blood vessel and the target region includes: Based on the coordinates of each blood vessel surface mesh vertex in the blood vessel surface mesh vertex coordinate set and the coordinates of each target surface mesh vertex in the target surface mesh vertex coordinate set, using an intersection operation in a Boolean operation to perform a surface intersection operation on the blood vessel and the diseased target area; If there is an operation result, it is determined that there is an intersection surface between the blood vessel and the target area.

4. The method according to claim 1, characterized in that The performing of distance calculation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersecting surface between the blood vessel and the target area includes: For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: calculating a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set; and determining a first minimum distance among the calculated first distances; Taking the maximum value among the determined first minimum distances as the first distance calculation result; If the first distance calculation result is less than or equal to a second preset threshold, it is determined that there is an intersection surface between the blood vessel and the target area.

5. The method according to claim 1, characterized in that The performing of distance calculation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set to determine that there is an intersecting surface between the blood vessel and the target area includes: Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain the centerline coordinates of the blood vessel centerline; For each center line coordinate: calculating a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, and determining a second minimum distance among the calculated second distances; The maximum value among the determined second minimum distances is taken as the second distance calculation result; If the second operation result is less than or equal to a third preset threshold, it is determined that there is an intersecting surface between the blood vessel and the target area.

6. The method according to claim 5, characterized in that The step of determining the blood vessel branch to be marked according to the calculation result includes: The center line coordinate corresponding to the minimum value in the determined second minimum distance is used as the target center line coordinate; Determine a centerline branch corresponding to the target centerline coordinates according to the target centerline coordinates and the adjacent neighborhood of the point on the centerline; The blood vessel corresponding to the centerline branch is used as the blood vessel branch to be marked.

7. The method according to claim 6, characterized in that The step of determining the center line branch where the target center line coordinates are located according to the target center line coordinates and the adjacent neighborhood of the point on the center line includes: On the center line, determining a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate; In a first direction, taking the first center line coordinate as a starting point, sequentially determine first numbers of adjacent center line coordinates; and in a second direction, taking the second center line coordinate as a starting point, sequentially determine second numbers of adjacent center line coordinates; If the first number is greater than a fourth preset threshold, and the second number is greater than the fourth preset threshold, the center line between the center line coordinates corresponding to the first number and the center line coordinates corresponding to the second number is used as the center line branch.

8. The method according to any one of claims 1 to 7, characterized in that: The method further includes: Based on the mesh vertex coordinate set of the blood vessel surface and the mesh vertex coordinate set of the lesion surface, performing a surface intersection operation or a distance operation on the blood vessel and the lesion to determine that there is no intersection surface between the blood vessel and the lesion; Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, a surface intersection operation or a distance operation is performed on the blood vessel and the lesion safe resection area to determine that there is an intersection surface between the blood vessel and the lesion safe switching area.

9. A laparoscopic augmented reality fusion display device, characterized in that: The device includes a processor and a display screen; The processor is used to obtain a lesion surface mesh vertex coordinate set and a blood vessel surface mesh vertex coordinate set based on the registered visceral three-dimensional model; based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, perform a surface intersection operation or a distance operation on the blood vessel and the target area to determine that there is an intersection surface between the blood vessel and the target area, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set; determine a blood vessel branch to be marked based on the operation result, and mark the blood vessel branch to be marked accordingly; The display screen is used for laparoscopic augmented reality fusion display of the marked vascular branches.

10. A laparoscopic augmented reality fusion display device, characterized in that: include: An acquisition module, used for acquiring a mesh vertex coordinate set of a lesion surface and a mesh vertex coordinate set of a blood vessel surface based on the registered visceral three-dimensional model; A calculation module, for performing a surface intersection calculation or a distance calculation on the blood vessel and the target area based on the blood vessel surface mesh vertex coordinate set and the target surface mesh vertex coordinate set, to determine that the blood vessel and the target area have an intersection surface, wherein the target surface mesh vertex coordinate set includes the lesion surface mesh vertex coordinate set, or a safety surface mesh vertex coordinate set corresponding to a lesion safe resection area; the safety surface mesh vertex coordinate set is calculated based on the lesion surface mesh vertex coordinate set and the blood vessel surface mesh vertex coordinate set; Marking display module: used to determine the vascular branches to be marked according to the calculation results, and display the vascular branches to be marked on the laparoscopic image of the current view after marking them accordingly.

Citation Information

Patent Citations

  • Visualization of depth and position of blood vessels and robot guided visualization of blood vessel cross section

    CN105979900A

  • Imaging apparatus for imaging a first object within a second object

    CN106456254A

  • An augmented reality auxiliary operation method and system

    CN109360219A

  • Coronary artery fractional flow reserve obtaining method and device, computer equipment and storage medium

    CN113499090A

  • Augmented reality based cannula guide system for interventional cardiology procedures and method thereof

    KR101843992B1

Cited By

  • Blood vessel endpoint determination method and device, storage medium and electronic equipment

    CN121544525A