A laparoscopic augmented reality surgical navigation method, device and apparatus

Abstract

The application discloses a laparoscope augmented reality surgery navigation method, device and apparatus, and the method comprises the following steps: firstly, based on the registered internal organ three-dimensional model, obtaining a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set; then, based on the obtained coordinate set, performing surface intersection operation or distance operation; after determining that the blood vessel and the target region have an intersection surface, determining the blood vessel branch to be marked according to the operation result, marking the blood vessel branch, and displaying the blood vessel branch on the current view of the laparoscope image; and the target region can be the region where the lesion is located or the safe resection region for resecting the lesion. Since the blood vessel in the target region can be marked and displayed on the laparoscope image, the blood vessel in the target region is displayed more intuitively, the navigation auxiliary function is provided for the doctor in the surgery process, and therefore, the accuracy of the augmented reality surgery navigation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical applications, in particular to a laparoscopic augmented reality surgery navigation method, device and apparatus. BACKGROUND

[0002] Laparoscopic hepatectomy has the advantages of small incision, less pain, fast recovery and small scar, and has become the development trend of current surgery.

[0003] At present, in the laparoscopic augmented reality navigation technology, it is impossible to specially display the blood vessels in a certain specific area, for example, the blood vessels inside the lesion or the blood vessels in the safe resection area of the lesion. Whether there are blood vessels in the lesion or the safe resection area of the lesion can only be judged by the experience of the doctor. The accuracy of this judgment method is low, and the blood vessel display is not intuitive enough. SUMMARY

[0004] The present application provides a laparoscopic augmented reality surgery navigation method, device and apparatus to solve the problem of insufficient intuitive blood vessel display in the laparoscopic augmented reality surgery navigation technology in the prior art.

[0005] In a first aspect, the present application provides a laparoscopic augmented reality surgery navigation method, which comprises:

[0006] Based on the registered internal organ three-dimensional model, a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set are obtained;

[0007] Based on the blood vessel surface grid vertex coordinate set and a target surface grid vertex coordinate set, surface intersection operation or distance operation is performed on the blood vessels and the target area to determine that the blood vessels and the target area have an intersection surface, wherein the target surface grid vertex coordinate set includes the lesion surface grid vertex coordinate set or a safety surface grid vertex coordinate set corresponding to the safe resection area of the lesion; the safety surface grid vertex coordinate set is calculated according to the lesion surface grid vertex coordinate set and the blood vessel surface grid vertex coordinate set;

[0008] According to the operation result, the blood vessel branches to be marked are determined, and the blood vessel branches to be marked are marked and displayed on the current view of the laparoscopic image.

[0009] In a possible implementation, the safety surface grid vertex coordinate set is calculated by the following method:

[0010] For each of the lesion surface grid vertex coordinates in the set of the lesion surface grid vertex coordinates: according to a triangular facet to which the lesion surface grid vertex coordinate belongs, an outer normal coordinate corresponding to the lesion surface grid vertex coordinate is calculated; the outer normal coordinate is multiplied by a first preset threshold value, and the product is added to the lesion surface grid vertex coordinate to obtain a safe surface grid vertex coordinate;

[0011] The obtained safe surface grid vertex coordinates are taken as the set of the safe surface grid vertex coordinates.

[0012] In a possible implementation, the surface intersection operation is performed on the blood vessel and the target region based on the set of the blood vessel surface grid vertex coordinates and the set of the target surface grid vertex coordinates to determine that the blood vessel and the target region have an intersection surface, including:

[0013] An intersection operation in Boolean operation is adopted based on each of the blood vessel surface grid vertex coordinates in the set of the blood vessel surface grid vertex coordinates and each of the target surface grid vertex coordinates in the set of the target surface vertex coordinates to perform the surface intersection operation on the blood vessel and the target region.

[0014] If the operation result exists, it is determined that the blood vessel and the target region have an intersection surface.

[0015] In a possible implementation, the distance operation is performed on the blood vessel and the target region based on the set of the blood vessel surface grid vertex coordinates and the set of the target surface grid vertex coordinates to determine that the blood vessel and the target region have an intersection surface, including:

[0016] For each of the blood vessel surface grid vertex coordinates in the set of the blood vessel surface grid vertex coordinates: a first distance between the blood vessel surface grid vertex coordinate and each of the target surface grid vertex coordinates in the set of the target surface grid vertex coordinates is calculated; and a first minimum distance in the calculated first distances is determined.

[0017] A maximum value in the determined first minimum distances is taken as a first distance operation result.

[0018] If the first distance operation result is less than or equal to a second preset threshold value, it is determined that the blood vessel and the target region have an intersection surface.

[0019] In a possible implementation, the distance operation is performed on the blood vessel and the target region based on the set of the blood vessel surface grid vertex coordinates and the set of the target surface grid vertex coordinates to determine that the blood vessel and the target region have an intersection surface, including:

[0020] Based on the set of the blood vessel surface grid vertex coordinates, a centerline coordinate of a centerline of the blood vessel is obtained by using an average curvature flow algorithm.

[0021] For each center line coordinate: calculate a second distance between the center line coordinate and each target surface mesh vertex coordinate in the set of target surface mesh vertex coordinates, determine a second minimum distance in the calculated second distances;

[0022] take a maximum value in the determined second minimum distances as a second distance operation result;

[0023] If the second operation result is less than or equal to a third preset threshold, it is determined that the blood vessel and the target region exist an intersection surface.

[0024] In a possible implementation, the operation of determining the blood vessel branch to be marked according to the operation result comprises:

[0025] take a center line coordinate corresponding to a minimum value in the determined second minimum distances as a target center line coordinate;

[0026] determine a center line branch corresponding to the target center line coordinate according to the target center line coordinate and a neighboring neighborhood of a point on the center line;

[0027] take a blood vessel corresponding to the center line branch as the blood vessel branch to be marked.

[0028] In a possible implementation, the operation of determining the center line branch corresponding to the target center line coordinate according to the target center line coordinate and the neighboring neighborhood of the point on the center line comprises:

[0029] determine a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate on the center line;

[0030] determine a first number of adjacent center line coordinates in a first direction starting from the first center line coordinate, and determine a second number of adjacent center line coordinates in a second direction starting from the second center line coordinate;

[0031] if the first number is greater than a fourth preset threshold and the second number is greater than the fourth preset threshold, take a center line between a center line coordinate corresponding to the first number and a center line coordinate corresponding to the second number as the center line branch.

[0032] In a possible implementation, the method further comprises:

[0033] perform a surface intersection operation or a distance operation on the blood vessel and the lesion based on the set of blood vessel surface mesh vertex coordinates and the set of lesion surface mesh vertex coordinates, and determine that the blood vessel and the lesion do not exist an intersection surface;

[0034] The processor is configured to perform surface intersection operation or distance operation on the blood vessel and the lesion safe resection region based on the blood vessel surface grid vertex coordinate set and the safe surface grid vertex coordinate set, and determine that the blood vessel and the lesion safe resection region have an intersection surface.

[0035] In a second aspect, the embodiments of the present application provide a laparoscope augmented reality fusion display device, the device comprising a processor and a display screen;

[0036] The processor is configured to obtain a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set based on the registered internal organ three-dimensional model, perform surface intersection operation or distance operation on the blood vessel and a target region based on the blood vessel surface grid vertex coordinate set and a target surface grid vertex coordinate set, and determine that the blood vessel and the target region have an intersection surface, wherein the target surface grid vertex coordinate set comprises the lesion surface grid vertex coordinate set or a safe surface grid vertex coordinate set corresponding to a lesion safe resection region; the safe surface grid vertex coordinate set is calculated based on the lesion surface grid vertex coordinate set and the blood vessel surface grid vertex coordinate set; and a blood vessel branch to be marked is determined according to the operation result, and the blood vessel branch to be marked is marked accordingly.

[0037] The display screen is configured to display the marked blood vessel branch in laparoscope augmented reality fusion.

[0038] In a possible implementation, the processor calculates the safe surface grid vertex coordinate set in the following manner:

[0039] For each lesion surface grid vertex coordinate in the lesion surface grid vertex coordinate set, an external normal coordinate corresponding to the lesion surface grid vertex coordinate is calculated according to a triangular facet to which the lesion surface grid vertex coordinate belongs; a multiplication operation is performed on the external normal coordinate and a first preset threshold value; and a product of the multiplication operation is added to the lesion surface grid vertex coordinate to obtain a safe surface grid vertex coordinate.

[0040] The obtained safe surface grid vertex coordinate is taken as the safe surface grid vertex coordinate set.

[0041] In a possible implementation, the processor is specifically configured to:

[0042] The processor is configured to perform surface intersection operation on the blood vessel and the target region based on each blood vessel surface grid vertex coordinate in the blood vessel surface grid vertex coordinate set and each target surface grid vertex coordinate in the target surface grid vertex coordinate set by using intersection operation in Boolean operation.

[0043] If there is an operation result, it is determined that the blood vessel and the target region have an intersection surface.

[0044] In a possible implementation, the processor is specifically configured to:

[0045] For each blood vessel surface mesh vertex coordinate in the set of blood vessel surface mesh vertex coordinates: calculate a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the set of target surface mesh vertex coordinates; determine a first minimum distance in the calculated first distances;

[0046] Take the maximum value in the determined first minimum distances as a first distance operation result;

[0047] If the first distance operation result is less than or equal to a second preset threshold, it is determined that the blood vessel and the target region exist an intersection surface.

[0048] In a possible implementation, the processor is specifically configured to:

[0049] Based on the set of blood vessel surface mesh vertex coordinates, a centerline coordinate of a centerline of the blood vessel is obtained by using an average curvature flow algorithm;

[0050] For each centerline coordinate: calculate a second distance between the centerline coordinate and each target surface mesh vertex coordinate in the set of target surface mesh vertex coordinates, and determine a second minimum distance in the calculated second distances;

[0051] Take the maximum value in the determined second minimum distances as a second distance operation result;

[0052] If the second operation result is less than or equal to a third preset threshold, it is determined that the blood vessel and the target region exist an intersection surface.

[0053] In a possible implementation, the processor is specifically configured to:

[0054] Take the centerline coordinate corresponding to the minimum value in the determined second minimum distances as a target centerline coordinate;

[0055] According to the target centerline coordinate and a neighboring neighborhood of a point on the centerline, a centerline branch in which the target centerline coordinate is located is determined;

[0056] Take the blood vessel corresponding to the centerline branch as a blood vessel branch to be marked.

[0057] In a possible implementation, the processor is specifically configured to:

[0058] On the centerline, a first centerline coordinate and a second centerline coordinate adjacent to the target centerline coordinate are determined;

[0059] In the first direction, a first number of adjacent centerline coordinates is determined successively starting from the first centerline coordinate, and in the second direction, a second number of adjacent centerline coordinates is determined successively starting from the second centerline coordinate.

[0060] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, a centerline between the centerline coordinate corresponding to the first number and the centerline coordinate corresponding to the second number is taken as the centerline branch.

[0061] In a possible implementation, the processor is further configured to:

[0062] Based on the set of vessel surface mesh vertex coordinates and the set of lesion surface mesh vertex coordinates, surface intersection operation or distance operation is performed on the vessel and the lesion to determine that the vessel and the lesion do not have intersecting surfaces.

[0063] Based on the set of vessel surface mesh vertex coordinates and the set of safe surface mesh vertex coordinates, surface intersection operation or distance operation is performed on the vessel and the lesion safe resection region to determine that the vessel and the lesion safe resection region have intersecting surfaces.

[0064] In a third aspect, an embodiment of the present application provides a laparoscope augmented reality fusion display device, comprising:

[0065] The acquisition module is configured to acquire a set of lesion surface mesh vertex coordinates and a set of vessel surface mesh vertex coordinates based on the registered internal organ three-dimensional model.

[0066] The operation module is configured to perform surface intersection operation or distance operation on the vessel and a target region based on the set of vessel surface mesh vertex coordinates and a set of target surface mesh vertex coordinates to determine that the vessel and the target region have intersecting surfaces, wherein the set of target surface mesh vertex coordinates includes the set of lesion surface mesh vertex coordinates or a set of safe surface mesh vertex coordinates corresponding to a lesion safe resection region; the set of safe surface mesh vertex coordinates is calculated according to the set of lesion surface mesh vertex coordinates and the set of vessel surface mesh vertex coordinates.

[0067] The marking display module is configured to determine a vessel branch to be marked according to the operation result, mark the vessel branch to be marked, and display the marked vessel branch on the current view of the laparoscope image.

[0068] In a possible implementation, the operation module is specifically configured to:

[0069] For each of the lesion surface grid vertex coordinates in the set of lesion surface grid vertex coordinates, an outer normal coordinate corresponding to the lesion surface grid vertex coordinate is calculated according to a triangular facet to which the lesion surface grid vertex coordinate belongs; the outer normal coordinate is multiplied by a first preset threshold, and the product is added to the lesion surface grid vertex coordinate to obtain a safe surface grid vertex coordinate;

[0070] The obtained safe surface grid vertex coordinates are taken as the set of safe surface grid vertex coordinates.

[0071] In a possible implementation, the operation module is specifically configured to:

[0072] Based on each of the blood vessel surface grid vertex coordinates in the set of blood vessel surface grid vertex coordinates and each of the target surface grid vertex coordinates in the set of target surface grid vertex coordinates, an intersection operation in a Boolean operation is adopted to perform surface intersection operation on the blood vessel and the target region.

[0073] If the operation result exists, it is determined that the blood vessel and the target region have an intersection surface.

[0074] In a possible implementation, the operation module is specifically configured to:

[0075] For each of the blood vessel surface grid vertex coordinates in the set of blood vessel surface grid vertex coordinates, a first distance between the blood vessel surface grid vertex coordinate and each of the target surface grid vertex coordinates in the set of target surface grid vertex coordinates is calculated; and a first minimum distance in the calculated first distances is determined.

[0076] A maximum value in the determined first minimum distances is taken as a first distance operation result.

[0077] If the first distance operation result is less than or equal to a second preset threshold, it is determined that the blood vessel and the target region have an intersection surface.

[0078] In a possible implementation, the operation module is specifically configured to:

[0079] Based on the set of blood vessel surface grid vertex coordinates, an average curvature flow algorithm is adopted to obtain a centerline coordinate of a centerline of the blood vessel.

[0080] For each of the centerline coordinates, a second distance between the centerline coordinate and each of the target surface grid vertex coordinates in the set of target surface grid vertex coordinates is calculated, and a second minimum distance in the calculated second distances is determined.

[0081] A maximum value in the determined second minimum distances is taken as a second distance operation result.

[0082] If the second operation result is less than or equal to a third preset threshold, it is determined that the blood vessel and the target region exist an intersection surface.

[0083] In a possible implementation, the marking display module is specifically configured to:

[0084] a minimum value in the determined second minimum distances is taken as a target centerline coordinate;

[0085] According to the target centerline coordinate and the adjacent neighborhood of the point on the centerline, a centerline branch in which the target centerline coordinate is located is determined.

[0086] The centerline branch corresponds to a blood vessel branch to be marked.

[0087] In a possible implementation, the marking display module is specifically configured to:

[0088] On the centerline, a first centerline coordinate and a second centerline coordinate adjacent to the target centerline coordinate are determined.

[0089] In a first direction, a first number of adjacent centerline coordinates is determined successively from the first centerline coordinate as a starting point, and in a second direction, a second number of adjacent centerline coordinates is determined successively from the second centerline coordinate as a starting point.

[0090] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, a centerline between a centerline coordinate corresponding to the first number and a centerline coordinate corresponding to the second number is taken as the centerline branch.

[0091] In a possible implementation, the operation module is specifically configured to:

[0092] Based on the blood vessel surface mesh vertex coordinate set and the lesion surface mesh vertex coordinate set, surface intersection operation or distance operation is performed on the blood vessel and the lesion, to determine that the blood vessel and the lesion do not exist an intersection surface.

[0093] Based on the blood vessel surface mesh vertex coordinate set and the safety surface mesh vertex coordinate set, surface intersection operation or distance operation is performed on the blood vessel and the lesion safety switch region, to determine that the blood vessel and the lesion safety switch region exist an intersection surface.

[0094] The present application has the following advantages:

[0095] The laparoscope augmented reality surgery navigation method, device and apparatus provided by the embodiment of the present application firstly obtains a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set based on a registered internal organ three-dimensional model, then performs surface intersection operation or distance operation based on the obtained coordinate set, determines the blood vessel branch to be marked according to the operation result after determining that the blood vessel and the target region have an intersection surface, and displays the blood vessel branch to be marked on the current view of the laparoscope image, wherein the target region can be a region where the lesion is located or a safe resection region for resecting the lesion. Since the laparoscope augmented reality surgery navigation method provided by the embodiment of the present application can mark the blood vessel in the target region and display the blood vessel on the laparoscope image, the blood vessel in the target region is displayed more intuitively, and the navigation auxiliary function is provided for the doctor in the surgery process, thereby improving the accuracy of the augmented reality surgery navigation. BRIEF DESCRIPTION OF DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0097] Figure 1 A structure schematic diagram of a laparoscope augmented reality surgery navigation system provided by the embodiment of the present application;

[0098] Figure 2 A flowchart of a laparoscope augmented reality surgery navigation method provided by the embodiment of the present application;

[0099] Figure 3 A schematic diagram of a liver three-dimensional model provided by the embodiment of the present application;

[0100] Figure 4 A schematic diagram of a blood vessel branch to be marked provided by the embodiment of the present application;

[0101] Figure 5 A flowchart of a method for determining the blood vessel branch to be marked provided by the embodiment of the present application;

[0102] Figure 6 A flowchart of another method for determining the blood vessel branch to be marked provided by the embodiment of the present application;

[0103] Figure 7 A structure schematic diagram of a blood vessel branch provided by the embodiment of the present application;

[0104] Figure 8 A flowchart of another method for determining the blood vessel branch to be marked provided by the embodiment of the present application;

[0105] Figure 9 A schematic diagram of a determined centerline branch provided for an embodiment of the present application;

[0106] Figure 10 A schematic diagram of a determined blood vessel branch to be marked provided for an embodiment of the present application;

[0107] Figure 11 A complete flowchart of a laparoscopic augmented reality surgery navigation method provided for an embodiment of the present application;

[0108] Figure 12 A structural schematic diagram of a laparoscopic augmented reality surgery navigation device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0109] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. The described embodiments are some embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0110] In the description of the embodiments of the present application, unless otherwise specified, “ / ” represents the meaning of or, for example, A / B can represent A or B; the “and / or” in the text only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, “multiple” means two or more than two.

[0111] Hereinafter, the terms “first” and “second” are only used for description purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include one or more features, and in the description of the embodiments of the present application, unless otherwise specified, the meaning of “multiple” is two or more than two.

[0112] In order to facilitate understanding of the laparoscopic augmented reality surgery navigation method, device and apparatus provided by the embodiments of the present application, some terms in the embodiments of the present application will be explained below to facilitate understanding by those skilled in the art.

[0113] (1) The laparoscopic augmented reality surgery navigation technology is the main form of the current laparoscopic augmented reality navigation, and the implementation mainly reconstructs the internal organ three-dimensional model before surgery from the CT / MRI medical image, realizes registration of the internal organ three-dimensional model before surgery and the laparoscopic image during surgery through the image registration technology, generates the internal organ three-dimensional model during surgery, and finally completes the augmented display of the intraoperative scene through the augmented reality display technology, so as to realize the image navigation.

[0114] (2) The augmented reality (AR) technology is a new technology of "seamless" integration of real world information and virtual world information, and is to simulate and then superimpose the virtual information applied to the real world through the computer and other scientific technologies, so that the virtual information is perceived by human senses, thereby achieving the sensory experience beyond reality. The real environment and the virtual object are real-time superimposed in the same picture or space at the same time. The augmented reality technology contains new technologies and new means such as multimedia, three-dimensional modeling, real-time video display and control, multi-sensor integration, real-time tracking and registration, scene fusion, etc.

[0115] The application scene of the laparoscopic augmented reality surgery navigation method provided by the embodiment of the application will be introduced below in combination with the drawings. As shown in the laparoscopic augmented reality surgery navigation system shown in FIG. 1, the laparoscopic augmented reality surgery navigation system includes a laparoscope 2 and a laparoscopic augmented reality surgery navigation device 3, wherein the laparoscopic augmented reality surgery navigation device 3 includes a processor 101 and a display screen 102, and the processor 101 is configured to perform the following steps. Figure 1

[0116] The processor 101 is configured to acquire a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set based on the registered internal organ three-dimensional model, perform surface intersection operation or distance operation on the blood vessel and the target region based on the blood vessel surface grid vertex coordinate set and a target surface grid vertex coordinate set, determine that the blood vessel and the target region exist intersection surface, wherein the target surface grid vertex coordinate set includes the lesion surface grid vertex coordinate set or a safety surface grid vertex coordinate set corresponding to a lesion safe resection region; the safety surface grid vertex coordinate set is calculated according to the lesion surface grid vertex coordinate set and the blood vessel surface grid vertex coordinate set; determine a blood vessel branch to be marked according to the operation result, and mark the blood vessel branch to be marked accordingly.

[0117] The display screen 102 is configured to display the marked blood vessel branch on the laparoscopic image of the current view.

[0118] Figure 1 ​In the diagram, 1 represents an internal organ. A preoperative three-dimensional model of internal organ 1 is obtained before surgery. During the operation of internal organ 1 using laparoscopy 2, relevant information of internal organ 1 is collected in real time using laparoscopy 2. The preoperative three-dimensional model of internal organ 1 and the relevant information collected during surgery are registered to obtain a registered three-dimensional model. Then, the model is overlaid on the laparoscopic image in the current view using enhanced display technology.

[0119] The methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of this application are not limited thereto.

[0120] like Figure 2 The flowchart of the laparoscopic augmented reality surgical navigation method according to an embodiment of this application is shown below, and the specific steps are as follows:

[0121] S201. Based on the registered 3D model of the internal organs, obtain the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh.

[0122] S202. Based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the target surface mesh, perform surface intersection or distance calculation on the blood vessel and the target region to determine that the blood vessel and the target region have intersecting surfaces. The vertex coordinate set of the target surface mesh includes the vertex coordinate set of the lesion surface mesh, or the vertex coordinate set of the safe surface mesh corresponding to the safe resection area of ​​the lesion. The vertex coordinate set of the safe surface mesh is calculated based on the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh.

[0123] S203. Determine the vascular branches to be marked based on the calculation results, and then display the vascular branches to be marked on the laparoscopic image in the current view after marking them accordingly.

[0124] The laparoscopic augmented reality surgical navigation method provided in this application first obtains the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh based on the registered 3D model of the internal organs. Then, based on the obtained coordinate sets, surface intersection or distance calculations are performed. After determining that there is an intersecting surface between the blood vessel and the target area, the blood vessel branches to be marked are determined according to the calculation results. After the blood vessel branches are marked accordingly, they are displayed on the laparoscopic image in the current view. Here, the target area can be the area where the lesion is located or the safe resection area for removing the lesion. Since the laparoscopic augmented reality surgical navigation method provided in this application can mark the blood vessels in the target area and display them on the laparoscopic image, the blood vessels in the target area are displayed more intuitively, providing navigation assistance to the doctor during the operation, thereby improving the accuracy of augmented reality surgical navigation.

[0125] The visceral 3D model used in this application embodiment is a registered visceral 3D model, which is obtained by registering the preoperative visceral 3D model with relevant information collected from the book.

[0126] Using a registered 3D model of internal organs can improve the accuracy of augmented reality surgical navigation, enabling augmented reality display of the intraoperative field of view and real-time navigation of the surgical process.

[0127] In specific implementation, the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh can be directly obtained from the registered 3D model of the internal organs. In this embodiment, the vertex coordinate set of the blood vessel surface mesh is of the same type, such as the vertex coordinate set of the portal vein. If the 3D model of the internal organs is a 3D model of the liver, the vertex coordinates of the blood vessel surface mesh can also be the vertex coordinates of the hepatic vein.

[0128] like Figure 3 The image shown is a partial schematic diagram of a three-dimensional model of the liver, which includes lesions (not shown), portal veins, and hepatic veins within the liver. Figure 3 As can be seen, the 3D model of the internal organs is composed of multiple triangular facets. Each vertex of a triangular facet represents the vertex coordinates of a surface mesh, which are 3D coordinates. This 3D model of the liver includes the vertex coordinates of the liver surface mesh, the portal vein surface mesh, the hepatic vein surface mesh, and the lesion surface mesh.

[0129] In one embodiment, when the target area is a safe resection area, the coordinates of the safe surface grid vertices corresponding to the safe resection area can be obtained from the set of lesion surface grid vertex coordinates.

[0130] Specifically, for each vertex coordinate of the lesion surface grid in the vertex coordinate set: calculate the outward normal coordinate corresponding to the vertex coordinate of the lesion surface grid according to the triangular facet to which the vertex coordinate of the lesion surface grid belongs; multiply the outward normal coordinate with a first preset threshold, add the product to the vertex coordinate of the lesion surface grid to obtain the vertex coordinate of the safe surface grid; use the obtained vertex coordinate of the safe surface grid as the set of vertex coordinates of the safe surface grid.

[0131] When calculating the external normal coordinates corresponding to the vertex coordinates of the lesion surface grid based on the triangular facet to which the vertex coordinates of the lesion surface grid belong, the triangular facet to which the vertex coordinates of the lesion surface grid belong is first determined. Then, the face normal vector corresponding to each triangular facet is calculated. The average value of the calculated face normal vectors is then calculated, and the resulting value is the external normal coordinate corresponding to the vertex coordinates of the lesion surface grid.

[0132] For example, the triangular patches to which the lesion surface grid vertex coordinate a belongs include triangular patch 1, triangular patch 2 and triangular patch 3, wherein the vertex coordinates of each triangular patch are known, the face normal vector 1 corresponding to the triangular patch 1 is calculated according to the three vertex coordinates corresponding to the triangular patch 1, the face normal vector 2 corresponding to the triangular patch 2 is calculated according to the three vertex coordinates corresponding to the triangular patch 2, the face normal vector 3 corresponding to the triangular patch 3 is calculated according to the three vertex coordinates corresponding to the triangular patch 3, and the average value of the face normal vector 1, the face normal vector 2 and the face normal vector 3 is calculated to obtain the result, which is the external normal vector corresponding to the lesion surface grid vertex coordinate a.

[0133] It should be noted that in the specific implementation, the three vertices of the triangular patch are usually marked in a clockwise order. Assuming that the coordinates of vertex 1 of the triangular patch are V1, the coordinates of vertex 2 are V2, and the coordinates of vertex 3 are V3, the first direction d1 = V2-V1, the second direction d2 = V3-V2, and the face normal vector of the triangular patch is d1*d2.

[0134] After obtaining the external normal vector, for each lesion surface grid vertex coordinate, the external normal vector is multiplied by the lesion surface grid vertex coordinate, and the product is added to the lesion surface grid vertex coordinate corresponding to the product to obtain the safety surface grid vertex coordinate.

[0135] For example, the external normal vector is Xi, the lesion surface grid vertex coordinate set is {c1, c2, c3}, and the obtained safety surface grid vertex coordinate set is {c1+Xi*c1, c2+Xi*c2, c3+Xi*c3}.

[0136] The laparoscope augmented reality surgery navigation method provided by the embodiments of the present application can mark the blood vessels in the two regions, the blood vessels in the lesion and the blood vessels in the safety resection region of the lesion. The embodiments of the present application will be described in detail below by taking the blood vessels in the lesion as an example.

[0137] In the specific implementation, after obtaining the lesion surface grid vertex coordinate set and the blood vessel surface grid vertex coordinate set, the surface intersection operation or distance operation of the blood vessels and the lesion is performed based on the blood vessel surface grid vertex coordinate set and the lesion surface grid vertex coordinate set to determine whether the blood vessels and the lesion have an intersection surface, that is, to determine whether the blood vessels invade the lesion. In other words, it is determined whether there are blood vessels in the lesion. If there are blood vessels, the blood vessel branches of the blood vessels invading the lesion need to be determined, and the blood vessel branches are marked accordingly, for example, marked in red, and then displayed on the current view of the laparoscope image to prompt the doctor that the blood vessel branches are in the lesion and that the blood vessels need to be paid attention to during the resection of the lesion and need to be stopped.

[0138] Specifically, based on the blood vessel surface grid vertex coordinate set and the lesion surface grid vertex coordinate set, whether the blood vessel and the lesion exist intersecting surfaces can be determined by the following three ways.

[0139] The first way is to perform surface intersection operation on the blood vessel and the lesion based on the blood vessel surface grid vertex coordinate set and the lesion surface grid vertex coordinate set by using intersection operation in Boolean operation. If there is no operation result, it is determined that the blood vessel and the lesion do not exist intersecting surfaces. If there is an operation result, it is determined that the blood vessel and the lesion exist intersecting surfaces.

[0140] In an embodiment, the blood vessel branch to be marked can be determined according to the operation result, for example, as shown in FIG. 6, the shaded part in the figure is the intersection result calculated, and the part of the blood vessel is marked red. Figure 4

[0141] In the embodiment of the present application, the intersection operation in Boolean operation can directly calculate the blood vessel branch to be marked, without the need to determine whether there is intersection, so that the calculation efficiency can be improved.

[0142] As shown in FIG. 7, it is a flowchart for determining the blood vessel branch to be marked provided by the embodiment of the present application. Figure 5

[0143] S501, based on the blood vessel surface grid vertex coordinate set and the lesion surface grid vertex coordinate set, surface intersection operation is performed on the blood vessel and the lesion by using intersection operation in Boolean operation;

[0144] S502, it is determined whether there is an operation result. If there is, S503 is executed, otherwise S507 is executed.

[0145] S503, it is determined that the lesion invades the blood vessel.

[0146] S504, the blood vessel branch to be marked is determined according to the operation result.

[0147] S505, the blood vessel branch is marked red.

[0148] S506, the blood vessel branch marked red is displayed on the laparoscope image of the current view.

[0149] S507, it is determined that the lesion does not invade the blood vessel.

[0150] ​​The second method includes: for each blood vessel surface grid vertex coordinate in the blood vessel surface grid vertex coordinate set, calculating a first distance of a difference between the blood vessel surface grid vertex coordinate and each lesion surface grid vertex coordinate in the lesion surface grid vertex coordinate set; determining a first minimum distance in the calculated first distances; taking a maximum value in the determined multiple first minimum distances as a first distance operation result; if the first distance operation result is less than or equal to a second preset threshold, determining that the blood vessel and the lesion exist an intersection surface, and if the first distance operation result is greater than the first preset threshold, determining that the blood vessel and the lesion do not exist the intersection surface.

[0151] For example, the lesion surface grid vertex coordinate set is {A1, A2, A3}, the blood vessel surface grid vertex coordinate set is {B1, B2, B3}, the distance d11 between the three-dimensional coordinate A1 and the three-dimensional coordinate B1 is calculated, the distance d12 between the three-dimensional coordinate A1 and the three-dimensional coordinate B2 is calculated, the distance d13 between the three-dimensional coordinate A1 and the three-dimensional coordinate B3 is calculated; the distance d21 between the three-dimensional coordinate A2 and the three-dimensional coordinate B1 is calculated, the distance d22 between the three-dimensional coordinate A2 and the three-dimensional coordinate B2 is calculated, the distance d23 between the three-dimensional coordinate A2 and the three-dimensional coordinate B3 is calculated; the distance d31 between the three-dimensional coordinate A3 and the three-dimensional coordinate B1 is calculated, the distance d32 between the three-dimensional coordinate A3 and the three-dimensional coordinate B2 is calculated, and the distance d33 between the three-dimensional coordinate A1 and the three-dimensional coordinate B3 is calculated. The minimum value determined from the distances d11, d12 and d13 is d11, the minimum value determined from the distances d21, d22 and d23 is d21, the minimum value determined from the distances d31, d32 and d33 is d32, the maximum value selected from the distances d11, d21 and d32 is d11, and finally d11 is compared with the first preset threshold. If d11 is less than or equal to the second preset threshold, it is determined that the blood vessel and the lesion exist the intersection surface, and if d11 is greater than the second preset threshold, it is determined that the blood vessel and the lesion do not exist the intersection surface.

[0152] After it is determined that the blood vessel and the lesion exist the intersection surface, a Boolean operation intersection operation is used, the lesion surface grid vertex coordinate set and the blood vessel surface grid vertex coordinate set are used, the intersection operation in the Boolean operation is used to perform surface intersection operation on the lesion and the blood vessel, and the blood vessel intersecting with the lesion is determined according to the operation result. For details, refer to Figure 4 , which will not be described here.

[0153] The method for determining the blood vessel branch to be marked provided in the second method first determines whether the lesion invades the blood vessel according to the distance, and if it is determined that the lesion invades the blood vessel, the invaded blood vessel branch is determined according to the Boolean operation of the three-dimensional grid.

[0154] As shown in Figure 6 , it is a flowchart of another method for determining the blood vessel branch to be marked provided by the embodiments of the present application.

[0155] S601, calculate a distance between each lesion surface grid vertex coordinate and each blood vessel surface grid vertex coordinate in the blood vessel surface grid vertex coordinate set;

[0156] S602, determine a minimum distance in the calculated distances;

[0157] S603, take a maximum value in the determined multiple minimum distances as a first distance operation result;

[0158] S604, judge whether the first distance operation result is less than or equal to a first preset threshold, if yes, execute S605, otherwise execute S610;

[0159] S605, determine that the blood vessel and the lesion exist an intersection surface;

[0160] S606, based on the lesion surface grid vertex coordinate set and the blood vessel surface grid vertex coordinate set, perform surface intersection operation on the lesion and the blood vessel by using intersection operation in Boolean operation;

[0161] S607, determine a blood vessel branch to be marked according to the calculation result;

[0162] S608, mark the blood vessel branch into red;

[0163] S609, display the blood vessel branch marked into red on the current view of the laparoscope image;

[0164] S610, determine that the blood vessel and the lesion do not exist an intersection surface.

[0165] In a third mode, based on the blood vessel surface grid vertex coordinate set, a center line coordinate of a center line of the blood vessel is obtained by using an average curvature flow algorithm; for each center line coordinate, a second distance between the center line coordinate and each lesion surface grid vertex coordinate in the lesion surface grid vertex coordinate set is calculated; a second minimum distance in the calculated second distances is determined; a maximum value in the determined second minimum distances is taken as a second distance operation result; if the second distance operation result is less than or equal to a third preset threshold, it is determined that the blood vessel and the lesion exist an intersection surface, and if the second distance operation result is greater than the third preset threshold, it is determined that the blood vessel and the lesion do not exist an intersection surface.

[0166] The third preset threshold can be a radius of the blood vessel.

[0167] The calculation of the second distance operation result can refer to the calculation of the distance of the first distance operation result, which will not be repeated here.

[0168] After determining the intersecting surfaces of blood vessels and lesions using the above method, the centerline coordinates corresponding to the minimum value of the determined second minimum distance are taken as the target centerline coordinates. Then, based on the target centerline coordinates and the adjacent neighborhoods of points on the centerline, the centerline branch where the target centerline coordinates are located is determined; the blood vessels corresponding to the determined centerline branches are taken as the blood vessel branches to be marked.

[0169] In this embodiment, the target centerline coordinates are the centerline coordinates closest to the lesion. Since it has been determined that there is an intersecting surface between the blood vessel and the lesion, the blood vessel corresponding to the centerline branch where the centerline coordinates closest to the lesion are located is the blood vessel intersecting with the lesion.

[0170] Specifically, to determine the centerline branch where the target centerline coordinates are located, the first centerline coordinates and the second centerline coordinates adjacent to the target centerline coordinates can be determined on the centerline. In the first direction, starting from the first centerline coordinates, the first number of adjacent centerline coordinates is determined sequentially, and in the second direction, starting from the second centerline coordinates, the second number of adjacent centerline coordinates is determined sequentially. If the first number is greater than a fourth preset threshold and the second number is greater than the fourth preset threshold, then the centerline between the centerline coordinates corresponding to the first number and the centerline coordinates corresponding to the second number is taken as the centerline branch.

[0171] Compare the images, such as Figure 7 As shown, the blood vessel includes four branches: branch L1, branch L2, branch L3, branch L4, and branch L5. The centerline coordinate e represents the target centerline coordinate. Figure 7As can be seen, the center line coordinates adjacent to the center line coordinate e are the center line coordinates e11 and the center line e21. Starting from the center line e11, in the first direction, the number of center line coordinates adjacent to the center line coordinate e11 is determined to be 2, that is, the center line coordinate e and the center line coordinate e12. Then the number of center line coordinates adjacent to the center line coordinate e12 is determined to be 2, that is, the center line coordinate e11 and the center line coordinate e13. Then the number of center line coordinates adjacent to the center line coordinate e13 is determined to be 3, that is, the center line coordinate e12, the center line coordinate e31 and the center line coordinate e41. The number 3 is greater than the fourth preset threshold 2, so it is determined that the center point coordinate e13 is the first end point of the blood vessel branch to be marked. In the second direction, the number of center line coordinates adjacent to the center line coordinate e21 is determined to be 2, that is, the center line coordinate e and the center line coordinate e22. Then the number of center line coordinates adjacent to the center line coordinate e22 is determined to be 2, that is, the center line coordinate e21 and the center line coordinate e23. Then the number of center line coordinates adjacent to the center line coordinate e23 is determined to be 2, that is, the center line coordinate e22 and the center line coordinate e24. Then the number of center line coordinates adjacent to the center line coordinate e24 is determined to be 3, that is, the center line coordinate e23, the center line coordinate e51 and the center line coordinate e61. The number 3 is greater than the fourth preset threshold 2, so it is determined that the center point coordinate e24 is the second end point of the blood vessel branch to be marked. After the first end point and the second end point are determined, the blood vessel between the first end point and the second end point is taken as the blood vessel branch to be marked, that is Figure 7 the blood vessel branch L1 in FIG.

[0172] As Figure 8 shown, a flowchart of another method for determining a blood vessel branch to be marked provided by an embodiment of the present application.

[0173] S801, based on the blood vessel surface mesh vertex coordinate set, using the average curvature flow algorithm, obtaining the center line coordinates of the center line of the blood vessel;

[0174] S802, calculating the distance between each center line coordinate and each lesion surface mesh vertex coordinate in the lesion surface mesh vertex coordinate set;

[0175] S803, determining the minimum distance in the calculated distance;

[0176] S804, taking the maximum value in the determined multiple minimum distances as a second distance operation result;

[0177] S805, judging whether the second distance operation result is less than or equal to a third preset threshold, if yes, executing S806, otherwise executing S810;

[0178] S806, determining that the blood vessel and the lesion exist an intersection surface;

[0179] S807, taking the center line coordinate corresponding to the minimum value in the determined minimum distances as a target center line coordinate;

[0180] S808, determining a center line branch corresponding to the target center line coordinate according to the target center line coordinate and the adjacent neighborhood of the point on the center line;

[0181] S809, taking the blood vessel corresponding to the center line branch as a blood vessel branch to be marked;

[0182] S810, determining that the blood vessel and the lesion do not exist an intersection surface.

[0183] In the embodiments of the present application, since the number of center line coordinates is much smaller than the number of blood vessel surface grid vertex coordinates, the determination of whether the blood vessel and the lesion exist an intersection surface by using the center line coordinates and the lesion surface grid vertex coordinates can reduce the calculation amount and improve the display efficiency of the blood vessel.

[0184] In an embodiment, after determining that the blood vessel and the lesion exist an intersection surface according to the center line coordinates, the blood vessel branch to be marked can also be determined through Boolean operation. For details, refer to the method in Mode Two, and the repeated parts will not be described herein.

[0185] In an embodiment, after determining the distance between each lesion surface grid vertex coordinate and each blood vessel surface grid vertex coordinate in the blood vessel surface grid vertex coordinate set in Mode Two, the blood vessel branch to be marked can also be determined through the method of center line coordinates. For details, refer to the method in Mode Three, and the repeated parts will not be described herein.

[0186] As shown in FIG. 8, Figure 9 is a schematic diagram of a determined center line branch provided by an embodiment of the present application, Figure 9 the center line branch marked by the black color in the figure is the center line branch corresponding to the blood vessel branch to be marked.

[0187] As shown in FIG. 9, Figure 10 is a blood vessel branch to be marked determined by an embodiment of the present application. After the blood vessel branch to be marked is marked with a corresponding color, the blood vessel branch to be marked can be displayed in the laparoscope image.

[0188] In a specific implementation, in order to facilitate the doctor to view the distance between the blood vessel and the tumor surface, the laparoscope augmented reality surgery navigation method provided by the embodiment of the present application can also mark the distance from the blood vessel branch located in the safe area to the tumor surface.

[0189] As shown in FIG. 10, Figure 11 is a complete flowchart of a laparoscope augmented reality surgery navigation method provided by an embodiment of the present application;

[0190] S1201, obtain a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set based on the registered visceral three-dimensional model;

[0191] S1202, perform surface intersection operation or distance operation on the blood vessel and the lesion based on the blood vessel surface grid vertex coordinate set and the lesion surface grid vertex coordinate set;

[0192] S1203, determine whether the blood vessel and the lesion exist intersecting surfaces according to the operation result, if yes, perform S1204, otherwise, perform S1206;

[0193] S1204, determine a first blood vessel branch to be marked;

[0194] S1205, display the first blood vessel branch to be marked on the laparoscope image after marking the first blood vessel branch to be marked;

[0195] S1206, obtain a safe surface grid vertex coordinate set of resected lesion according to each lesion surface grid vertex coordinate;

[0196] S1207, perform surface intersection operation or distance operation on the blood vessel and the safe region according to the safe surface grid vertex coordinate set and the blood vessel surface vertex coordinate set;

[0197] S1208, determine whether the blood vessel and the safe region exist intersecting surfaces according to the operation result, if yes, perform S1209, otherwise, perform S1211;

[0198] S1209, determine a second blood vessel branch to be marked;

[0199] S1210, display the second blood vessel branch to be marked on the laparoscope image after marking the second blood vessel branch to be marked;

[0200] S1211, determine that there is no blood vessel in the safe region.

[0201] Based on the same inventive concept, the principle of the laparoscope augmented reality surgery navigation device provided by the embodiments of the present application to solve the problem is similar to the principle of the above method to solve the problem, and the repeated parts will not be described again.

[0202] In an embodiment, the processor 101 calculates the safe surface grid vertex coordinate set in the following manner:

[0203] For each lesion surface grid vertex coordinate in the lesion surface grid vertex coordinate set: calculate an outward normal coordinate corresponding to the lesion surface grid vertex coordinate according to a triangular patch to which the lesion surface grid vertex coordinate belongs; multiply the outward normal coordinate and a first preset threshold; add the product and the lesion surface grid vertex coordinate to obtain a safe surface grid vertex coordinate;

[0204] The obtained safety surface mesh vertex coordinates are taken as the safety surface mesh vertex coordinate set.

[0205] In an embodiment, the processor 101 is specifically configured to:

[0206] Based on each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set, a surface intersection operation is performed on the blood vessel and the target region by using an intersection operation in Boolean operations;

[0207] If the operation result exists, it is determined that the blood vessel and the target region have an intersection surface.

[0208] In an embodiment, the processor 101 is specifically configured to:

[0209] For each blood vessel surface mesh vertex coordinate in the blood vessel surface mesh vertex coordinate set: a first distance between the blood vessel surface mesh vertex coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set is calculated; and a first minimum distance in the calculated first distances is determined.

[0210] The maximum value in the determined first minimum distances is taken as a first distance operation result.

[0211] If the first distance operation result is less than or equal to a second preset threshold value, it is determined that the blood vessel and the target region have an intersection surface.

[0212] In an embodiment, the processor 101 is specifically configured to:

[0213] Based on the blood vessel surface mesh vertex coordinate set, a centerline coordinate of a centerline of the blood vessel is obtained by using an average curvature flow algorithm.

[0214] For each centerline coordinate: a second distance between the centerline coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set is calculated; and a second minimum distance in the calculated second distances is determined.

[0215] The maximum value in the determined second minimum distances is taken as a second distance operation result.

[0216] If the second operation result is less than or equal to a third preset threshold value, it is determined that the blood vessel and the target region have an intersection surface.

[0217] In an embodiment, the processor 101 is specifically configured to:

[0218] The centerline coordinate corresponding to the minimum value in the determined second minimum distances is taken as a target centerline coordinate.

[0219] determining, according to the target centerline coordinate and the adjacent neighborhood of the point on the centerline, a centerline branch corresponding to the centerline branch where the target centerline coordinate is located;

[0220] taking the blood vessel corresponding to the centerline branch as a blood vessel branch to be marked.

[0221] In an embodiment, the processor 101 is specifically configured to:

[0222] determining, on the centerline, a first centerline coordinate and a second centerline coordinate adjacent to the target centerline coordinate;

[0223] determining, in a first direction, a first number of adjacent centerline coordinates starting from the first centerline coordinate, and determining, in a second direction, a second number of adjacent centerline coordinates starting from the second centerline coordinate;

[0224] if the first number is greater than or equal to a fourth preset threshold value and the second number is greater than or equal to the fourth preset threshold value, taking the centerline between the centerline coordinate corresponding to the first number and the centerline coordinate corresponding to the second number as the centerline branch.

[0225] In an embodiment, the processor 101 is further configured to:

[0226] performing surface intersection operation or distance operation on the blood vessel and the lesion based on the blood vessel surface grid vertex coordinate set and the lesion surface grid vertex coordinate set, to determine that the blood vessel and the lesion do not have an intersection surface.

[0227] performing surface intersection operation or distance operation on the blood vessel and the lesion safety resection region based on the blood vessel surface grid vertex coordinate set and the safety surface grid vertex coordinate set, to determine that the blood vessel and the lesion safety resection region have an intersection surface.

[0228] Based on the same inventive concept, the embodiments of the present application also provide a laparoscope augmented reality surgery navigation device, which has a similar problem solving principle to the above method and the repeated parts will not be described here.

[0229] As shown in Figure 12 The laparoscope augmented reality surgery navigation device provided by the embodiments of the present application comprises:

[0230] The acquisition module 1301 is configured to acquire a lesion surface grid vertex coordinate set and a blood vessel surface grid vertex coordinate set based on the registered internal organ three-dimensional model.

[0231] The operation module 1302 is configured to perform surface intersection operation or distance operation on the blood vessel and the target region based on the blood vessel surface grid vertex coordinate set and the target surface grid vertex coordinate set, and determine that the blood vessel and the target region have intersection surfaces, wherein the target surface grid vertex coordinate set includes the lesion surface grid vertex coordinate set or a safety surface grid vertex coordinate set corresponding to a safe resection region of the lesion; and the safety surface grid vertex coordinate set is calculated based on the lesion surface grid vertex coordinate set.

[0232] The marking display module 1303 is configured to determine a blood vessel branch to be marked according to the operation result, mark the blood vessel branch to be marked, and display the marked blood vessel branch on the current laparoscope image.

[0233] In an embodiment, the operation module 1302 is specifically configured to:

[0234] For each lesion surface grid vertex coordinate in the lesion surface grid vertex coordinate set, an external normal coordinate corresponding to the lesion surface grid vertex coordinate is calculated according to a triangular patch to which the lesion surface grid vertex coordinate belongs; the external normal coordinate is multiplied by a first preset threshold, and the product is added to the lesion surface grid vertex coordinate to obtain a safety surface grid vertex coordinate.

[0235] The obtained safety surface grid vertex coordinate is taken as the safety surface grid vertex coordinate set.

[0236] In an embodiment, the operation module 1302 is specifically configured to:

[0237] For each blood vessel surface grid vertex coordinate in the blood vessel surface grid vertex coordinate set and each target surface grid vertex coordinate in the target surface grid vertex coordinate set, an intersection operation in Boolean operation is adopted to perform surface intersection operation on the blood vessel and the target region.

[0238] If there is an operation result, it is determined that the blood vessel and the target region have intersection surfaces.

[0239] In an embodiment, the operation module 1302 is specifically configured to:

[0240] For each blood vessel surface grid vertex coordinate in the blood vessel surface grid vertex coordinate set, a first distance between the blood vessel surface grid vertex coordinate and each target surface grid vertex coordinate in the target surface grid vertex coordinate set is calculated; and a first minimum distance in the calculated first distances is determined.

[0241] A maximum value in the determined first minimum distances is taken as a first distance operation result.

[0242] If the first distance operation result is less than or equal to a second preset threshold, it is determined that the blood vessel and the target region exist intersecting surfaces.

[0243] In an embodiment, the operation module 1302 is specifically configured to:

[0244] Based on the blood vessel surface mesh vertex coordinate set, a mean curvature flow algorithm is used to obtain a center line coordinate of a center line of the blood vessel;

[0245] For each center line coordinate, a second distance between the center line coordinate and each target surface mesh vertex coordinate in the target surface mesh vertex coordinate set is calculated, and a second minimum distance in the calculated second distances is determined;

[0246] The maximum value in the determined second minimum distances is taken as a second distance operation result;

[0247] If the second operation result is less than or equal to a third preset threshold, it is determined that the blood vessel and the target region exist intersecting surfaces.

[0248] In an embodiment, the marking display module 1303 is specifically configured to:

[0249] The center line coordinate corresponding to the minimum value in the determined second minimum distances is taken as a target center line coordinate;

[0250] According to the target center line coordinate and a neighboring neighborhood of a point on the center line, a center line branch in which the target center line coordinate is located is determined;

[0251] The blood vessel corresponding to the center line branch is taken as a blood vessel branch to be marked.

[0252] In an embodiment, the marking display module 1303 is specifically configured to:

[0253] On the center line, a first center line coordinate and a second center line coordinate adjacent to the target center line coordinate are determined;

[0254] In a first direction, a first number of adjacent center line coordinates is determined successively from the first center line coordinate as a starting point, and in a second direction, a second number of adjacent center line coordinates is determined successively from the second center line coordinate as a starting point;

[0255] If the first number is greater than or equal to a fourth preset threshold, and the second number is greater than or equal to the fourth preset threshold, a center line between the center line coordinate corresponding to the first number and the center line coordinate corresponding to the second number is taken as the center line branch.

[0256] In an embodiment, the operation module 1302 is specifically configured to:

[0257] performing surface intersection operation or distance operation on the blood vessel and the lesion based on the blood vessel surface mesh vertex coordinate set and the lesion surface mesh vertex coordinate set, to determine that the blood vessel and the lesion do not have intersecting surface;

[0258] performing surface intersection operation or distance operation on the blood vessel and the lesion safety resection region based on the blood vessel surface mesh vertex coordinate set and the lesion safety resection region surface mesh vertex coordinate set, to determine that the blood vessel and the lesion safety resection region have intersecting surface.

[0259] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0260] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as a combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for performing the function specified by the flow or flows and / or block or blocks.

[0261] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0262] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flows and / or blocks Figure 1 Figure 1steps of the functions specified in the one or more blocks.

[0263] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A laparoscopic augmented reality surgical navigation method, characterized in that, The method includes: Based on the registered 3D model of the internal organs, the coordinate sets of the vertex grids on the lesion surface and the coordinate sets of the vertex grids on the blood vessel surface are obtained. Based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the target surface mesh, surface intersection or distance calculations are performed on the blood vessel and the target region to determine that there is an intersecting surface between the blood vessel and the target region. The vertex coordinate set of the target surface mesh includes the vertex coordinate set of the lesion surface mesh, or the vertex coordinate set of the safe surface mesh corresponding to the safe resection area of ​​the lesion. The vertex coordinate set of the safe surface mesh is calculated based on the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh. Based on the calculation results of the intersecting surfaces, the vascular branches to be marked within the target area are determined, and the vascular branches to be marked are then displayed on the laparoscopic image in the current view.

2. The method as described in claim 1, characterized in that, The set of vertex coordinates for the secure surface mesh is calculated using the following method: For each vertex coordinate of the lesion surface grid in the vertex coordinate set: calculate the outward normal coordinate corresponding to the vertex coordinate of the lesion surface grid according to the triangular facet to which the vertex coordinate of the lesion surface grid belongs; multiply the outward normal coordinate with a first preset threshold, and add the product to the vertex coordinate of the lesion surface grid to obtain the vertex coordinate of the safe surface grid; The obtained vertex coordinates of the safe surface mesh are used as the vertex coordinate set of the safe surface mesh.

3. The method as described in claim 1, characterized in that, The step of performing surface intersection calculations on the blood vessel and the target region based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the target surface mesh to determine that there is an intersecting surface between the blood vessel and the target region includes: Based on the vertex coordinates of each blood vessel surface grid in the blood vessel surface grid vertex coordinate set and the vertex coordinates of each target surface grid in the target surface grid vertex coordinate set, the intersection operation in Boolean operation is used to perform surface intersection operation on the blood vessel and the target disease region. If a calculation result exists, it is determined that the blood vessel and the target region have an intersecting surface.

4. The method as described in claim 1, characterized in that, The step of performing distance calculations on the blood vessel and the target region based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the target surface mesh to determine whether the blood vessel and the target region have intersecting surfaces includes: For each vertex coordinate of the blood vessel surface mesh in the set of vertex coordinates of the blood vessel surface mesh: calculate a first distance between the vertex coordinates of the blood vessel surface mesh and the vertex coordinates of each vertex coordinate of the target surface mesh in the set of vertex coordinates of the target surface mesh; determine a first minimum distance in the calculated first distance; The maximum value among the determined first minimum distances is taken as the result of the first distance calculation. If the result of the first distance calculation is less than or equal to the second preset threshold, it is determined that the blood vessel and the target region have an intersecting surface.

5. The method as described in claim 1, characterized in that, The step of performing distance calculations on the blood vessel and the target region based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the target surface mesh to determine whether the blood vessel and the target region have intersecting surfaces includes: Based on the vertex coordinate set of the blood vessel surface mesh, the centerline coordinates of the blood vessel's centerline are obtained using the average curvature flow algorithm. For each centerline coordinate: calculate the second distance between the centerline coordinate and the vertex coordinates of each target surface mesh in the target surface mesh vertex coordinate set, and determine the second minimum distance among the calculated second distances; The maximum value among the determined second minimum distances is taken as the result of the second distance calculation. If the result of the second distance calculation is less than or equal to the third preset threshold, it is determined that the blood vessel and the target region have an intersecting surface.

6. The method as described in claim 5, characterized in that, The step of determining the vascular branches to be marked within the target region based on the calculation results of the intersecting surfaces includes: The centerline coordinates corresponding to the minimum value in the determined second minimum distance are taken as the target centerline coordinates; Based on the target centerline coordinates and the neighboring regions of the points on the centerline, determine the centerline branch where the target centerline coordinates are located; The blood vessels corresponding to the central line branches are designated as the blood vessel branches to be marked.

7. The method as described in claim 6, characterized in that, The step of determining the branch of the centerline corresponding to the target centerline coordinates based on the target centerline coordinates and the neighboring regions of points on the centerline includes: On the centerline, determine the coordinates of a first centerline and a second centerline adjacent to the coordinates of the target centerline. In the first direction, starting from the first centerline coordinates, the first number of adjacent centerline coordinates is determined sequentially; and in the second direction, starting from the second centerline coordinates, the second number of adjacent centerline coordinates is determined sequentially. If the first number is greater than the fourth preset threshold, and the second number is greater than the fourth preset threshold, then the centerline between the centerline coordinates corresponding to the first number and the centerline coordinates corresponding to the second number is taken as the centerline branch.

8. The method according to any one of claims 1 to 7, characterized in that, The method also includes: Based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the lesion surface mesh, surface intersection calculation or distance calculation is performed on the blood vessel and the lesion to determine that there is no intersecting surface between the blood vessel and the lesion. Based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the safe surface mesh, surface intersection or distance calculations are performed on the blood vessel and the safe resection area of ​​the lesion to determine that there is an intersecting surface between the blood vessel and the safe resection area of ​​the lesion.

9. A laparoscopic augmented reality fusion display device, characterized in that, The device includes a processor and a display screen; The processor is configured to acquire the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh based on the registered 3D model of the internal organs; perform surface intersection or distance calculations on the blood vessels and the target region based on the vertex coordinate set of the blood vessel surface mesh and the target region to determine that there is an intersecting surface between the blood vessels and the target region, wherein the vertex coordinate set of the target surface mesh includes the vertex coordinate set of the lesion surface mesh or the vertex coordinate set of the safe surface mesh corresponding to the safe resection area of ​​the lesion; the vertex coordinate set of the safe surface mesh is calculated based on the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh; determine the blood vessel branches to be marked within the target region based on the calculation results of the intersecting surface, and mark the blood vessel branches to be marked accordingly; The display screen is used to perform laparoscopic augmented reality fusion display on the marked vascular branches.

10. A laparoscopic augmented reality fusion display device, characterized in that, include: The acquisition module is used to acquire the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh based on the registered 3D model of the internal organs; The calculation module is used to perform surface intersection or distance calculations on the blood vessel and the target region based on the vertex coordinate set of the blood vessel surface mesh and the vertex coordinate set of the target surface mesh, to determine that there is an intersecting surface between the blood vessel and the target region. The vertex coordinate set of the target surface mesh includes the vertex coordinate set of the lesion surface mesh, or the vertex coordinate set of the safe surface mesh corresponding to the safe resection area of ​​the lesion; the vertex coordinate set of the safe surface mesh is calculated based on the vertex coordinate set of the lesion surface mesh and the vertex coordinate set of the blood vessel surface mesh. The marking display module is used to determine the vascular branches to be marked within the target area based on the calculation results of the intersecting surfaces, and to mark the vascular branches to be marked accordingly and display them on the laparoscopic image in the current view.

Images