Augmented reality visualization method, system, device and medium for vascular anatomical structure

Abstract

The present invention proposes an augmented reality visualization method, system, device and medium for vascular anatomical structure, the method comprising: constructing a 3D vascular structure model according to preoperative image data and performing vascular segmentation processing on the model; creating a 2D abstract graph and a mapping relationship between the 3D model and the 2D abstract graph based on the preoperative image data and the 3D vascular structure model; using an AR device to visualize the 2D abstract graph in an off-body manner with a patient, performing intraoperative navigation based on the 2D abstract graph, and using the nodes corresponding to the vascular segmentation in the 2D abstract graph as an interactive interface for 3D vascular structure visualization; using an AR device to visualize the 3D vascular model in an on-body manner with a patient, performing local geometric visualization of a selected set of 3D structures of the 3D vascular model through an interactive interface, and then visually outputting attribute information of the 3D vascular structure through texture mapping.

Description

Technical Field

[0001] The present invention relates to the field of surgical navigation technology, and in particular to an augmented reality visualization method, system, device and medium for vascular anatomical structure. Background Art

[0002] For example, augmented reality technology can overlay virtual information on the real world in real time and be used in scenarios such as situational teaching, information guidance, and entertainment. It can provide real-time information guidance for surgical operations, improve surgical accuracy, shorten operation time, and benefit patients.

[0003] However, in existing surgical augmented reality visualization guidance applications, the guidance information is usually relatively simple, mainly superimposed on anatomical structures. The geometric structure and spatial relationship of the human vascular system (such as blood vessels, arteries, veins and trachea) are extremely complex. If only the preoperatively reconstructed vascular structure is superimposed on the patient, this method has the following defects:

[0004] (1) It is difficult to browse and search for the target vascular segment, and it is difficult to grasp the direction of vascular branches and the relationship between each segment;

[0005] (2) It fails to present various properties of blood vessels, such as wall thickness and diameter, which are often crucial for disease analysis;

[0006] (3) Presenting the entire virtual anatomical structure of the vascular system in the augmented reality system not only affects the visual effect but also increases the complexity of rendering.

[0007] Therefore, there is an urgent need for an augmented reality visualization method of vascular anatomical structure to solve the above problems. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides an augmented reality visualization method, system, device and medium for vascular anatomical structure.

[0009] A first aspect of the present invention discloses an augmented reality visualization method for a vascular anatomical structure, comprising:

[0010] S1: obtaining preoperative imaging data of the patient for preprocessing, constructing a 3D vascular structure model according to the preoperative imaging data and performing vascular segmentation processing on the model, wherein the 3D vascular structure model includes a surface model and a voxel model;

[0011] S2: creating a 2D abstract graph and a mapping relationship between the 3D model and the 2D abstract graph according to the preoperative image data and the 3D vascular structure model, wherein the task of creating the 2D abstract graph includes extracting a topological structure and data encoding, and the mapping relationship includes a geometric mapping relationship, an attribute mapping relationship, and an operation mapping relationship;

[0012] S3: Using an AR device to visualize the 2D abstract image in a patient-external manner, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure;

[0013] S4: The 3D vascular structure model reconstructed before the operation of the patient is uniformly registered with the vascular structure set data collected in real time during the operation in the same coordinate system through a coarse registration algorithm and a fine registration algorithm respectively, the 3D vascular model is visualized in the patient's body through an AR device, the selected set 3D structure of the 3D vascular model is locally geometrically visualized through the interactive interface, and the attribute information of the 3D vascular structure is visualized and output through texture mapping.

[0014] In an optional embodiment, the step of obtaining pre-processing the patient's pre-operative image data, constructing a 3D vascular structure model based on the pre-operative image data, and performing vascular segmentation processing on the model includes:

[0015] S11: performing filtering processing, artifact elimination and image enhancement on the preoperative image data in sequence;

[0016] S12: Using convolutional neural network to perform image segmentation on the pre-processed preoperative image data to extract the patient's vascular structure;

[0017] S13: performing surface reconstruction and voxelization processing on the extracted vascular structure to obtain a 3D vascular model including a surface model and a voxel model;

[0018] S14: Based on the spatial relationship between the segments of the patient's vascular structure, the geometric data of the 3D vascular model is organized through a hierarchical structure tree, and then the 3D vascular model is segmented.

[0019] In an optional embodiment, the extracting the topological structure in the creating of the 2D abstract graph according to the preoperative image data and the 3D vascular structure model comprises:

[0020] S21: extracting the center line of each vascular segment in the preoperative image data and / or the 3D vascular structure model by a refinement algorithm, and connecting the center lines of each vascular segment by a center line completion method to obtain a continuous 3D curve cluster;

[0021] S22: Create a vascular node for each vascular segment, create edges between any two adjacent vascular nodes according to the connection relationship between the vascular segments in the 3D curve cluster, and convert the 3D curve cluster into a 2D abstract graph structure by projection and / or flattening.

[0022] In an optional embodiment, in creating a 2D abstract image based on the preoperative image data and the 3D vascular structure model, data encoding includes:

[0023] S23: A data model G(V,E) representing a 2D abstract graph as a tree structure and including a node set V and an edge set E, wherein each node v of the node set V i represents a vascular segment in the vascular structure, each edge e of the edge set E i Indicates the connection relationship between two adjacent vascular segments;

[0024] S24: the node v i The vascular information is stored, and the vascular information includes node v i A 3D model corresponding to the vascular segmentation, a 3D centerline extracted from the 3D model, and a 2D centerline obtained by projection and / or flattening of the 3D centerline, and encoding the properties of the vessel on the 3D centerline and the 2D centerline, wherein the centerline is defined as a parametric equation of a continuous model and / or a set of key points of a discrete model, and the encoded attribute is defined as a parametric equation representation and / or an attribute vector group representation.

[0025] In an optional embodiment, the creating a mapping relationship between the 3D model and the 2D abstract image according to the preoperative image data and the 3D vascular structure model includes:

[0026] S25: through the ID relationship between the vascular segmentation and the 2D abstract graph node, each vascular segmentation file of the 3D model is bound to the 2D abstract graph node, the 3D center line corresponding to the vascular segment is uniformly sampled, and the coordinate sampling results are stored at discrete points of the 2D center line to establish a geometric mapping relationship between the 3D vascular model and the geometric information corresponding to the 2D abstract graph, and the geometric mapping relationship is used to provide data association for AR visualization interaction;

[0027] S26: Define the number of attribute vectors according to the coded attributes, store the coded attributes of the vascular segmentation in the attribute vector, perform non-uniform key point sampling in the 3D vascular model space, the definition of the key points includes the starting point, the end point and the middle point of the vascular segmentation, and store the attribute values ​​of the key points through the attribute vector. The dimension of the attribute vector is the same as the number of key points, so as to establish an attribute mapping relationship between the 3D vascular model and the corresponding attribute information of the 2D abstract graph, and provide an interactive interface and data support for attribute visualization for AR visualization interaction through the attribute mapping relationship.

[0028] In an optional embodiment, the use of an AR device to visualize the 2D abstract image in a patient-external manner, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure includes:

[0029] S31: All nodes v in the data model G(V,E) i The corresponding 2D centerline is rendered, and the 2D abstract graph AR visualization of the vascular segment connection relationship is performed through the geometric data of the centerline defined as the parametric equation of the continuous model and / or the key point set of the discrete model;

[0030] S32: the user customizes the vascular visualization scheme according to the interactive configuration, draws each vascular segment by reading the corresponding attribute vector of the vascular to form a grid expansion diagram of vascular visualization, and performs 2D visualization of vascular geometric attributes through the grid expansion diagram, wherein the 2D visualization of geometric attributes includes visualization of diameter information of the vascular segment and visualization of wall thickness information of the vascular segment;

[0031] S33: The user obtains a 2D coordinate point set of the target visualization through a point selection operation and / or a box selection operation, queries and / or interpolates the attribute data corresponding to the 2D coordinate point set through an attribute vector to obtain an attribute data result set and outputs and displays it in a list, and the 2D visualization of the non-geometric attributes of the blood vessels is presented in a 2D curve and / or a grid expansion diagram using color coding;

[0032] S34: using a positioning and tracking sensor device to obtain spatial position information of the end of the surgical instrument, calculating the position of the end of the surgical instrument in the vascular segment and the corresponding 2D abstract map based on the spatial geometric mapping relationship, displaying the end of the surgical instrument in the 2D abstract map through an icon, and marking the movement trajectory of the surgical instrument in the 2D abstract map with a light color and marking the path between the position of the end of the surgical instrument and the lesion with a dark color;

[0033] S35: Setting each node in the 2D abstract graph as an interactive object through a development engine and / or a development toolkit, and executing interactive visualization of the 2D abstract graph interactive interface and the 3D vascular structure through a script.

[0034] In an optional embodiment, the using of an AR device to visualize the 3D vascular model in a patient-in-body manner, performing local geometric visualization of a selected set of 3D structures of the 3D vascular model through the interactive interface, and then visually outputting attribute information of the 3D vascular structure through texture mapping includes:

[0035] S41: selecting a 2D selection set of nodes for local geometric visualization in the 2D abstract graph by point selection and / or box selection, placing a vascular segment corresponding to each node in the 2D selection set into a 3D selection set according to a geometric mapping relationship, generating a selection set by a preoperative planning path and placing the vascular segments associated with the surgical path into the 3D selection set, visualizing the vascular structure in the selection set, and performing display and hiding control and / or highlight rendering display on the vascular structure segments;

[0036] S42: Query and calculate the attribute values ​​of each key point through the 3D centerline coding attributes associated with the vascular segmentation, use texture mapping to map the attribute information to the pixels of the grid expansion image corresponding to the vascular segmentation to generate a texture map, map the texture map to the geometric surface of the vascular segment and visualize the attribute information through gradient color coding.

[0037] A second aspect of the present invention discloses an augmented reality visualization system for vascular anatomical structures, the system comprising:

[0038] A 3D modeling module is used to obtain the patient's preoperative imaging data for preprocessing, construct a 3D vascular structure model based on the preoperative imaging data, and perform vascular segmentation processing on the model, wherein the 3D vascular structure model includes a surface model and a voxel model;

[0039] A 2D modeling module, used to create a 2D abstract graph and a mapping relationship between a 3D model and a 2D abstract graph according to the preoperative imaging data and the 3D vascular structure model, wherein the creation task of the 2D abstract graph includes extracting a topological structure and data encoding, and the mapping relationship includes a geometric mapping relationship, an attribute mapping relationship, and an operation mapping relationship;

[0040] A 2D visualization interaction module, for visually presenting the 2D abstract image in an out-of-body manner using an AR device, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure;

[0041] The 3D visualization interaction module is used to uniformly align the 3D vascular structure model reconstructed before the patient's operation with the vascular structure set data collected in real time during the operation in the same coordinate system through a coarse alignment algorithm and a fine alignment algorithm respectively, and use an AR device to visualize the 3D vascular model in the patient's body, perform local geometric visualization of the selected set 3D structure of the 3D vascular model through the interactive interface, and then visualize the attribute information of the 3D vascular structure through texture mapping.

[0042] A third aspect of the present invention discloses an augmented reality visualization device for vascular anatomical structures, comprising:

[0043] at least one processor, and

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the augmented reality visualization method of the vascular anatomical structure as disclosed in any one of the first aspects of the present invention.

[0046] A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to execute an augmented reality visualization method for a vascular anatomical structure as described in any one of the first aspect of the present invention.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] The present invention reconstructs a 3D vascular structure model through the patient's preoperative imaging data, creates a corresponding abstract graph, and uses the mapping relationship between the abstract graph and the 3D vascular structure to realize AR visualization interaction. It supports tasks such as overview, retrieval, and local operation in the AR environment, and can select browsing and retrieval methods according to the user's intention in the AR scene. It can effectively support personalized preoperative AR planning analysis and intraoperative AR guidance for patients, and provide solutions for precise surgical diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A flowchart of the method for augmented reality visualization of vascular anatomical structures of the present invention;

[0051] Figure 2 Schematic diagram of an augmented reality visualization system for vascular anatomical structures of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] Example 1

[0057] See also Figure 1 , an embodiment of the present invention discloses an augmented reality visualization method for vascular anatomical structure, comprising:

[0058] S1: obtaining preoperative imaging data of the patient for preprocessing, constructing a 3D vascular structure model according to the preoperative imaging data and performing vascular segmentation processing on the model, wherein the 3D vascular structure model includes a surface model and a voxel model;

[0059] S2: creating a 2D abstract graph and a mapping relationship between the 3D model and the 2D abstract graph according to the preoperative image data and the 3D vascular structure model, wherein the task of creating the 2D abstract graph includes extracting a topological structure and data encoding, and the mapping relationship includes a geometric mapping relationship, an attribute mapping relationship, and an operation mapping relationship;

[0060] S3: Using an AR device to visualize the 2D abstract image in a patient-external manner, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure;

[0061] S4: The 3D vascular structure model reconstructed before the operation of the patient is uniformly registered with the vascular structure set data collected in real time during the operation in the same coordinate system through a coarse registration algorithm and a fine registration algorithm respectively, the 3D vascular model is visualized in the patient's body through an AR device, the selected set 3D structure of the 3D vascular model is locally geometrically visualized through the interactive interface, and the attribute information of the 3D vascular structure is visualized and output through texture mapping.

[0062] It should be noted that the 3D vascular structure model is reconstructed through preoperative imaging data to create a corresponding abstract 2D graphical structure (referred to as 2D abstract graph), and the mapping relationship between the abstract graph and the 3D vascular structure is used to realize AR visualization interaction, supporting tasks such as overview, retrieval and local operations.

[0063] Furthermore, the 3D model of the blood vessels is reconstructed using the preoperative imaging data of the personalized patient. The task of creating the abstract graph includes the extraction of the topological structure and the encoding of the data, as well as the establishment of a mapping definition that associates the 3D vascular model with the abstract graph. The topological structure extraction of the abstract graph can be directly generated from the preoperative imaging data, or it can be achieved through algorithms such as refinement based on the 3D vascular structure. The purpose of the mapping is to provide a basis for visualization interaction.

[0064] In an optional embodiment, the step of obtaining pre-processing the patient's pre-operative image data, constructing a 3D vascular structure model based on the pre-operative image data, and performing vascular segmentation processing on the model includes:

[0065] S11: performing filtering processing, artifact elimination and image enhancement on the preoperative image data in sequence;

[0066] S12: Using convolutional neural network to perform image segmentation on the pre-processed preoperative image data to extract the patient's vascular structure;

[0067] S13: performing surface reconstruction and voxelization processing on the extracted vascular structure to obtain a 3D vascular model including a surface model and a voxel model;

[0068] S14: Based on the spatial relationship between the segments of the patient's vascular structure, the geometric data of the 3D vascular model is organized through a hierarchical structure tree, and then the 3D vascular model is segmented.

[0069] In an optional embodiment, the extracting the topological structure in the creating of the 2D abstract graph according to the preoperative image data and the 3D vascular structure model comprises:

[0070] S21: extracting the center line of each vascular segment in the preoperative image data and / or the 3D vascular structure model by a refinement algorithm, and connecting the center lines of each vascular segment by a center line completion method to obtain a continuous 3D curve cluster;

[0071] S22: Create a vascular node for each vascular segment, create edges between any two adjacent vascular nodes according to the connection relationship between the vascular segments in the 3D curve cluster, and convert the 3D curve cluster into a 2D abstract graph structure by projection and / or flattening.

[0072] In an optional embodiment, in creating a 2D abstract image based on the preoperative image data and the 3D vascular structure model, data encoding includes:

[0073] S23: A data model G(V,E) representing a 2D abstract graph as a tree structure and including a node set V and an edge set E, wherein each node v of the node set V i represents a vascular segment in the vascular structure, each edge e of the edge set E i Indicates the connection relationship between two adjacent vascular segments;

[0074] S24: the node v i The vascular information is stored, and the vascular information includes node v i A 3D model corresponding to the vascular segmentation, a 3D centerline extracted from the 3D model, and a 2D centerline obtained by projection and / or flattening of the 3D centerline, and encoding the properties of the vessel on the 3D centerline and the 2D centerline, wherein the centerline is defined as a parametric equation of a continuous model and / or a set of key points of a discrete model, and the encoded attribute is defined as a parametric equation representation and / or an attribute vector group representation.

[0075] Furthermore, in general, there is no loop in the 2D abstract graph structure, and it is presented as a tree structure. The branches do not intersect each other and there is no mutual occlusion. However, there may be a cyclic structure, so G(V,E) is used to represent the abstract graph. Each node vi in ​​the vertex set V represents a vascular segment; each edge ei in the edge set E represents the connection relationship between adjacent vascular segments, and there is a many-to-many relationship between vascular segments. The nodes store a variety of information including the 3D model of the vascular segment, the centerline extracted from the 3D model, and the 2D centerline obtained after processing. The diameter, wall thickness and other attributes of the vessel can be encoded on the 3D centerline and the 2D centerline. The centerline can be defined as a continuous parametric equation or a discrete set of key points, and accordingly, the attributes can also be represented by a series of parametric equations or attribute vector groups.

[0076] In an optional embodiment, the creating a mapping relationship between the 3D model and the 2D abstract image according to the preoperative image data and the 3D vascular structure model includes:

[0077] S25: through the ID relationship between the vascular segmentation and the 2D abstract graph node, each vascular segmentation file of the 3D model is bound to the 2D abstract graph node, the 3D center line corresponding to the vascular segment is uniformly sampled, and the coordinate sampling results are stored at discrete points of the 2D center line to establish a geometric mapping relationship between the 3D vascular model and the geometric information corresponding to the 2D abstract graph, and the geometric mapping relationship is used to provide data association for AR visualization interaction;

[0078] S26: Define the number of attribute vectors according to the coded attributes, store the coded attributes of the vascular segmentation in the attribute vector, perform non-uniform key point sampling in the 3D vascular model space, the definition of the key points includes the starting point, the end point and the middle point of the vascular segmentation, and store the attribute values ​​of the key points through the attribute vector. The dimension of the attribute vector is the same as the number of key points, so as to establish an attribute mapping relationship between the 3D vascular model and the corresponding attribute information of the 2D abstract graph, and provide an interactive interface and data support for attribute visualization for AR visualization interaction through the attribute mapping relationship.

[0079] Furthermore, geometric mapping establishes the corresponding relationship between the 3D vascular model and the 2D abstract graph in terms of geometric information. Through the relationship between the vascular segment ID and the abstract graph node ID, the 3D model file of each vascular segment is bound to the node of the abstract graph. Suppose the 3D centerline corresponding to the vascular segment is, uniformly sample the coordinates of the centerline, and store the sampling results at the corresponding 2D centerline discrete points. Geometric mapping prepares the necessary data associations for AR visualization interaction, and can also be used to calculate the key anatomical structures adjacent to surgical instruments, generate risk relationship heat maps, and present them in a visual manner.

[0080] Furthermore, attribute mapping establishes the corresponding relationship between the attribute information of the 3D vascular model and the 2D abstract graph, so as to use the attribute vector of the 2D abstract graph to encode various attribute information for visualization of the abstract graph and visualization of the attributes of the 3D vascular segment with the abstract graph as the interactive interface. For vascular segmentation, m vectors are used to store the attributes of the vascular segmentation (such as diameter, wall thickness, etc.), and the number m is consistent with the number of attributes that need to be encoded. The dimension of each vector is d, storing the attribute values ​​of d key points. Non-uniform key point sampling is performed in the 3D vascular space, and the key points are defined as the starting point, the end point, and the key intermediate point (i.e., the point where the parameter changes greatly).

[0081] In an optional embodiment, the use of an AR device to visualize the 2D abstract image in a patient-external manner, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure includes:

[0082] S31: All nodes v in the data model G(V,E) i The corresponding 2D centerline is rendered, and the 2D abstract graph AR visualization of the vascular segment connection relationship is performed through the geometric data of the centerline defined as the parametric equation of the continuous model and / or the key point set of the discrete model;

[0083] S32: the user customizes the vascular visualization scheme according to the interactive configuration, draws each vascular segment by reading the corresponding attribute vector of the vascular to form a grid expansion diagram of vascular visualization, and performs 2D visualization of vascular geometric attributes through the grid expansion diagram, wherein the 2D visualization of geometric attributes includes visualization of diameter information of the vascular segment and visualization of wall thickness information of the vascular segment;

[0084] S33: The user obtains a 2D coordinate point set of the target visualization through a point selection operation and / or a box selection operation, queries and / or interpolates the attribute data corresponding to the 2D coordinate point set through an attribute vector to obtain an attribute data result set and outputs and displays it in a list, and the 2D visualization of the non-geometric attributes of the blood vessels is presented in a 2D curve and / or a grid expansion diagram using color coding;

[0085] Furthermore, the AR visualization content of the 2D abstract graph includes: the connection relationship of the vascular segments, the visualization of the diameter information of the vascular segments, the visualization of the wall thickness information of the vascular segments, the visualization of the distance information between the vascular segments and the lesions, and the visualization of other attributes in the vascular segments.

[0086] Furthermore, the 2D centerline corresponding to each node is rendered from the data model G(V,E) of the 2D abstract graph. The geometric data comes from continuous parameterized equations or discrete models. For the visualization of attributes, the interactive interface allows users to customize the visualization scheme and present different effects according to the user's interactive configuration. For the size of the vascular diameter, it can be represented schematically by the thickness of the line, or the corresponding attribute vector can be read to draw each vascular segment with the actual diameter. This visualization is similar to the grid expansion diagram of the vascular. For the visualization of non-geometric attribute data, color coding is used to directly present it in the 2D curve or 2D grid expansion diagram. The user clicks or selects in the 2D abstract graph to obtain the 2D coordinates of the selected point set, and then queries or interpolates in the attribute vector to display the attribute data result set in a list format.

[0087] S34: using a positioning and tracking sensor device to obtain spatial position information of the end of the surgical instrument, calculating the position of the end of the surgical instrument in the vascular segment and the corresponding 2D abstract map based on the spatial geometric mapping relationship, displaying the end of the surgical instrument in the 2D abstract map through an icon, and marking the movement trajectory of the surgical instrument in the 2D abstract map with a light color and marking the path between the position of the end of the surgical instrument and the lesion with a dark color;

[0088] Furthermore, the method in step S3 is used to display a 2D abstract image in the AR scene, the position of the lesion is illustrated in the abstract image, and the spatial position information of the end of the surgical instrument is obtained by using a positioning and tracking sensor device. According to the spatial geometric mapping relationship, the vascular segment where the end of the surgical instrument is located and its position in the 2D abstract image are calculated and displayed in the abstract image with a schematic icon. In the 2D abstract image, the trajectory of the surgical instrument is marked with a light color, and the path between the current end position of the instrument and the lesion is marked with a dark color.

[0089] S35: Setting each node in the 2D abstract graph as an interactive object through a development engine and / or a development toolkit, and executing interactive visualization of the 2D abstract graph interactive interface and the 3D vascular structure through a script.

[0090] Furthermore, the 3D vascular structure visualization with the abstract graph as the interactive interface displays the 2D abstract graph in the AR scene, with the abstract graph as the main interactive interface, so that it is always facing the user. With the help of common development engines (such as Unity and UnrealEngine) and AR development toolkits (such as Microsoft's Mixed Reality ToolKit, MRTK), each node of the 2D abstract graph is set as an interactive object, and different visualization and interactive operation functions are realized using scripting languages.

[0091] It should be noted that in the AR surgical guidance system, the complete or partial 3D vascular structure is superimposed on the patient. In the preoperative AR planning system, the complete or partial 3D vascular structure can be superimposed on the 3D printed model or phantom. AR hardware devices can choose head-mounted displays, pads or smartphones, etc., and usually professional head-mounted displays are selected to obtain higher immersion and positioning accuracy.

[0092] Furthermore, a two-stage registration algorithm combining coarse and fine is used to unify the preoperatively reconstructed 3D vascular structure and the real-time acquired geometric data into the same coordinate system. The device is calibrated according to the sensors built into the hardware device, and the registered virtual and real objects are converted into the device's observation coordinate system. The user's observation angle is used as the rendering parameter of the virtual 3D model to achieve geometrically consistent virtual and real fusion presentation.

[0093] Furthermore, 3D vascular structures have rich spatial information, but the related attributes and semantic information need to be interpreted or measured by professionals. Usually, the spatial direction of vascular structures is complex, and there are cross-occlusions between vascular segments and between vascular and other organ tissues, which makes it difficult to search and view local structures and form an overall overview. After operations such as projection, cross removal, flattening and optimized layout, 2D abstract images reduce the user's visual burden, support quick selection, search and browsing tasks, and are more conducive to doctors analyzing the condition and intraoperative navigation. However, abstract images cannot intuitively grasp the spatial information of vascular structures. Therefore, 3D vascular structures and abstract images can be displayed simultaneously in AR scenes. Among them, 3D vascular structures are usually displayed in vivo, and 3D registration of preoperative data and intraoperative data is required. Abstract images are presented in an ex vivo manner. Abstract images have two main functions: one is for disease analysis and intraoperative 2D navigation, and the other is as an operation interface for the visualization of 3D vascular structures. When used as an interface, it can be set in the form of a billboard so that it is always facing the user.

[0094] In an optional embodiment, the using of an AR device to visualize the 3D vascular model in a patient-in-body manner, performing local geometric visualization of a selected set of 3D structures of the 3D vascular model through the interactive interface, and then visually outputting attribute information of the 3D vascular structure through texture mapping includes:

[0095] S41: selecting a 2D selection set of nodes for local geometric visualization in the 2D abstract graph by point selection and / or box selection, placing a vascular segment corresponding to each node in the 2D selection set into a 3D selection set according to a geometric mapping relationship, generating a selection set by a preoperative planning path and placing the vascular segments associated with the surgical path into the 3D selection set, visualizing the vascular structure in the selection set, and performing display and hiding control and / or highlight rendering display on the vascular structure segments;

[0096] S42: Query and calculate the attribute values ​​of each key point through the 3D centerline coding attributes associated with the vascular segmentation, use texture mapping to map the attribute information to the pixels of the grid expansion image corresponding to the vascular segmentation to generate a texture map, map the texture map to the geometric surface of the vascular segment and visualize the attribute information through gradient color coding.

[0097] Furthermore, local 3D geometry visualization can simultaneously or selectively perform visualization options such as visibility control, highlighting, and color attribute coding display on the 3D structures in the selection set. In general, the method for generating a selection set uses a 2D abstract graph as the operation interface. Left-click or select a single or multiple nodes in the 2D abstract graph, and put the vascular segments corresponding to each node in the 2D selection set into the 3D selection set according to the previously defined geometric mapping relationship. The selection set can also be automatically generated using the preoperative planning path, and the vascular segments involved in the surgical path are placed in the selection set. Visualize the vascular structures in the selection set. Perform visibility control or highlight rendering on the vascular structure segments in the selection set, allowing doctors to focus on the local vascular segments of interest. Create special materials or shaders, and use mixed materials, stroke effects, or reflection effects to apply highlight materials or shaders to the vascular structure geometry in the selection set.

[0098] Furthermore, the attribute information of the 3D vascular structure is visualized. For the vascular segment, the attribute value of each key point is queried and calculated from the attribute information of the associated 3D centerline code, and the attribute vector of the middle point is generated by interpolation calculation. Using the texture mapping method, the attribute data is mapped to the pixels of the grid expansion map corresponding to the vascular segment si to generate a texture map. The texture image is then mapped to the geometric surface of the vascular segment to achieve attribute visualization in a gradient color coding manner.

[0099] It should be explained that AR surgical guidance technology can allow doctors to see obscured organs and tissues clearly and reduce surgical risks by superimposing virtual models on the patient in situ. However, in existing surgical AR guidance applications, the guidance information is usually relatively simple, mainly superimposed anatomical structures. The geometric structure and spatial relationship of the human vascular system are extremely complex. If the preoperatively reconstructed vascular structure is simply superimposed on the patient, there will be problems such as visual occlusion, difficulty in operation, poor interactivity, high rendering complexity, and lack of necessary semantic information support.

[0100] Therefore, the present invention proposes to reconstruct a 3D vascular structure model based on the patient's preoperative imaging data, create a corresponding abstract graph, and use the mapping relationship between the abstract graph and the 3D vascular structure to achieve AR visualization interaction, support overview, retrieval, and local operations in the AR environment, and be able to choose the browsing and retrieval method according to the user's intention in the AR scene. Experiments have shown that it can effectively support personalized preoperative AR planning analysis and intraoperative AR guidance for patients, providing solutions for precision surgical diagnosis and treatment.

[0101] like Figure 2 As shown, the second aspect of the present invention discloses an augmented reality visualization system for vascular anatomical structures, the system comprising:

[0102] A 3D modeling module is used to obtain the patient's preoperative imaging data for preprocessing, construct a 3D vascular structure model based on the preoperative imaging data, and perform vascular segmentation processing on the model, wherein the 3D vascular structure model includes a surface model and a voxel model;

[0103] A 2D modeling module, used to create a 2D abstract graph and a mapping relationship between a 3D model and a 2D abstract graph according to the preoperative imaging data and the 3D vascular structure model, wherein the creation task of the 2D abstract graph includes extracting a topological structure and data encoding, and the mapping relationship includes a geometric mapping relationship, an attribute mapping relationship, and an operation mapping relationship;

[0104] A 2D visualization interaction module, for visually presenting the 2D abstract image in an out-of-body manner using an AR device, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure;

[0105] The 3D visualization interaction module is used to uniformly align the 3D vascular structure model reconstructed before the patient's operation with the vascular structure set data collected in real time during the operation in the same coordinate system through a coarse alignment algorithm and a fine alignment algorithm respectively, and use an AR device to visualize the 3D vascular model in the patient's body, perform local geometric visualization of the selected set 3D structure of the 3D vascular model through the interactive interface, and then visualize the attribute information of the 3D vascular structure through texture mapping.

[0106] A third aspect of the present invention discloses an augmented reality visualization device for vascular anatomical structures, comprising:

[0107] at least one processor, and

[0108] a memory communicatively connected to the at least one processor; wherein,

[0109] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the augmented reality visualization method of the vascular anatomical structure as disclosed in any one of the first aspects of the present invention.

[0110] The computer device may be a terminal, and the computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an augmented reality visualization method of a vascular anatomical structure is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0111] A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to execute an augmented reality visualization method for a vascular anatomical structure as described in any one of the first aspect of the present invention.

[0112] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned augmented reality visualization method through the vascular anatomical structure. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] Alternatively, if the above modules of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An augmented reality visualization method for vascular anatomical structure, characterized in that: The method comprises: S1: obtaining preoperative imaging data of the patient for preprocessing, constructing a 3D vascular structure model according to the preoperative imaging data and performing vascular segmentation processing on the model, wherein the 3D vascular structure model includes a surface model and a voxel model; S2: creating a 2D abstract graph and a mapping relationship between the 3D model and the 2D abstract graph according to the preoperative image data and the 3D vascular structure model, wherein the task of creating the 2D abstract graph includes extracting a topological structure and data encoding, and the mapping relationship includes a geometric mapping relationship, an attribute mapping relationship, and an operation mapping relationship; S3: Using an AR device to visualize the 2D abstract image in a patient-external manner, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure; S4: The 3D vascular structure model reconstructed before the operation of the patient and the vascular structure set data collected in real time during the operation are uniformly registered in the same coordinate system by a coarse registration algorithm and a fine registration algorithm respectively, the 3D vascular structure model is visualized in the patient's body by using an AR device, the selected set 3D structure of the 3D vascular structure model is locally geometrically visualized by using the interactive interface, and the attribute information of the 3D vascular structure is visualized and output by texture mapping; The extracting of the topological structure in the 2D abstract graph created according to the preoperative image data and the 3D vascular structure model comprises: S21: extracting the centerline of each vascular segment in the preoperative image data and / or the 3D vascular structure model by a refinement algorithm, and connecting the centerlines of each vascular segment by a centerline completion method to obtain a continuous 3D curve cluster; S22: creating a vascular node for each vascular segment, creating edges of any two adjacent vascular nodes according to the connection relationship between the vascular segments in the 3D curve cluster, and converting the 3D curve cluster into a 2D abstract graph structure by projection and / or flattening; In creating a 2D abstract image based on the preoperative image data and the 3D vascular structure model, data encoding includes: S23: Representing a 2D abstract graph as a tree structure consisting of a set of nodes and edge sets Data Model , where the node set Each node represents a vascular segment in the vascular structure, and the edge set Each edge of Indicates the connection relationship between two adjacent vascular segments; S24: the node The vascular information is stored, and the vascular information includes nodes A 3D model of a corresponding vascular segment, a 3D centerline extracted from the 3D model, and a 2D centerline obtained by projecting and / or flattening the 3D centerline, encoding the attributes of the vascular on the 3D centerline and the 2D centerline, wherein the centerline is defined as a parametric equation of a continuous model and / or a set of key points of a discrete model, and the encoded attribute is defined as a parametric equation representation and / or an attribute vector group representation; The mapping relationship between creating a 3D model and a 2D abstract image according to the preoperative image data and the 3D vascular structure model includes: S25: through the ID relationship between the vascular segmentation and the 2D abstract graph node, each vascular segmentation file of the 3D model is bound to the 2D abstract graph node, the 3D center line corresponding to the vascular segmentation is uniformly sampled, and the coordinate sampling results are stored at discrete points of the 2D center line to establish a geometric mapping relationship between the 3D vascular structure model and the geometric information corresponding to the 2D abstract graph, and the geometric mapping relationship is used to provide data association for AR visualization interaction; S26: Define the number of attribute vectors according to the coded attributes, store the coded attributes of the vascular segmentation in the attribute vector, perform non-uniform key point sampling in the 3D vascular structure model space, the definition of the key points includes the starting point, the end point and the middle point of the vascular segmentation, and store the attribute values ​​of the key points through the attribute vector. The dimension of the attribute vector is the same as the number of key points, so as to establish an attribute mapping relationship between the 3D vascular structure model and the attribute information corresponding to the 2D abstract graph, and provide an interactive interface and data support for attribute visualization for AR visualization interaction through the attribute mapping relationship.

2. The method for augmented reality visualization of vascular anatomical structure according to claim 1, characterized in that: The obtaining of pre-processing of the patient's pre-operative image data, constructing a 3D vascular structure model according to the pre-operative image data and performing vascular segmentation processing on the model comprises: S11: performing filtering processing, artifact elimination and image enhancement on the preoperative image data in sequence; S12: Using convolutional neural network to perform image segmentation on the pre-processed preoperative image data to extract the patient's vascular structure; S13: performing surface reconstruction and voxelization processing on the extracted vascular structure to obtain a 3D vascular structure model including a surface model and a voxel model; S14: Based on the spatial relationship between the segments of the patient's vascular structure, the geometric data of the 3D vascular structure model is organized through a hierarchical structure tree, and then the 3D vascular structure model is segmented.

3. The method for augmented reality visualization of vascular anatomical structure according to claim 1, characterized in that: The method of using an AR device to visualize the 2D abstract image in a patient-external manner, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure includes: S31: The data model All nodes in The corresponding 2D centerline is rendered, and the 2D abstract graph AR visualization of the vascular segment connection relationship is performed through the geometric data of the centerline defined as the parametric equation of the continuous model and / or the key point set of the discrete model; S32: the user customizes the vascular visualization scheme according to the interactive configuration, draws each vascular segment by reading the corresponding attribute vector of the vascular to form a grid expansion diagram of vascular visualization, and performs 2D visualization of vascular geometric attributes through the grid expansion diagram, wherein the 2D visualization of geometric attributes includes visualization of diameter information of the vascular segment and visualization of wall thickness information of the vascular segment; S33: The user obtains a 2D coordinate point set of the target visualization through a point selection operation and / or a box selection operation, queries and / or interpolates the attribute data corresponding to the 2D coordinate point set through an attribute vector to obtain an attribute data result set and outputs and displays it in a list, and the 2D visualization of the non-geometric attributes of the blood vessels is presented in a 2D curve and / or a grid expansion diagram using color coding; S34: using a positioning and tracking sensor device to obtain spatial position information of the end of the surgical instrument, calculating the position of the end of the surgical instrument in the vascular segment and the corresponding 2D abstract map based on the spatial geometric mapping relationship, displaying the end of the surgical instrument in the 2D abstract map through an icon, and marking the movement trajectory of the surgical instrument in the 2D abstract map with a light color and marking the path between the position of the end of the surgical instrument and the lesion with a dark color; S35: Setting each node in the 2D abstract graph as an interactive object through a development engine and / or a development toolkit, and executing interactive visualization of the 2D abstract graph interactive interface and the 3D vascular structure through a script.

4. The method for augmented reality visualization of vascular anatomical structure according to claim 3, characterized in that: The method of using an AR device to visualize the 3D vascular structure model in a patient's body, performing local geometric visualization of a selected set of 3D structures of the 3D vascular structure model through the interactive interface, and then visually outputting the attribute information of the 3D vascular structure through texture mapping includes: S41: selecting a 2D selection set of nodes for local geometric visualization in the 2D abstract graph by point selection and / or box selection, placing a vascular segment corresponding to each node in the 2D selection set into a 3D selection set according to a geometric mapping relationship, generating a selection set by a preoperative planning path and placing the vascular segments associated with the surgical path into the 3D selection set, visualizing the vascular structure in the selection set, and performing display and hiding control and / or highlight rendering display on the vascular structure segments; S42: Query and calculate the attribute values ​​of each key point through the 3D centerline coding attributes associated with the vascular segmentation, use texture mapping to map the attribute information to the pixels of the grid expansion image corresponding to the vascular segmentation to generate a texture map, map the texture map to the geometric surface of the vascular segment and visualize the attribute information through gradient color coding.

5. An augmented reality visualization system for vascular anatomical structures, characterized in that: The system comprises: A 3D modeling module is used to obtain the patient's preoperative imaging data for preprocessing, construct a 3D vascular structure model based on the preoperative imaging data, and perform vascular segmentation processing on the model, wherein the 3D vascular structure model includes a surface model and a voxel model; A 2D modeling module, used to create a 2D abstract graph and a mapping relationship between a 3D model and a 2D abstract graph according to the preoperative imaging data and the 3D vascular structure model, wherein the creation task of the 2D abstract graph includes extracting a topological structure and data encoding, and the mapping relationship includes a geometric mapping relationship, an attribute mapping relationship, and an operation mapping relationship; A 2D visualization interaction module, for visually presenting the 2D abstract image in an out-of-body manner using an AR device, performing intraoperative navigation based on the 2D abstract image, and using the nodes corresponding to the vascular segments in the 2D abstract image as an interactive interface for visualizing the 3D vascular structure; A 3D visualization interaction module is used to uniformly register the 3D vascular structure model reconstructed before the patient's operation with the vascular structure set data collected in real time during the operation in the same coordinate system through a coarse registration algorithm and a fine registration algorithm, visualize the 3D vascular structure model in the patient's body using an AR device, perform local geometric visualization of the selected set 3D structure of the 3D vascular structure model through the interactive interface, and then visualize the attribute information of the 3D vascular structure through texture mapping; The extracting of the topological structure in the 2D abstract graph created according to the preoperative image data and the 3D vascular structure model comprises: Extracting the center lines of each vascular segment in the preoperative image data and / or the 3D vascular structure model by a refinement algorithm, connecting the center lines of each vascular segment by a centerline completion method to obtain a continuous 3D curve cluster; creating a vascular node for each vascular segment, creating edges of any two adjacent vascular nodes according to the connection relationship between the vascular segments in the 3D curve cluster, and converting the 3D curve cluster into a 2D abstract graph structure by projection and / or flattening method; In creating a 2D abstract image based on the preoperative image data and the 3D vascular structure model, data encoding includes: Represents a 2D abstract graph as a tree structure consisting of a set of nodes and edge sets Data Model , where the node set Each node represents a vascular segment in the vascular structure, and the edge set Each edge of Represents the connection relationship between two adjacent vascular segments; the node The vascular information is stored, and the vascular information includes nodes A 3D model of a corresponding vascular segment, a 3D centerline extracted from the 3D model, and a 2D centerline obtained by projecting and / or flattening the 3D centerline, encoding the attributes of the vascular on the 3D centerline and the 2D centerline, wherein the centerline is defined as a parametric equation of a continuous model and / or a set of key points of a discrete model, and the encoded attribute is defined as a parametric equation representation and / or an attribute vector group representation; The mapping relationship between the 3D model and the 2D abstract graph is created based on the preoperative image data and the 3D vascular structure model, including: binding each vascular segmentation file of the 3D model to the 2D abstract graph node through the ID relationship between the vascular segmentation and the 2D abstract graph node, uniformly sampling the coordinates of the 3D centerline corresponding to the vascular segmentation, storing the coordinate sampling results at discrete points of the 2D centerline, so as to establish a geometric mapping relationship between the geometric information corresponding to the 3D vascular structure model and the 2D abstract graph, and providing data association for AR visualization interaction through the geometric mapping relationship; defining the number of attribute vectors according to the coded attributes, storing the coded attributes of the vascular segmentation in the attribute vector, performing non-uniform key point sampling in the 3D vascular structure model space, wherein the definition of the key point includes the starting point, the end point and the middle point of the vascular segmentation, storing the attribute values ​​of the key points through the attribute vector, and the dimension of the attribute vector is the same as the number of key points, so as to establish an attribute mapping relationship between the attribute information corresponding to the 3D vascular structure model and the 2D abstract graph, and providing an interactive interface and data support for attribute visualization for AR visualization interaction through the attribute mapping relationship.

6. An augmented reality visualization device for vascular anatomical structures, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor so as to enable the at least one processor to perform the augmented reality visualization method for a vascular anatomical structure according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the augmented reality visualization method for a vascular anatomical structure according to any one of claims 1 to 4.

Images