Heart structure reconstruction neural network training method, heart structure reconstruction method and product

Abstract

The invention discloses a heart structure reconstruction neural network training method, a heart structure reconstruction method and products, and relates to the technical field of artificial intelligence and three-dimensional reconstruction, and the method comprises the steps: obtaining a heart structure three-dimensional template and three-dimensional point cloud data of a heart structure; performing spatial registration on the heart structure three-dimensional template and the three-dimensional point cloud data to obtain registration data; and taking the registration data as input and the target heart structure three-dimensional model as output to train a heart structure reconstruction neural network to obtain a heart structure reconstruction neural model, so that the target heart structure three-dimensional model can be obtained only through one-time reasoning after the model is trained by using a deep learning technology. According to the method, the problems that an existing KBR method needs multiple iterations in each time of modeling, and time consumption is long are solved, the three-dimensional model of the target heart structure is directly obtained through the neural network, and compared with a method based on an implicit function in the prior art, an extraction process is not needed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and three-dimensional reconstruction technology, and in particular to a template-based cardiac structure reconstruction neural network training, a cardiac structure reconstruction method and a product. Background Art

[0002] Echocardiography is one of the most commonly used imaging methods in cardiac examinations. Obtaining a three-dimensional model of the heart structure on an echocardiogram can help doctors more accurately quantify and analyze the patient's cardiac function, such as measuring ventricular volume and ejection fraction. However, ultrasound images are noisy and sparsely sampled, making it difficult to directly obtain a three-dimensional model of the heart structure from an echocardiogram.

[0003] It is an effective method to mark the location of anatomical structures on echocardiograms and reconstruct them based on the coordinates of these locations converted into point clouds in a three-dimensional coordinate system. At present, the main cardiac structure reconstruction method for echocardiograms is the knowledge-based reconstruction algorithm (KBR), which uses a piecewise smooth subdivision surface (PSSS) algorithm to fit a template mesh to a 3D point cloud. However, this method requires the annotation of the features of each edge of the template mesh, which not only requires professional knowledge, but is also time-consuming and laborious. On the other hand, the KBR algorithm requires multiple rounds of iterative optimization in each reconstruction process, which is time-consuming.

[0004] With the development of deep learning technology, neural network and other technologies are also used in surface reconstruction based on point cloud. Existing methods use implicit function fitting + Mesh (3D mesh) extraction to reconstruct the surface of point cloud. These methods first use neural network to fit the implicit function of the object in 3D space (such as sighed / unsined distance function, Occupancy function), and then use marching cube and other algorithms to extract Mesh based on the implicit function. However, these methods cannot directly obtain the 3D model of the target object. Summary of the invention

[0005] The purpose of this application is to provide a cardiac structure reconstruction neural network training, a cardiac structure reconstruction method and a product, which can directly obtain a three-dimensional model of the target cardiac structure without the need for multiple rounds of iterative optimization in each reconstruction process.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a method for training a neural network for cardiac structure reconstruction, the method comprising:

[0008] Acquire a three-dimensional template of a heart structure and three-dimensional point cloud data of the heart structure; the three-dimensional template of the heart structure is a triangular mesh structure;

[0009] Performing spatial registration on the three-dimensional template of the heart structure and the three-dimensional point cloud data to obtain registration data;

[0010] Taking the registration data as input and the target cardiac structure three-dimensional model as output, a cardiac structure reconstruction neural network is trained to obtain a cardiac structure reconstruction neural model; the cardiac structure reconstruction neural network includes a point cloud encoder, a triangular mesh structure encoder, a feature fusion module and a triangular mesh deformation module.

[0011] Optionally, the point cloud encoder is used to extract features of three-dimensional point cloud data in the input registration data to obtain point cloud features;

[0012] The triangular mesh structure encoder is used to extract the features of the three-dimensional template of the heart structure in the input registration data to obtain the template features;

[0013] The feature fusion module is used to fuse the point cloud feature into the template feature to obtain a fused feature;

[0014] The triangular mesh deformation module is used for:

[0015] Predicting the offset of each vertex in the three-dimensional template of the heart structure according to the fusion feature;

[0016] Each vertex of the three-dimensional template of the heart structure is moved according to the offset to obtain a three-dimensional model of the target heart structure.

[0017] Optionally, the point cloud encoder comprises a multi-layer perceptron of channel dimension;

[0018] The triangular mesh structure encoder includes a graph convolutional network;

[0019] The triangular mesh deformation module includes a graph convolutional network.

[0020] Optionally, the loss function of the cardiac structure reconstruction neural model includes: mean square error loss, point-to-triangular mesh distance error in the point cloud, and volume error.

[0021] Optionally, the acquiring of the three-dimensional template of the heart structure and the three-dimensional point cloud data of the heart structure specifically includes:

[0022] Acquire a three-dimensional template of a heart structure; the three-dimensional template of the heart structure is a triangular mesh structure; the triangular mesh structure includes: a vertex matrix, an edge matrix and a face matrix;

[0023] Acquiring echocardiographic data and corresponding three-dimensional spatial information;

[0024] The echocardiographic data is converted into three-dimensional point cloud data according to the three-dimensional spatial information.

[0025] Optionally, after acquiring the three-dimensional template of the heart structure and the three-dimensional point cloud data of the heart structure, the method further includes:

[0026] The three-dimensional template of the heart structure and the three-dimensional point cloud data are normalized, and the translation distance and scaling ratio are recorded.

[0027] In a second aspect, the present application provides a method for cardiac structure reconstruction, the method comprising:

[0028] Acquire three-dimensional point cloud data of the heart structure to be reconstructed;

[0029] The three-dimensional point cloud data of the heart structure to be reconstructed is input into a heart structure reconstruction neural model to obtain a heart structure reconstruction model; the heart structure reconstruction neural model is trained using any of the above-mentioned heart structure reconstruction neural network training methods.

[0030] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-mentioned cardiac structure reconstruction neural network training methods or cardiac structure reconstruction methods.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned cardiac structure reconstruction neural network training methods or cardiac structure reconstruction methods.

[0032] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned cardiac structure reconstruction neural network training methods or cardiac structure reconstruction methods.

[0033] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0034] The present application provides a cardiac structure reconstruction neural network training, cardiac structure reconstruction method and product. First, a cardiac structure three-dimensional template and three-dimensional point cloud data of the cardiac structure are obtained; then the cardiac structure three-dimensional template and the three-dimensional point cloud data are spatially registered to obtain the registered data; then the cardiac structure three-dimensional template is used as input and the target cardiac structure three-dimensional model is used as output to train the cardiac structure reconstruction neural network to obtain a cardiac structure reconstruction neural model, thereby using deep learning technology. After the model is trained, only one inference is required each time to obtain the target cardiac structure three-dimensional model, which solves the problem that the existing KBR method requires multiple rounds of iterations in each modeling and is time-consuming. In addition, in the prior art, the implicit function-based method does not directly obtain the three-dimensional model, but requires an extraction process, which increases the error. The present application uses a neural network to directly obtain the target cardiac structure three-dimensional model without the need for an extraction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 This is a diagram of an application environment of a neural network training method for cardiac structure reconstruction in one embodiment of the present application;

[0037] Figure 2 A flowchart of a neural network training method for cardiac structure reconstruction provided in one embodiment of the present application;

[0038] Figure 3 A schematic diagram of a cardiac structure reconstruction method flow chart provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of a neural network model constructed according to an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the final result of the cardiac structure reconstruction method provided in one embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0044] The cardiac structure reconstruction neural network training method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the three-dimensional template of the heart structure to be processed and the three-dimensional point cloud data of the heart structure to the server 104. After the server 104 receives the three-dimensional template of the heart structure to be processed and the three-dimensional point cloud data of the heart structure, the three-dimensional template of the heart structure is spatially registered with the three-dimensional point cloud data to obtain the registration data; with the registration data as input and the target three-dimensional model of the heart structure as output, the heart structure reconstruction neural network is trained to obtain a heart structure reconstruction neural model; the heart structure reconstruction neural network includes a point cloud encoder, a triangular mesh structure encoder, a feature fusion module, and a triangular mesh deformation module. The server 104 can feed back the obtained heart structure reconstruction neural model to the terminal 102. In addition, in some embodiments, the neural network training method for cardiac structure reconstruction can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the three-dimensional template of the cardiac structure to be processed and the three-dimensional point cloud data of the cardiac structure, or the server 104 can obtain the three-dimensional template of the cardiac structure to be processed and the three-dimensional point cloud data of the cardiac structure from the data storage system and process them.

[0045] At the same time, the cardiac structure reconstruction method provided in the embodiment of the present application can also be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the three-dimensional point cloud data of the heart structure to be reconstructed to the server 104. After the server 104 receives the three-dimensional point cloud data of the heart structure to be reconstructed, it inputs the three-dimensional point cloud data of the heart structure to be reconstructed into the heart structure reconstruction neural model to obtain the heart structure reconstruction model. The server 104 can feed back the obtained heart structure reconstruction model to the terminal 102. In addition, in some embodiments, the heart structure reconstruction method can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly process the three-dimensional point cloud data of the heart structure to be reconstructed, or the server 104 can obtain the three-dimensional point cloud data of the heart structure to be reconstructed from the data storage system and process it.

[0046] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0047] In an exemplary embodiment, Figure 2 As shown, a method for training a neural network for cardiac structure reconstruction is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S1 to S3. Among them:

[0048] S1. Acquire a three-dimensional template of a heart structure and three-dimensional point cloud data of the heart structure; the three-dimensional template of the heart structure is a triangular mesh structure.

[0049] Acquisition of 3D point cloud data: First, ultrasound equipment and sensors are used to collect echocardiographic data and record its 3D spatial information. Then, the coordinates of key anatomical structures are marked on the 2D echocardiogram. Finally, the 2D coordinates are transformed into 3D through the recorded 3D spatial information to obtain 3D point cloud data.

[0050] Specifically, the collected 3D point cloud data consists of two parts: point cloud coordinates and point cloud anatomical structure categories. Assuming there are np points in the point cloud, the point cloud coordinates are represented by a matrix of (np,3) (i.e., np xyz coordinates); the anatomical structure is represented by a matrix of (np,c), which is the one-hot encoding of the point category in the point cloud (c is the number of anatomical structure categories).

[0051] Get the three-dimensional template of the heart structure, which is represented by a mesh (triangular mesh structure) and consists of vertices (vertices, V), edges (edges, E) and faces (faces, F), where V is a (nv, 3) matrix, representing the three-dimensional xyz coordinates of nv points, E is a (ne, 2) matrix, representing the sequence numbers of the two endpoints of ne edges in V, and F is (nf, 3) representing the sequence numbers of the three endpoints of nf triangles in V.

[0052] V matrix: Vertex, vertex, shape is nv rows, 3 columns, (nv: numbervertices). Represents the three-dimensional coordinates of nv points. Each row is the coordinate x, y, z value of a point.

[0053] E matrix: Edge, edge, the shape is ne rows and 2 columns, (ne: number edges). It represents ne edges, and the line connecting two vertices is an edge. The two values ​​in each row are the subscripts of the two vertices corresponding to this edge. The subscript refers to the row number of a vertex in the V matrix.

[0054] F matrix: Face, the shape is nf rows and 3 columns, (nf: number faces). It represents nf faces. A face is a triangle formed by three vertices connected by lines. The three values ​​in each row are the subscripts of the three vertices of the triangle in the V matrix.

[0055] The coordinates of the vertices of the three-dimensional point cloud data and the corresponding three-dimensional template of the heart structure are normalized, and the translation distance and scaling ratio are recorded.

[0056] The heart structure 3D template is constructed by taking the average of all the triangular mesh structures in the training data set. In addition, the category of each vertex in the triangular mesh structure is also marked (there are c possible categories), and each vertex may belong to multiple categories.

[0057] S2. spatially registering the three-dimensional template of the heart structure with the three-dimensional point cloud data to obtain registration data.

[0058] This step is the initial spatial registration, and its purpose is to match the heart structure 3D template with the annotated 3D point cloud data in spatial position. The specific method is to use the ICP (Iterative Closest Point) algorithm to minimize the distance between the vertices and point clouds of the heart structure 3D templates of the same category by rotating, translating, and scaling the heart structure 3D template.

[0059] S3. Taking the registration data as input and the target cardiac structure three-dimensional model as output, a cardiac structure reconstruction neural network is trained to obtain a cardiac structure reconstruction neural model; the cardiac structure reconstruction neural network includes a point cloud encoder, a triangular mesh structure encoder, a feature fusion module and a triangular mesh deformation module.

[0060] The point cloud encoder is used to extract the features of the three-dimensional point cloud data in the input registration data to obtain the point cloud features; the triangular mesh structure encoder is used to extract the features of the three-dimensional template of the heart structure in the input registration data to obtain the template features; the feature fusion module is used to fuse the point cloud features into the template features to obtain the fusion features; the triangular mesh deformation module is used to: predict the offset of each vertex in the three-dimensional template of the heart structure according to the fusion features; move each vertex of the three-dimensional template of the heart structure according to the offset to obtain the target three-dimensional model of the heart structure. Specifically, it includes the following steps:

[0061] The point cloud encoder is composed of a multi-layer perceptron (MLP) in the channel dimension. The input (np,3+c) of this module is concatenated with the point cloud coordinates (np,3) and the category one-hot code (np,c), where np is the number of points in the point cloud, 3 represents the three-dimensional coordinates of the point, and c is the number of point categories. The output is the point cloud feature (np,d), where d is the feature dimension.

[0062] The triangular mesh structure encoder is composed of a graph convolutional network (GCN). The input is the points (V: nm, 3) and edges (E: ne, 2) of the triangular mesh structure, and the output is the features (nm, d) of each point in the triangular mesh structure.

[0063] The feature fusion module fuses the point cloud features into the template features in the form of a triangular mesh. Specifically, the point cloud features of (np, d) are first aggregated in the point dimension to obtain the global point cloud features (d,). Then, the global point cloud features are copied nm times (nm, d) and spliced ​​onto the template features in the form of a triangular mesh to obtain the fused features, that is, the triangular mesh vertex features (nm, 2*d) after fusion.

[0064] The triangular mesh deformation module is also a GCN. It takes the fused features as input, predicts the offset Δ(nm,3) of each vertex of the three-dimensional template of the heart structure, and then moves the vertex position of the three-dimensional template of the heart structure according to Δ to obtain the predicted triangular mesh vertex coordinates, thereby obtaining the target three-dimensional model of the heart structure.

[0065] Among them, the specific method of predicting vertex coordinates based on vertex positions is:

[0066] The prediction process of vertex position is equivalent to moving each vertex from the initial position (x, y, z) to the target position (x', y', z'). Then in the process of moving, the vertex moves x'-x, y'-y, z'-z distances in the direction of the x-axis, y-axis, and z-axis respectively. Assume Δx=x'-x, Δy=y'-y, Δz=z'-z. The specific prediction process is: the input nm vertices are nm three-dimensional coordinates (x, y, z); the model predicts the offset Δx, Δy, Δz corresponding to each vertex in the x-axis, y-axis and z-axis. Then the vertex is finally moved to (x+Δx, y+Δy, z+Δz).

[0067] In order to train better, it is necessary to design a suitable loss function. The loss function of this embodiment is divided into three parts: mean square error loss (MSE), point to mesh distance error (P2MD) in the point cloud, and volume error (VE). Among them, MSE is the mean square error between the predicted triangular mesh vertex coordinates and the true vertex coordinates, P2MD is the mean distance between each point in the point cloud and the target cardiac structure three-dimensional model, and VE is the ratio of the root mean square error between the target cardiac structure three-dimensional model and the true volume to the volume of the target cardiac structure three-dimensional model. The final loss function is the weighted sum of the three loss functions:

[0068]

[0069] Among them, n v is the number of vertices of the triangle mesh, v i and are the true value and predicted value of the triangle mesh vertex coordinates respectively.

[0070]

[0071] Among them, n p is the number of points in the point cloud, p i is a point in the point cloud, is the predicted triangular mesh, For point p i The distance to the predicted triangle mesh.

[0072]

[0073] Where V is the real volume, To predict the volume of the triangular mesh.

[0074] After the model is trained, the trained cardiac structure reconstruction neural model is used to predict the three-dimensional point cloud data, and the predicted results are restored to the actual size and posture based on the translation and scaling information recorded during the normalization process.

[0075] In general, this embodiment can help doctors accurately and intuitively analyze the patient's cardiac function through echocardiography. This method uses the annotated 3D point cloud of the anatomical structure as input, uses a neural network to fit the template model to the target model, and realizes the three-dimensional reconstruction of the heart structure on the echocardiogram. Specifically, this embodiment first collects three-dimensional point cloud data and designs an initial template as the input of the algorithm. Then, a point cloud reconstruction network based on the initial template is built, and the network is trained so that it can extract the features of the template model and the three-dimensional point cloud data, and finally output the target model. Finally, a series of post-processing operations are performed on the model prediction results to obtain a heart structure model of real size and posture.

[0076] In an exemplary embodiment, a method for reconstructing a cardiac structure is provided, the method comprising:

[0077] A1 obtains three-dimensional point cloud data of the heart structure to be reconstructed;

[0078] A2 inputs the three-dimensional point cloud data of the heart structure to be reconstructed into the heart structure reconstruction neural model to obtain the heart structure reconstruction model; the heart structure reconstruction neural model is trained using the heart structure reconstruction neural network training method described in the above embodiment.

[0079] In an exemplary embodiment, a method for reconstructing a cardiac structure is provided. Figure 3 , which comprises the following steps:

[0080] Step 1: Obtain the 3D point cloud data of the heart structure and the target 3D model. Specifically:

[0081] 1.1 The acquisition process of three-dimensional point cloud data is as follows: first, ultrasound equipment is used to collect echocardiograms, 7-15 views from different angles are collected for each case, and the three-dimensional posture information of the ultrasound probe is obtained by using sensors. Then, the coordinates of the anatomical structure points (np, 2) are marked on the 2D echocardiogram, including: tricuspid valve ring, right ventricular septum, right ventricular endocardium, base, apex, pulmonary valve ring, right ventricular septum edge, etc. Finally, the two-dimensional coordinates are transformed into three-dimensional space through the recorded three-dimensional posture information, and the coordinate points of different views of the same heart at the same stage are merged to obtain a three-dimensional point cloud (np, 3) and its category code (np, 7). In this embodiment, multiple views are generally not collected at the same time, and each view is collected separately. The merging of different views here refers to the merging of coordinate points on the video frames representing the same stage (heart contraction to minimum / diastole to maximum moment) in the cardiac cycle.

[0082] 1.2 The target three-dimensional model is represented by a triangular mesh, which is represented by nodes (V: (np, 3)), edges (E: (ne, 2)) and faces (F: (nf, 3)).

[0083] To facilitate subsequent training, all point clouds and mesh coordinates are normalized to [-1, 1] through translation and scaling. It is worth noting that the range after normalization here and the coordinate value range of the template mesh below do not have to be [-1, 1], but can also be [0, 1] or other, as long as different point clouds are normalized to the same range and the template mesh also conforms to this range.

[0084] Step 2: construct a template mesh. The construction process is to average all target meshes in the training data set. The coordinate values ​​of the template mesh are also in the range of [-1, 1].

[0085] Step 3: align the template mesh with the 3D point cloud data. Specifically, the ICP algorithm is used to iteratively optimize the transformation matrix of the template mesh, which minimizes the distance between the template mesh and the 3D point cloud data of the same category.

[0086] Step 4: Build Figure 4 The neural network model shown in the figure specifically includes a point cloud encoder, a template mesh encoder, a feature fusion module and a mesh deformation module. The point cloud encoder and the template mesh encoder extract the features of the input point cloud and the template mesh respectively. The feature fusion module is used to fuse the features of the two. The deformation module predicts the offset of each node of the template mesh based on the fused features. Finally, the mesh nodes are moved according to the offset to obtain the deformed target mesh. The specific structure of each module is as follows:

[0087] 4.1 The point cloud encoder is an MLP for extracting 3D point cloud features. As an example, the present invention uses a three-layer channel-dimensional MLP, with ReLU used as the activation layer and the dimensions of the hidden layers being 32, 64, and 128. Other MLPs or point cloud representation networks (such as point-wise convolution, PointNet) can also be used.

[0088] 4.2 The mesh encoder uses GCN. As an example, the GCN of the present invention includes 6 graph convolution blocks, each of which contains two graph convolution operations, and uses residual connections within the block, and uses ReLU as the activation function. Other graph convolution networks are also applicable.

[0089] 4.3 The feature fusion module is used to fuse point cloud features into Mesh nodes. Specifically, the point cloud features are first globally aggregated in the point dimension. As an example, the present invention uses global maximum pooling to obtain the point cloud global feature: (d,). Then, the feature is copied into nv copies (nv, d) and spliced ​​onto the Mesh feature to obtain the fused mesh feature (nv, 2*d).

[0090] In addition, feature fusion based on the attention mechanism can also be used. The specific approach is: use the point cloud feature as the key, the mesh feature as the query, and calculate the attention. As an example, the present invention uses soft attention (other methods such as multi-head attention, transformer, etc. can also be used). Then, based on the attention, the weighted sum features of the point cloud corresponding to each node of the mesh are obtained and spliced ​​onto the mesh feature. The attention calculation process is:

[0091]

[0092] Among them, Q, K are mesh features and point cloud features, d k is the dimension of the feature vector.

[0093] 4.4 The mesh deformation module uses GCN. As an example, this embodiment uses the same structure as step 4.2 and sets the output dimension to 3 (representing the xyz coordinates of the offset Δ).

[0094] Step 5: Train the above neural network. The specific configuration is: use adamw as the optimizer, use cosine annealing as the learning rate scheduler, the initial learning rate is 0.001, and the batch size is 32. The loss function weight is 1*MSE+1*P2MD+0.5*VE:

[0095] Step 6: Use the trained model to predict and post-process on the test set. Based on the recorded translation and scaling information, restore the predicted mesh to its true size and pose. The final result is as follows: Figure 5 shown.

[0096] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a three-dimensional template of a heart structure and three-dimensional point cloud data of a heart structure or to store three-dimensional point cloud data of a heart structure to be reconstructed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a neural network training method for reconstructing a heart structure or a method for reconstructing a heart structure is implemented.

[0097] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0099] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0100] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0102] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0103] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for training a neural network for cardiac structural reconstruction, characterized in that: The cardiac structure reconstruction neural network training method comprises: Acquire a three-dimensional template of a heart structure and three-dimensional point cloud data of the heart structure; the three-dimensional template of the heart structure is a triangular mesh structure; Performing spatial registration on the three-dimensional template of the heart structure and the three-dimensional point cloud data to obtain registration data; Taking the registration data as input and the target cardiac structure three-dimensional model as output, a cardiac structure reconstruction neural network is trained to obtain a cardiac structure reconstruction neural model; the cardiac structure reconstruction neural network includes a point cloud encoder, a triangular mesh structure encoder, a feature fusion module and a triangular mesh deformation module.

2. The method for training a neural network for cardiac structural reconstruction according to claim 1, characterized in that: The point cloud encoder is used to extract features of the three-dimensional point cloud data in the input registration data to obtain point cloud features; The triangular mesh structure encoder is used to extract the features of the three-dimensional template of the heart structure in the input registration data to obtain the template features; The feature fusion module is used to fuse the point cloud feature into the template feature to obtain a fused feature; The triangular mesh deformation module is used for: Predicting the offset of each vertex in the three-dimensional template of the heart structure according to the fusion feature; Each vertex of the three-dimensional template of the heart structure is moved according to the offset to obtain a three-dimensional model of the target heart structure.

3. The method for training a neural network for cardiac structural reconstruction according to claim 1, characterized in that: The point cloud encoder includes a multi-layer perceptron of channel dimension; The triangular mesh structure encoder includes a graph convolutional network; The triangular mesh deformation module includes a graph convolutional network.

4. The method for training a neural network for cardiac structural reconstruction according to claim 1, characterized in that: The loss function of the neural model for cardiac structure reconstruction includes: mean square error loss, point-to-triangular mesh distance error in the point cloud, and volume error.

5. The method for training a neural network for cardiac structural reconstruction according to claim 1, characterized in that: The step of obtaining the three-dimensional template of the heart structure and the three-dimensional point cloud data of the heart structure specifically includes: Acquire a three-dimensional template of a heart structure; the three-dimensional template of the heart structure is a triangular mesh structure; the triangular mesh structure includes: a vertex matrix, an edge matrix and a face matrix; Acquiring echocardiographic data and corresponding three-dimensional spatial information; The echocardiographic data is converted into three-dimensional point cloud data according to the three-dimensional spatial information.

6. The method for training a neural network for cardiac structural reconstruction according to claim 1, characterized in that: After the three-dimensional template of the heart structure and the three-dimensional point cloud data of the heart structure are obtained, the method further includes: The three-dimensional template of the heart structure and the three-dimensional point cloud data are normalized, and the translation distance and scaling ratio are recorded.

7. A method for cardiac structure reconstruction, characterized in that: The cardiac structure reconstruction method comprises: Acquire three-dimensional point cloud data of the heart structure to be reconstructed; The three-dimensional point cloud data of the heart structure to be reconstructed is input into a heart structure reconstruction neural model to obtain a heart structure reconstruction model; the heart structure reconstruction neural model is trained using the heart structure reconstruction neural network training method according to any one of claims 1 to 6.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cardiac structure reconstruction neural network training method described in any one of claims 1 to 6 or the cardiac structure reconstruction method described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the neural network training method for cardiac structure reconstruction described in any one of claims 1 to 6 or the method for cardiac structure reconstruction described in claim 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the neural network training method for cardiac structure reconstruction described in any one of claims 1 to 6 or the method for cardiac structure reconstruction described in claim 7 is implemented.

Images