A method and apparatus for predicting congenital heart disease

Abstract

The application provides a congenital heart disease prediction method and device, including: obtaining a target three-dimensional heart point cloud of a target patient; inputting the target three-dimensional heart point cloud into a pre-constructed segmentation and classification model, determining a normal vector of each point in the target three-dimensional heart point cloud, and clustering points on the same surface based on the normal vector of each point to determine a three-dimensional structure point cloud of each heart structure; for each heart structure, performing weighted processing on the three-dimensional structure point cloud of the heart structure according to a preset structure weight of the heart structure to determine a three-dimensional weighted structure point cloud of each heart structure; inputting the three-dimensional weighted structure point cloud of each heart structure into the segmentation and classification model for congenital heart disease prediction to determine at least one prediction result. In this way, through the scheme of the application, accurate heart structure segmentation and simultaneous prediction of multiple congenital heart diseases can be performed, thereby effectively improving the accuracy of congenital heart disease recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a congenital heart disease prediction method and device. BACKGROUND

[0002] Congenital heart disease is the most common type of congenital malformation, which refers to the abnormality of anatomical structure caused by the formation obstacle or developmental abnormality of heart and great vessels during embryonic development, or the situation that the channel which should be automatically closed after birth fails to close (normal in fetuses).

[0003] With the vigorous development of computer technology, various artificial intelligence technologies gradually play an important role in human life. Using computer-aided diagnosis technology to assist in the classification and prediction of congenital heart disease has become a new technology means.

[0004] At present, many explorations and researches have been carried out in the research direction of lesion detection based on computer-aided detection at home and abroad. For example, deep neural network is used to classify tasks according to heart point cloud. However, due to the following two characteristics of congenital heart disease: first, the same case may have multiple types of congenital heart disease; second, the heart structure of the same type of congenital heart disease may also be different. Therefore, the accuracy of direct use of deep neural network based on heart point cloud for automatic diagnosis is very low. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a congenital heart disease prediction method and device, which can effectively improve the accuracy of congenital heart disease recognition by accurately segmenting the heart structure and simultaneously recognizing multiple congenital heart diseases.

[0006] The present application provides a congenital heart disease prediction method, which comprises:

[0007] Obtaining a target three-dimensional heart point cloud of a target patient;

[0008] Inputting the target three-dimensional heart point cloud into a pre-constructed segmentation and classification model, determining the normal vector of each point in the target three-dimensional heart point cloud, and clustering the points on the same surface based on the normal vector of each point to determine the three-dimensional structure point cloud of each heart structure;

[0009] For each heart structure, the three-dimensional structure point cloud of the heart structure is weighted according to the preset structure weight of the heart structure to determine the three-dimensional weighted structure point cloud of each heart structure;

[0010] Inputting the three-dimensional weighted structure point cloud including each heart structure into the segmentation and classification model for congenital heart disease prediction to determine at least one prediction result.

[0011] Optionally, the segmentation and classification model is constructed by the following steps:

[0012] a plurality of three-dimensional heart point clouds to be trained with real labels for heart structure segmentation are obtained; wherein the real labels include type labels of heart structures and type labels of congenital heart diseases;

[0013] For each three-dimensional heart point cloud to be trained, the three-dimensional heart point cloud to be trained is sequentially input into an initial segmentation and classification neural network to perform heart structure segmentation and congenital heart disease prediction, and the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained are determined;

[0014] Based on the segmentation result, the prediction result and the real label of the three-dimensional heart point cloud to be trained, a loss function of the initial segmentation and classification neural network is determined;

[0015] According to the loss function, the network parameters of the initial segmentation and classification neural network are updated until the loss function converges, the training is stopped, and the segmentation and classification model is generated.

[0016] Optionally, the three-dimensional heart point cloud to be trained is sequentially input into the initial segmentation and classification neural network to perform heart structure segmentation and congenital heart disease prediction, and the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained are determined, comprising:

[0017] The three-dimensional heart point cloud to be trained is input into a projection weight calculation layer in the initial segmentation and classification neural network for processing, and the first feature and the nonlinear projection weight matrix of the three-dimensional heart point cloud to be trained are output;

[0018] The first feature is input into an edge convolution layer connected with the projection weight calculation layer, and the nonlinear projection weight matrix is respectively input into N edge convolution layers, and the convolution features output by each edge convolution layer are determined through the sequential processing of the N edge convolution layers; wherein the projection weight calculation layer is sequentially connected with N edge convolution layers, and the convolution feature output by each edge convolution layer is determined according to the feature input by the previous connection layer and the nonlinear projection weight matrix;

[0019] The convolution features output by the first preset number of edge convolution layers are combined and input into a segmentation subnetwork in the initial segmentation and classification neural network, and the segmentation result of the three-dimensional heart point cloud to be trained is determined;

[0020] The convolution features output by the N edge convolution layers are combined and input into a classification subnetwork in the initial segmentation and classification neural network, and the prediction result of the three-dimensional heart point cloud to be trained is determined.

[0021] Optionally, the classification sub-network sequentially comprises a first max-pooling layer, a plurality of fully connected layers, and a first activation function layer.

[0022] Optionally, the segmentation sub-network sequentially comprises a first MLP layer, a second max-pooling layer, a plurality of second MLP layers, and a second activation function layer.

[0023] Optionally, the convolutional features output by the first preset number of edge convolutional layers are combined and input into the segmentation sub-network in the initial segmentation and classification neural network, and a segmentation result of the three-dimensional heart point cloud to be trained is determined, comprising:

[0024] The convolutional features output by the first preset number of edge convolutional layers are input into the first MLP layer, and are sequentially processed through the first MLP layer and the second max-pooling layer to output the pooled features of the three-dimensional heart point cloud to be trained; wherein the input of the second max-pooling layer is the output of the first MLP layer.

[0025] The pooled features and the convolutional features output by the first preset number of edge convolutional layers are combined and input into the second MLP layer connected to the second max-pooling layer, and are sequentially processed through the plurality of second MLP layers and the second activation function layer to determine the segmentation result of the three-dimensional heart point cloud to be trained.

[0026] Optionally, the edge convolutional layer sequentially comprises a neighborhood classification layer, a plurality of third MLP layers, and a third max-pooling layer.

[0027] The embodiments of the present application also provide a prediction device for congenital heart disease, the prediction device comprising:

[0028] An acquisition module is configured to acquire a target three-dimensional heart point cloud of a target patient.

[0029] A segmentation module is configured to input the target three-dimensional heart point cloud into a pre-constructed segmentation and classification model, determine normal vectors of points in the target three-dimensional heart point cloud, cluster points on the same surface based on the normal vectors of the points, and determine three-dimensional structure point clouds of heart structures.

[0030] A weighting module is configured to, for each heart structure, perform weighting processing on the three-dimensional structure point cloud of the heart structure according to a preset structure weight of the heart structure, and determine three-dimensional weighted structure point clouds of the heart structures.

[0031] A classification module is configured to input the three-dimensional weighted structure point clouds of the heart structures into the segmentation and classification model to perform prediction of congenital heart disease, and determine at least one prediction result.

[0032] Optionally, the prediction device further comprises a model construction module, and the model construction module is configured to:

[0033] obtain a plurality of three-dimensional heart point clouds to be trained with real labels of heart structure segmentation; wherein, the real labels comprise type labels of heart structures and type labels of congenital heart diseases;

[0034] for each three-dimensional heart point cloud to be trained, sequentially input the three-dimensional heart point cloud to be trained into an initial segmentation and classification neural network, perform heart structure segmentation and congenital heart disease prediction, and determine the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained;

[0035] based on the segmentation result, the prediction result and the real label of the three-dimensional heart point cloud to be trained, determine the loss function of the initial segmentation and classification neural network;

[0036] update the network parameters of the initial segmentation and classification neural network according to the loss function until the loss function converges, stop training, and generate the segmentation and classification model.

[0037] Optionally, when the model construction module is used for sequentially inputting the three-dimensional heart point cloud to be trained into the initial segmentation and classification neural network, performing heart structure segmentation and congenital heart disease prediction, and determining the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained, the model construction module is used for:

[0038] input the three-dimensional heart point cloud to be trained into a projection weight calculation layer in the initial segmentation and classification neural network for processing, and output the first feature and the nonlinear projection weight matrix of the three-dimensional heart point cloud to be trained;

[0039] input the first feature into an edge convolution layer connected with the projection weight calculation layer and input the nonlinear projection weight matrix into N edge convolution layers respectively, and sequentially process through the N edge convolution layers to determine the convolution feature output by each edge convolution layer; wherein, the projection weight calculation layer is sequentially connected with N edge convolution layers, and the convolution feature output by each edge convolution layer is determined according to the feature input by the previous connection layer and the nonlinear projection weight matrix;

[0040] combine the convolution features output by the first preset number of edge convolution layers and input into a segmentation subnetwork in the initial segmentation and classification neural network to determine the segmentation result of the three-dimensional heart point cloud to be trained;

[0041] combine the convolution features output by the N edge convolution layers and input into a classification subnetwork in the initial segmentation and classification neural network to determine the prediction result of the three-dimensional heart point cloud to be trained.

[0042] Optionally, the classification subnetwork sequentially comprises a first max-pooling layer, a plurality of fully connected layers and a first activation function layer.

[0043] Optionally, the segmentation sub-network sequentially comprises a first MLP layer, a second max-pooling layer, a plurality of second MLP layers, and a second activation function layer.

[0044] Optionally, when the model construction module is used to combine the convolution features output by the first preset number of edge convolution layers and input into the segmentation sub-network in the initial segmentation classification neural network to determine the segmentation result of the three-dimensional heart point cloud to be trained, the model construction module is configured to:

[0045] input the convolution features output by the first preset number of edge convolution layers into the first MLP layer, sequentially process the convolution features through the first MLP layer and the second max-pooling layer, and output the pooling features of the three-dimensional heart point cloud to be trained; wherein the input of the second max-pooling layer is the output of the first MLP layer.

[0046] combine the pooling features and the convolution features output by the first preset number of edge convolution layers and input into the second MLP layer connected to the second max-pooling layer, sequentially process the pooling features through the plurality of second MLP layers and the second activation function layer, and determine the segmentation result of the three-dimensional heart point cloud to be trained.

[0047] Optionally, the edge convolution layer sequentially comprises a neighborhood classification layer, a plurality of third MLP layers, and a third max-pooling layer.

[0048] The embodiments of the present application also provide an electronic device, comprising a processor, a memory, and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the prediction method as described above.

[0049] The embodiments of the present application also provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the prediction method as described above.

[0050] The method and device for predicting congenital heart disease provided by the embodiments of the present application comprise the following steps: obtaining a target three-dimensional heart point cloud of a target patient; inputting the target three-dimensional heart point cloud into a pre-constructed segmentation and classification model, determining the normal vector of each point in the target three-dimensional heart point cloud, and clustering the points on the same surface based on the normal vector of each point to determine the three-dimensional structure point cloud of each heart structure; for each heart structure, performing weighted processing on the three-dimensional structure point cloud of the heart structure according to the preset structure weight of the heart structure to determine the three-dimensional weighted structure point cloud of each heart structure; inputting the three-dimensional weighted structure point cloud of each heart structure into the segmentation and classification model to predict congenital heart disease and determine at least one prediction result.

[0051] In this way, by introducing manifold learning, the accurate segmentation of the heart structure and the simultaneous prediction of multiple congenital heart diseases can be performed, so that the accuracy of the recognition of congenital heart disease can be effectively improved.

[0052] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0054] Figure 1 The flowchart of the method for predicting congenital heart disease provided by the embodiments of the present application;

[0055] Figure 2 The flowchart of the method for constructing the segmentation and classification model provided by the embodiments of the present application;

[0056] Figure 3 The structure diagram of the segmentation and classification model provided by the embodiments of the present application;

[0057] Figure 4 The schematic diagram of the heart point cloud neighborhood projection based on the normal vector provided by the embodiments of the present application;

[0058] Figure 5 The structure diagram of the edge convolution layer provided by the present application;

[0059] Figure 6 The schematic diagram of the prediction process of congenital heart disease provided by the embodiments of the present application;

[0060] Figure 7 FIG. 1 is a structural schematic diagram of a congenital heart disease prediction device provided by an embodiment of the present application;

[0061] Figure 8 FIG. 2 is another structural schematic diagram of a congenital heart disease prediction device provided by an embodiment of the present application;

[0062] Figure 9 FIG. 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by a person skilled in the art without creative work falls within the scope of the present application.

[0064] Heart disease is a common circulatory disease. The circulatory system is composed of the heart, blood vessels, and nervous and humoral tissues that regulate blood circulation. Circulatory system diseases, also known as cardiovascular diseases, include diseases of all the above-mentioned tissues and organs, and are common diseases in internal medicine, among which heart disease is the most common and can significantly affect the patient's labor force.

[0065] With the vigorous development of computer technology, various artificial intelligence technologies gradually play an important role in human life. Using computer-aided diagnosis technology to assist in the classification and prediction of congenital heart disease has become a new technology.

[0066] At present, many exploratory and research work has been carried out in the field of computer-aided detection based lesion detection research at home and abroad. For example, deep neural networks are used to classify tasks based on heart point clouds, but congenital heart disease has the following two characteristics: first, the same case may have multiple types of congenital heart disease; second, the same type of congenital heart disease may have different heart structure lesions. Therefore, the accuracy of direct use of deep neural networks based on heart point clouds for automatic diagnosis is extremely low.

[0067] Based on this, the embodiments of the present application provide a congenital heart disease prediction method and device to improve the accuracy of congenital heart disease recognition.

[0068] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for predicting congenital heart disease provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the prediction method includes:

[0069] S101. Obtain the target patient's three-dimensional heart point cloud.

[0070] Here, the target three-dimensional cardiac point cloud of the target patient can be determined by 3D CNN processing based on the CT / MRI images of the target patient.

[0071] The target three-dimensional heart point cloud is a point cloud that includes the entire structure of the heart.

[0072] S102. Input the target three-dimensional heart point cloud into the pre-constructed segmentation and classification model, determine the normal vector of each point in the target three-dimensional heart point cloud, and cluster the points on the same surface based on the normal vector of each point to determine the three-dimensional structural point cloud of each heart structure.

[0073] Here, the segmentation and classification model can segment the various substructures in the heart, and can also predict the congenital heart disease of the target patient based on the segmented heart point cloud.

[0074] The method of clustering points on the same surface based on the normal vectors of each point includes: calculating the normal similarity between neighboring points based on the normal vectors of each point, determining the points on the same surface based on the normal similarity, and clustering the points on the same surface to achieve segmentation of the heart structure.

[0075] It should be noted that the 3D heart point cloud is a sample of a surface model; therefore, the normal of each point in the point cloud is the normal vector of the surface at that point. Normals reflect the geometric information of the surface, so this approach uses point cloud normal vectors to assist in representation learning. During the neighborhood construction process (heart structure segmentation), our method constructs a nonlinear projection based on the normal vectors. Unlike existing methods, our method only selects points on the same continuous surface for clustering to eliminate noise from different heart structure point features, thereby improving segmentation accuracy.

[0076] The neighborhood selection of the existing point cloud analysis method is usually constructed by the Euclidean distance metric, that is, the k nearest points to the center point are selected or all points are selected based on a certain radius, and the projection method is also a Euclidean space projection. In the mainstream point cloud analysis method, the representation learning does not introduce any surface information (such as tangent plane, normal vector, etc.), and when the heart point cloud of a complex structure is represented, the point cloud that is close in Euclidean distance but does not belong to the same heart structure is also used for the center point feature update when the neighborhood is constructed, which introduces noise and leads to poor representation effect, that is, the accuracy of the segmented heart structure is poor.

[0077] The heart structure generally includes the following structures: left ventricle, right ventricle, left atrium, right atrium, myocardium, aorta, and pulmonary artery.

[0078] The segmentation classification model is pre-constructed. Please refer to Figure 2 , Figure 2 The flowchart of the construction method of the segmentation classification model provided by the embodiments of the present application. As shown in Figure 2 The construction method provided by the embodiments of the present application includes:

[0079] S201, a plurality of three-dimensional heart point clouds to be trained with real labels and having been segmented for heart structure are obtained.

[0080] Here, the real label includes a type label of the heart structure and a type label of the congenital heart disease.

[0081] S202, for each three-dimensional heart point cloud to be trained, the three-dimensional heart point cloud to be trained is sequentially input into an initial segmentation classification neural network, and heart structure segmentation and congenital heart disease prediction are performed to determine the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained.

[0082] In this step, each three-dimensional heart point cloud to be trained corresponds to a group of segmentation results and prediction results. The segmentation result includes each heart structure and the type of each heart structure, and the prediction result includes the result of at least one predicted congenital heart disease.

[0083] Among them, the general congenital heart disease includes 17 diseases, and the prediction result can be at least one predicted congenital heart disease, or no congenital heart disease.

[0084] For example, please refer to Figure 3 , Figure 3 The structure diagram of the segmentation classification model provided by the embodiments of the present application. As shown in Figure 3As shown, the segmentation classification model includes one projection weight calculation layer, N edge convolution layers, one first max-pooling layer, M fully connected layers, one first activation function layer, a first first MLP layer, one second max-pooling layer, Q second MLP layers, and one second activation function layer.

[0085] In an implementation, the method further includes:

[0086] S2021, inputting the three-dimensional heart point cloud to be trained into a projection weight calculation layer in the initial segmentation classification neural network for processing, and outputting first features of the three-dimensional heart point cloud to be trained and a nonlinear projection weight matrix.

[0087] Here, the nonlinear projection weight matrix is determined based on cosine similarity between points in the point cloud and a nonlinear function introduced. The first features are determined by feature extraction of each point in the point cloud.

[0088] For example, refer to Figure 4 , Figure 4 The schematic diagram of the heart point cloud neighborhood projection based on the normal vector provided by the embodiments of the present application. Here, combined with Figure 4 The determination process of the nonlinear projection weight matrix in this step is briefly described.

[0089] First, a larger range of candidate neighborhood point set P = {p1,..., pn} is constructed by kNN. n , P represents a neighborhood point cloud set containing n points, wherein the features of all neighborhood points are centered standardized according to the feature f i of the center point p i . i As shown, the tangent plane of the surface where the center point p A is located is A, and the corresponding normal vector is n j . Assuming that any neighborhood point is p B , the corresponding tangent plane is B, and the normal vector is n ij . The angle between the two planes AB is defined as α, and the normal similarity between each point in the candidate neighborhood point set and the center point is calculated, and the nonlinear parameter is introduced to obtain the projection weight W. The specific mathematical expression includes:

[0090] The angle θ between the two tangent planes is calculated based on the normal vector:

[0091]

[0092] Based on the tangent plane included angle θ, the nonlinear projection weight W between the neighbor point and the center point is calculated by the cosine similarity plus a nonlinear function ij :

[0093]

[0094] wherein The specific definition of is:

[0095]

[0096] wherein is an activation function, having a nonlinearity, is a learnable projection function based on a multilayer perceptron. In this way, according to each determined W ij value, the nonlinear projection weight matrix is determined.

[0097] S2022, input the first feature into the edge convolution layer connected with the projection weight calculation layer, and input the nonlinear projection weight matrix into N edge convolution layers respectively, and determine the convolution feature output by each edge convolution layer through the processing of the N edge convolution layers in turn.

[0098] Here, the projection weight calculation layer is sequentially connected with N edge convolution layers, and the convolution feature output by each edge convolution layer is determined according to the feature input by the previous connection layer and the nonlinear projection weight matrix.

[0099] wherein the edge convolution layer sequentially includes a neighborhood classification layer, a plurality of third MLP layers and a third maximum pooling layer.

[0100] For examples, please refer to Figure 5 , Figure 5 The structure diagram of the edge convolution layer provided by the present application is shown in Figure 5 , which includes a neighborhood classification layer, R third MLP layers and a third maximum pooling layer in the edge convolution layer, and is sequentially connected in series. Among them, the algorithm used in the neighborhood classification layer is knn (nearest neighbor algorithm), which constructs the neighborhood. For example, the value of R can be 2.

[0101] For examples, please continue to refer to Figure 3 , as Figure 3 ​As shown, the convolutional features of the output of the edge convolution layer 1 are determined by processing the first features input to the projection weight calculation layer and the nonlinear projection weight matrix, the convolutional features of the output of the edge convolution layer 2 are determined by processing the input convolutional features of the edge convolution layer 1 and the nonlinear projection weight matrix input to the projection weight calculation layer, and the convolutional features of the output of the edge convolution layer N are determined by processing the input convolutional features of the edge convolution layer N-1 and the nonlinear projection weight matrix input to the projection weight calculation layer. For example, N can be 4 here.

[0102] It should be noted that after the nonlinear projection weight matrix in the neighborhood is constructed, feature aggregation needs to be performed on the center point in the neighborhood. To solve the point cloud rotation invariance and better extract features from local to global, the feature aggregation method in the PointManifold based on the edge convolution method of the graph neural network is adopted, and the new features f′ i of the center point p i are obtained based on the feature aggregation method.

[0103]

[0104] Θ=(θ1,...,θ z ,φ1,...,φ z ) are to-be-learned parameters, m represents any subscript in Θ, and z is the number of convolution kernels.

[0105] S2023, combine the convolutional features output by the first preset number of edge convolution layers and input them into the segmentation subnetwork in the initial segmentation and classification neural network to determine the segmentation result of the to-be-trained three-dimensional heart point cloud.

[0106] Here, the number of the first preset number is less than N. For example, assuming that N is 4, the preset number can be set to 3. In this way, combining the convolutional features output by the first preset number of edge convolution layers and inputting them into the segmentation subnetwork in the initial segmentation and classification neural network means inputting the convolutional features output by the first three edge convolution layers into the segmentation subnetwork.

[0107] Here, the segmentation subnetwork sequentially includes a first MLP layer, a second max-pooling layer, a plurality of second MLP layers, and a second activation function layer. For example, as shown in the figure, the connection relationship of the segmentation subnetwork is that the first MLP layer is connected with the second max-pooling layer, the second max-pooling layer is connected with the Q second MLP layers in series, and the last second MLP layer is connected with the second activation function layer. For example, the activation function used by the second activation function layer can be softmax, and the value of Q can be 3. Figure 4

[0108] ​In an embodiment provided in the application, the step of combining the convolution feature outputs of the first preset number of edge convolution layers and inputting the combined convolution feature into a segmentation subnetwork in the initial segmentation and classification neural network to determine the segmentation result of the three-dimensional heart point cloud to be trained comprises: inputting the convolution feature outputs of the first preset number of edge convolution layers into the first MLP layer, sequentially processing the convolution feature outputs through the first MLP layer and a second max-pooling layer, and outputting the pooling feature of the three-dimensional heart point cloud to be trained; wherein the input of the second max-pooling layer is the output of the first MLP layer; inputting the pooling feature and the convolution feature outputs of the first preset number of edge convolution layers into a second MLP layer connected to the second max-pooling layer, sequentially processing the pooling feature and the convolution feature outputs through the plurality of second MLP layers and a second activation function layer, and determining the segmentation result of the three-dimensional heart point cloud to be trained.

[0109] S2024, combine the convolution feature outputs of the N edge convolution layers and input the combined convolution feature into a classification subnetwork in the initial segmentation and classification neural network to determine the prediction result of the three-dimensional heart point cloud to be trained.

[0110] Here, the classification subnetwork sequentially comprises a first max-pooling layer, a plurality of fully connected layers, and a first activation function layer. The activation function in the first activation function layer can be Sigmoid.

[0111] Please continue to refer to Figure 3 The prediction result is sequentially processed through the first max-pooling layer, the M fully connected layers, and the first activation function layer, and the prediction result is output by the first activation function layer. The prediction result can be a multi-label classification result. For example, M can be 2.

[0112] S203, determine the loss function of the initial segmentation and classification neural network based on the segmentation result, the prediction result, and the real label of the three-dimensional heart point cloud to be trained.

[0113] Here, the loss function of the initial segmentation and classification neural network is determined once for each segmentation result and prediction result of the three-dimensional heart point cloud to be trained.

[0114] The cross-entropy can be used as the loss function.

[0115] S204, update the network parameters of the initial segmentation and classification neural network according to the loss function until the loss function converges, stop training, and generate the segmentation and classification model.

[0116] Here, the network parameters of the initial segmentation and classification neural network are updated once for each value of the loss function.

[0117] In this way, when the loss function converges, the training is stopped, that is, the construction of the segmentation classification model is completed.

[0118] After the three-dimensional structure point cloud of each cardiac structure is determined in step S102, the prediction method further includes:

[0119] S103, for each cardiac structure, the three-dimensional structure point cloud of the cardiac structure is weighted according to the preset structure weight of the cardiac structure, and the three-dimensional weighted structure point cloud of each cardiac structure is determined.

[0120] Here, the weight coefficient of each cardiac structure is preset.

[0121] Wherein, the segmentation output is defined as R={r1,…,r7}, wherein r i represents the point set of the i-th cardiac structure, and the subscript corresponds to: 1-left ventricle, 2-right ventricle, 3-left atrium, 4-right atrium, 5-myocardium, 6-aorta, and 7-pulmonary artery. The weight coefficients of each cardiac structure are artificially set as {w p1 ,…,w p7}, so that the three-dimensional weighted structure point cloud obtained after weighting is s i =r i ·w pi ,i∈(1,7). For example, the present application sets W p ={1,1,1,1,0,1,1}.

[0122] S104, input the three-dimensional weighted structure point cloud including each cardiac structure into the segmentation classification model to predict congenital heart disease, and determine at least one prediction result.

[0123] Here, the prediction result is a multi-label classification prediction result.

[0124] For example, please refer to Figure 6 , Figure 6 the schematic diagram of the prediction process of congenital heart disease provided by the embodiment of the present application. As Figure 6 shown, the weighting processing can improve the visualization degree of part of the cardiac structure, while reducing the visualization degree of another part of the cardiac structure, which is more conducive to the prediction of congenital heart disease, and can output a multi-label classification prediction result.

[0125] In addition, in order to prove that the technical solution is better than the prior art in performance, an experiment is performed, specifically, an experiment is performed on the PointCHD dataset, which contains 110 cases, 17 types of congenital heart diseases, contains heart structure annotation and disease category annotation. The experimental results are shown in Table 1 and Table 2. Table 1 is a comparison of the performance of the mainstream point cloud analysis method for heart structure segmentation task on PointCHD, and Table 2 is a comparison of the performance of the mainstream point cloud analysis method for automatic prediction of congenital heart disease type on PointCHD.

[0126] Table 1:

[0127]

[0128] Table 2:

[0129]

[0130] Therefore, according to Table 1 and Table 2, it can be seen that the method proposed in the technical solution has optimal performance compared with the mainstream method in the classification and segmentation task.

[0131] In this way, by introducing manifold learning, accurate heart structure segmentation and simultaneous prediction of multiple congenital heart diseases can be performed, thereby effectively improving the accuracy of congenital heart disease recognition.

[0132] Please refer to Figure 7 、 Figure 8 , Figure 7 is a structural schematic diagram of a congenital heart disease prediction device provided by an embodiment of the application; Figure 8 is a structural schematic diagram of a congenital heart disease prediction device provided by an embodiment of the application. As shown in Figure 7 The prediction device 700 comprises:

[0133] The acquisition module 710 is configured to acquire a target three-dimensional heart point cloud of a target patient.

[0134] The segmentation module 720 is configured to input the target three-dimensional heart point cloud into a pre-constructed segmentation and classification model, determine the normal vector of each point in the target three-dimensional heart point cloud, and cluster the points on the same surface based on the normal vector of each point to determine the three-dimensional structure point cloud of each heart structure.

[0135] The weighting module 730 is configured to, for each heart structure, perform weighting processing on the three-dimensional structure point cloud of the heart structure according to a preset structure weight of the heart structure to determine the three-dimensional weighted structure point cloud of each heart structure.

[0136] The classification module 740 is configured to input a three-dimensional weighted structure point cloud including various cardiac structures into the segmentation and classification model to perform congenital heart disease prediction, and determine at least one prediction result.

[0137] Optionally, as shown in Figure 8 The prediction device 700 further includes a model construction module 750, which is configured to:

[0138] obtain a plurality of three-dimensional heart point clouds to be trained with real labels for cardiac structure segmentation; wherein the real labels include type labels of cardiac structures and type labels of congenital heart diseases;

[0139] For each three-dimensional heart point cloud to be trained, the three-dimensional heart point cloud to be trained is sequentially input into the initial segmentation and classification neural network to perform cardiac structure segmentation and congenital heart disease prediction, and determine the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained;

[0140] Based on the segmentation result, the prediction result and the real label of the three-dimensional heart point cloud to be trained, a loss function of the initial segmentation and classification neural network is determined;

[0141] The network parameters of the initial segmentation and classification neural network are updated according to the loss function until the loss function converges, the training is stopped, and the segmentation and classification model is generated.

[0142] Optionally, when the model construction module 750 is configured to sequentially input the three-dimensional heart point cloud to be trained into the initial segmentation and classification neural network to perform cardiac structure segmentation and congenital heart disease prediction, and determine the segmentation result and the prediction result of the three-dimensional heart point cloud to be trained, the model construction module 750 is configured to:

[0143] input the three-dimensional heart point cloud to be trained into a projection weight calculation layer of the initial segmentation and classification neural network for processing, and output first features and a nonlinear projection weight matrix of the three-dimensional heart point cloud to be trained;

[0144] input the first features into an edge convolution layer connected to the projection weight calculation layer and input the nonlinear projection weight matrix into N edge convolution layers respectively, and sequentially process the N edge convolution layers to determine convolution features output by each edge convolution layer; wherein the projection weight calculation layer is sequentially connected to N edge convolution layers, and the convolution features output by each edge convolution layer are determined according to the features input by the previous connection layer and the nonlinear projection weight matrix;

[0145] combine the convolution features output by the first preset number of edge convolution layers and input into a segmentation subnetwork in the initial segmentation and classification neural network to determine the segmentation result of the three-dimensional heart point cloud to be trained;

[0146] The convolutional features output by the N edge convolutional layers are combined and input into a classification subnetwork in the initial segmentation and classification neural network to determine a prediction result of the three-dimensional heart point cloud to be trained.

[0147] Optionally, the classification subnetwork sequentially includes a first max-pooling layer, a plurality of fully connected layers, and a first activation function layer.

[0148] Optionally, the segmentation subnetwork sequentially includes a first MLP layer, a second max-pooling layer, a plurality of second MLP layers, and a second activation function layer.

[0149] Optionally, when the model construction module 750 is used to combine the convolutional features output by the first preset number of edge convolutional layers and input into a segmentation subnetwork in the initial segmentation and classification neural network to determine a segmentation result of the three-dimensional heart point cloud to be trained, the model construction module 750 is configured to:

[0150] The convolutional features output by the first preset number of edge convolutional layers are input into the first MLP layer, sequentially processed by the first MLP layer and the second max-pooling layer, and output as pooled features of the three-dimensional heart point cloud to be trained; wherein the input of the second max-pooling layer is the output of the first MLP layer.

[0151] The pooled features and the convolutional features output by the first preset number of edge convolutional layers are combined and input into a second MLP layer connected to the second max-pooling layer, sequentially processed by the plurality of second MLP layers and the second activation function layer, and a segmentation result of the three-dimensional heart point cloud to be trained is determined.

[0152] Optionally, the edge convolutional layer sequentially includes a neighborhood classification layer, a plurality of third MLP layers, and a third max-pooling layer.

[0153] Please refer to Figure 9 , Figure 9 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 9. Figure 9 As shown in FIG. 9, the electronic device 900 includes a processor 910, a memory 920, and a bus 930.

[0154] The memory 920 stores machine-readable instructions executable by the processor 910. When the electronic device 900 is running, the processor 910 communicates with the memory 920 through the bus 930. When the machine-readable instructions are executed by the processor 910, the steps in the method embodiments shown in Figure 1 、 Figure 2 、 Figure 6 may be implemented as described in the method embodiments, and will not be described here again.

[0155] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 , Figure 2 , Figure 6 The steps in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting congenital heart disease, characterized in that, The prediction method includes: Obtain the target 3D cardiac point cloud of the target patient; The target 3D heart point cloud is input into a pre-built segmentation and classification model to determine the normal vector of each point in the target 3D heart point cloud. Based on the normal vector of each point, the points on the same surface are clustered to determine the 3D structural point cloud of each heart structure. For each heart structure, the three-dimensional structural point cloud of the heart structure is weighted according to the preset structural weight of the heart structure to determine the three-dimensional weighted structural point cloud of each heart structure. The three-dimensional weighted structural point cloud, including each cardiac structure, is input into the segmentation and classification model to predict congenital heart disease and determine at least one prediction result. The segmentation and classification model is constructed using the following steps: Multiple 3D heart point clouds with real labels and pre-segmented heart structures are obtained for training; wherein, the real labels include heart structure type labels and congenital heart disease type labels. For each 3D heart point cloud to be trained, the 3D heart point cloud to be trained is sequentially input into the initial segmentation and classification neural network to perform heart structure segmentation and congenital heart disease prediction, and the segmentation result and prediction result of the 3D heart point cloud to be trained are determined. The process includes: inputting the 3D heart point cloud to be trained into the projection weight calculation layer of an initial segmentation and classification neural network for processing, outputting a first feature and a nonlinear projection weight matrix of the 3D heart point cloud to be trained; inputting the first feature into a convolutional layer connected to the projection weight calculation layer and inputting the nonlinear projection weight matrix into N convolutional layers respectively, and processing them sequentially through the N convolutional layers to determine the convolutional feature output by each convolutional layer; wherein, the projection weight calculation layer is followed by N convolutional layers sequentially, and the convolutional feature output by each convolutional layer is determined based on the feature input from the previous connected layer and the nonlinear projection weight matrix; combining the convolutional features output by the first preset number of convolutional layers and inputting them into the segmentation subnetwork of the initial segmentation and classification neural network to determine the segmentation result of the 3D heart point cloud to be trained; combining the convolutional features output by the N convolutional layers and inputting them into the classification subnetwork of the initial segmentation and classification neural network to determine the prediction result of the 3D heart point cloud to be trained; Based on the segmentation results, prediction results, and true labels of the three-dimensional heart point cloud to be trained, the loss function of the initial segmentation and classification neural network is determined. The network parameters of the initial segmentation and classification neural network are updated according to the loss function until the loss function converges, at which point training stops and the segmentation and classification model is generated.

2. The prediction method according to claim 1, characterized in that, The classification subnetwork includes, in sequence, a first max pooling layer, multiple fully connected layers, and a first activation function layer.

3. The prediction method according to claim 1, characterized in that, The segmentation subnetwork sequentially includes a first MLP layer, a second max pooling layer, multiple second MLP layers, and a second activation function layer.

4. The prediction method according to claim 3, characterized in that, The step of combining the convolutional features output from a predetermined number of edge convolutional layers and inputting them into the segmentation sub-network of the initial segmentation and classification neural network to determine the segmentation result of the 3D heart point cloud to be trained includes: The convolutional features output by the pre-preset number of edge convolutional layers are input into the first MLP layer, and then processed sequentially through the first MLP layer and the second max pooling layer to output the pooling features of the 3D heart point cloud to be trained; wherein, the input of the second max pooling layer is the output of the first MLP layer. The pooling features and the convolutional features output by the pre-preset number of edge convolutional layers are combined and input into the second MLP layer connected to the second max pooling layer. The mixture is then processed through the multiple second MLP layers and the second activation function layer to determine the segmentation result of the three-dimensional heart point cloud to be trained.

5. The prediction method according to claim 1, characterized in that, The edge convolutional layer sequentially includes a neighborhood classification layer, multiple third MLP layers, and a third max pooling layer.

6. A device for predicting congenital heart disease, characterized in that, The prediction device includes: The acquisition module is used to acquire the target 3D heart point cloud of the target patient; The segmentation module is used to input the target 3D heart point cloud into a pre-built segmentation and classification model, determine the normal vector of each point in the target 3D heart point cloud, and cluster the points on the same surface based on the normal vector of each point to determine the 3D structural point cloud of each heart structure. The weighting module is used to weight the three-dimensional structural point cloud of each heart structure according to the preset structural weight of the heart structure, and to determine the three-dimensional weighted structural point cloud of each heart structure. The classification module is used to input a three-dimensional weighted structural point cloud including various cardiac structures into the segmentation and classification model to predict congenital heart disease and determine at least one prediction result. The prediction device further includes a model building module, which is used for: Multiple 3D heart point clouds with real labels and pre-segmented heart structures are obtained for training; wherein, the real labels include heart structure type labels and congenital heart disease type labels. For each 3D heart point cloud to be trained, the 3D heart point cloud to be trained is sequentially input into the initial segmentation and classification neural network to perform heart structure segmentation and congenital heart disease prediction, and the segmentation result and prediction result of the 3D heart point cloud to be trained are determined. The process includes: inputting the 3D heart point cloud to be trained into the projection weight calculation layer of an initial segmentation and classification neural network for processing, outputting a first feature and a nonlinear projection weight matrix of the 3D heart point cloud to be trained; inputting the first feature into a convolutional layer connected to the projection weight calculation layer and inputting the nonlinear projection weight matrix into N convolutional layers respectively, and processing them sequentially through the N convolutional layers to determine the convolutional feature output by each convolutional layer; wherein, the projection weight calculation layer is followed by N convolutional layers sequentially, and the convolutional feature output by each convolutional layer is determined based on the feature input from the previous connected layer and the nonlinear projection weight matrix; combining the convolutional features output by the first preset number of convolutional layers and inputting them into the segmentation subnetwork of the initial segmentation and classification neural network to determine the segmentation result of the 3D heart point cloud to be trained; combining the convolutional features output by the N convolutional layers and inputting them into the classification subnetwork of the initial segmentation and classification neural network to determine the prediction result of the 3D heart point cloud to be trained; Based on the segmentation results, prediction results, and true labels of the three-dimensional heart point cloud to be trained, the loss function of the initial segmentation and classification neural network is determined. The network parameters of the initial segmentation and classification neural network are updated according to the loss function until the loss function converges, at which point training stops and the segmentation and classification model is generated.

7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the prediction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the prediction method as described in any one of claims 1 to 5.

Images