Heart point cloud reconstruction method and system for echocardiography

Abstract

The invention provides a heart point cloud reconstruction method and system for echocardiography, and the method comprises the following steps: obtaining three-dimensional heart structure data, and carrying out the preprocessing of the data; reading voxel data of the three-dimensional heart structure data, and performing point clouding to form voxel reconstruction point cloud data; performing standardization processing on the voxel reconstruction point cloud data to obtain all-center point cloud data, and performing segmentation acquisition to obtain standard section point cloud data; establishing a scanning guide network based on the heart structure data to be processed and the standard section point cloud data; and obtaining a test sample, inputting the test section point cloud data in the test sample into the scanning guide network, and outputting the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located. After the scheme is adopted, an echocardiogram scanning guide network based on the point cloud is provided, a complete three-dimensional heart model is constructed, and meanwhile, the tangent plane where the two-dimensional echocardiogram is located is determined in the three-dimensional heart model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent analysis of medical images, and in particular to a method and system for reconstructing cardiac point clouds of an echocardiogram. Background Art

[0002] Echocardiography is a low-cost and radiation-free means of observing cardiac structure and is essential for intelligent analysis of medical images. However, as existing automated algorithms are mainly used to classify standard views and lack the ability to perceive comprehensive three-dimensional structures, the precise positioning of the two-dimensional echocardiographic plane with the dynamic heart requires professional ultrasound physicians. Traditional measurement techniques can already infer three-dimensional cardiac geometry. Although artificial intelligence-based technologies have made progress in spatial awareness, they still cannot provide clear three-dimensional modeling based on two-dimensional images. Although CTA-based digital twin models are expected to achieve the establishment of a three-dimensional model of the heart, their development is hindered by high costs and the presence of radiation. As a cost-effective and radiation-free method, two-dimensional echocardiography still has limitations in providing three-dimensional perception.

[0003] Therefore, a new echocardiographic cardiac point cloud reconstruction method and system is needed to solve the practical needs of analyzing the impact of heart disease, while providing an economical, efficient, and easy-to-operate non-magnetic positioning echocardiographic scanning guidance network to significantly reduce the cost threshold of echocardiographic scanning. Summary of the invention

[0004] In order to overcome the above technical defects, the purpose of the present invention is to provide a cardiac point cloud reconstruction method and system for echocardiography, provide an echocardiography scanning guidance network based on point cloud, construct a complete three-dimensional heart model, and determine the section where the two-dimensional echocardiogram is located in the three-dimensional heart model.

[0005] The present invention discloses a cardiac point cloud reconstruction method for an echocardiogram, comprising the following steps:

[0006] Acquiring three-dimensional cardiac structure data and preprocessing the three-dimensional cardiac structure data;

[0007] Read the voxel data of the three-dimensional heart structure data, and convert the voxel data into point cloud to form voxel reconstruction point cloud data;

[0008] Standardize the voxel-reconstructed point cloud data to obtain the whole-center point cloud data, and segment and collect the whole-center point cloud data to obtain the standard section point cloud data;

[0009] Establishing a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data;

[0010] A test sample is obtained, and the test section point cloud data in the test sample is input into a scanning guidance network, and the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located are output.

[0011] Preferably, the steps of acquiring three-dimensional cardiac structure data and preprocessing the three-dimensional cardiac structure data include:

[0012] Collect cardiac volume data files and clinical data and medical records from multiple hospital units as basic data;

[0013] After preprocessing the basic data, a basic database is established;

[0014] A priori three-dimensional convolutional neural network is used to segment the three-dimensional cardiac structure in the cardiac volume data file in the basic database, and the segmented cardiac area is labeled.

[0015] Preferably, the steps of reading voxel data of three-dimensional cardiac structure data and converting the voxel data into point cloud to form voxel reconstruction point cloud data include:

[0016] Reading voxel data of three-dimensional cardiac structure data, converting the voxel data into a three-dimensional array form, and affine transforming the voxel data in the three-dimensional array form to adjust it to a unified coordinate system;

[0017] For the annotated cardiac chamber structures and arteriovenous structures in the voxel reconstruction point cloud data, the voxels in the eight neighborhoods of the cardiac chamber structures and arteriovenous structures are iterated to determine the cardiac chamber boundary point cloud of the cardiac chamber structure and the arteriovenous boundary point cloud of the arteriovenous structure, and to extract the boundary information of the cardiac chamber structure and the arteriovenous structure;

[0018] Scanning the eight neighborhoods of each pixel in the voxel reconstruction point cloud data to identify the inner point cloud and the outer point cloud relative to the current structure of interest, and defining an initial simulation normal vector, and smoothing the initial simulation normal vector;

[0019] Using the inner point coordinates and outer point coordinates of the inner point cloud and the outer point cloud in a unified coordinate system, respectively, and the initial simulation normal vector, the voxel data is gridded to obtain gridded data, and the original voxel data is retained;

[0020] For the gridded data, curvature-based point cloud resampling is used to normalize the number of cardiac chamber structure points and arteriovenous structure points in the gridded data to obtain voxel-reconstructed point cloud data.

[0021] Preferably, the steps of standardizing the voxel reconstructed point cloud data to obtain the whole-center point cloud data, and segmenting and collecting the whole-center point cloud data to obtain the standard section point cloud data include:

[0022] Perform point cloud pose correction and point cloud size correction on the voxel-reconstructed point cloud data to obtain standardized full-center point cloud data, so as to reduce the vertex coordinate values ​​of the voxel-reconstructed point cloud data;

[0023] Based on the standard sections involved in the cardiac two-dimensional medical imaging standard, the whole heart point cloud data is segmented and collected to obtain standard section point cloud data.

[0024] Preferably, the step of establishing a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data comprises:

[0025] Based on the PCN network, a multi-branch completion network is established by combining self-supervised learning and multi-branch reconstruction loss;

[0026] The cardiac structure data to be processed and the standard section point cloud data are divided into a training set, a validation set and a test set;

[0027] The three-dimensional heart structure data and the section point cloud where the standard section point cloud data are located are used as the target point cloud, and the two-dimensional point cloud of the standard section point cloud data is used as the input point cloud. The multi-branch completion network is trained to obtain the scanning guidance network.

[0028] Preferably, the step of taking the three-dimensional heart structure data and the section point cloud where the standard section point cloud data is located as the target point cloud and the two-dimensional point cloud of the standard section point cloud data as the input point cloud, training the multi-branch completion network, and obtaining the scanning guidance network includes:

[0029] Resample the 3D cardiac structure data and standard section point cloud data and organize them into sizes that can be received by the multi-branch completion network;

[0030] Initialize the multi-branch completion network;

[0031] Set the pre-hyperparameters of the multi-branch completion network, where the pre-hyperparameters include pre-weights and post-weights. In the pre-weights, the weight of the loss function on the coarse point cloud branch of the standard section point cloud data is increased, and in the post-weights, the weight of the loss function on the fine point cloud branch of the standard section point cloud data is increased.

[0032] The Adam optimizer is used to optimize the multi-branch completion network, and the network parameters with the smallest L1 chamfer distance on the validation set data are selected for storage.

[0033] Preferably, the loss function L compare for:

[0034] L total =L coarse +αL fine +αβL compare

[0035] Among them, α and β are pre-hyperparameters; the value of α increases with the progress of training, and is 0.01 when the number of training steps is less than 100, and is 0.1 when the number of training steps is more than 200; the value of β is fixed at 0.1; L coarse It includes the chamfer loss of the sparse point cloud of the six cardiac output structures, the chamfer loss of the reconstructed and rotated cross-sections, and the loss of the overall shape branch; L fine is the chamfer loss of dense point cloud; L compare To compare the reconstruction loss, it is used to measure the difference between the point clouds of the six three-dimensional cardiac structure data generated separately and the whole heart point cloud data output by the local generation module.

[0036] Preferably, the loss function L compare The three-part loss function L in coarse , L fine and L compare The calculation formulas are:

[0037] L coarse =∑ x∈Ω L CD (x,x gt )+L CD (Y coarse ,Y gtShape )+L CD (Y View ,X)+

[0038] L CD (Y Rotate ,Y gtView )

[0039] L fine =L CD (Y Shape ,Y gtShape )+L CD (Y Component ,Y gtShape )

[0040] L compare =L CD (Y Component ,Y Shape )

[0041] Where Ω is the rough point cloud branch set {S lv ,S la ,S rv ,S ra ,S myo ,S aorta}, x gt is the real data point cloud corresponding to x, and

[0042] L coarse , L fine and Lcompare The calculations are based on the chamfer distance under the L1 norm, where the output point cloud S is calculated. output and the real data point cloud S gt Chamfer distance L CD (S output ,S gt ) is:

[0043]

[0044] The L1 chamfer distance is calculated by the following formula:

[0045]

[0046] Among them, S1 and S2 are two point clouds for calculating the chamfer distance.

[0047] Preferably, the steps of obtaining a test sample, inputting the test section point cloud data in the test sample into a scanning guidance network, and outputting the three-dimensional cardiac structure data of the test sample and the cardiac point cloud data of the section where the test section point cloud data is located include:

[0048] Acquire a test sample, input the test section point cloud data in the test sample into a scanning guidance network, output the three-dimensional heart structure data of the test sample, locate the test section in the three-dimensional heart structure, and verify the accuracy of the test section point cloud data inferring the position of the heart valve;

[0049] Evaluation criteria were used to evaluate the performance of the scan-guided network in the shape reconstruction task and the view localization task, including the chamfer distance based on the L1 norm, the chamfer distance based on the L2 norm, and the FScore with a threshold of 2 mm.

[0050] Specifically, the present invention first provides a standardized method for collecting and processing three-dimensional cardiac point cloud data, which specifically includes the following process:

[0051] 1. Collection, organization and annotation of cardiac structure data; details are as follows:

[0052] (1) Data collection: Collect sufficient cardiac volume data files and clinical data medical records from multiple hospitals at different levels and in different regions; for example, collect DICOM files and corresponding medical records of cardiac volume data from central hospitals in multiple cities and towns.

[0053] (2) Data organization: This includes steps such as data cleaning, renaming, and format conversion. The preprocessed data will be reviewed and used to establish a basic database.

[0054] (3) Data labeling: including labeling process confirmation, labeling object and labeling task confirmation, labeling rule clarification, labeling quality control and evaluation, and data statistics.

[0055] Annotation process: clarify the annotation object, personnel division of labor, annotation steps, result review, unify the annotation rules, confirm the meaning, form and output format of the annotation results, and describe the decision-making mechanism;

[0056] Labeling objects and labeling tasks: clarify the labeling objects and labeling tasks. In particular, the present invention automatically segments CTA images by utilizing a previously verified three-dimensional convolutional neural network. The network provides ten labels including the left atrium, left ventricle, right ventricle, right atrium, left ventricular myocardium, aorta, and left and right pulmonary arteries. The outline results of senior physicians whose professional level meets the recognized standards in the industry are used as the gold standard, and they must be reviewed and confirmed by the reviewers before proceeding to the next step. After labeling with the outline software, export using the nii format.

[0057] Annotation quality control: During the annotation process, the annotation quality of the annotators should be supervised, and both consistency and accuracy indicators should be considered. Senior radiologists further visually checked the quality of the 3D segmentation results to ensure that they met strict standards. After annotation, the accuracy, consistency, comprehensibility, accessibility, data security, and traceability were evaluated.

[0058] 2. Obtain the labeled voxel data and convert it into point cloud: including voxel data reading, intersection point cloud sampling, normal generation and correction, grid point cloud, and point cloud resampling. The details are as follows:

[0059] (1) Voxel data reading: First, the voxel data of the labeled structure (such as nii data) is read and converted into a three-dimensional array. Then, the three-dimensional array is affine transformed according to the stored affine matrix to adjust the data to a unified coordinate system, providing a basis for subsequent processing.

[0060] (2) Junction point cloud sampling: Each annotated cardiac chamber and arteriovenous structure is considered a positive sample, and all voxels in its eight-neighborhood are iterated in turn. By identifying the coordinates of voxels with negative labels, the boundary point cloud of each structure is determined, so that the junction information between adjacent structures can be accurately extracted. Different structures are stored as different color information to achieve distinction. In particular, the valve point is located at the junction of the chamber annotations, where the mitral valve takes the junction point of the left ventricle and left atrium, the tricuspid valve takes the junction point of the right ventricle and right atrium, and the aortic valve takes the junction point of the left ventricle and aorta.

[0061] (3) Normal generation and correction: In order to improve the quality of point cloud data, the present invention introduces the normal generation and correction steps. Specifically, based on the current voxel data, the eight-connected domain of each pixel is scanned to identify the inner and outer points relative to the current structure of interest. Then, the direction from the centroid of the inner point to the centroid of the outer point is defined as the initial simulated normal vector. Furthermore, the minimum spanning tree algorithm is used to smooth the simulated normal vector to obtain more accurate normal vector information.

[0062] (4) Meshing point cloud: Using the obtained point cloud coordinates and their corrected normal vector information, the point cloud is meshed using triangulation or Poisson reconstruction algorithm. This process is performed one by one in a structural order to ensure that the reconstructed mesh not only conforms to the geometric characteristics of the original data, but also retains rich detail information. At the same time, in order to facilitate subsequent processing, the present invention also retains the original point cloud data.

[0063] (5) Point cloud resampling: For the grid data, curvature-based point cloud resampling is implemented to standardize the number of points of each structure and obtain voxel-reconstructed point cloud data. This operation can automatically adjust the sampling density according to the curvature change of the grid surface, thereby effectively reducing data redundancy and improving processing efficiency while maintaining important geometric features. Specifically, the structures included in the resampled data are: left ventricle, left ventricular wall, left atrium, mitral valve, right ventricle, right atrium, tricuspid valve, aorta and aortic valve.

[0064] 3. Full-center point cloud standardization and section point cloud generation: including point cloud standardization, standard section point cloud collection and other steps. The details are as follows:

[0065] (1) Point cloud standardization includes point cloud pose correction and point cloud size correction. Among them, point cloud pose correction involves the regulation and unification of the spatial coordinate system, and point cloud size correction involves strengthening the mean consistency of the coordinates of points in different sample point clouds, which can reduce the difficulty of echocardiography scanning guidance network fitting. In particular, the present invention uses the centroid of the whole heart structure as the origin of the spatial coordinate system, the direction from the centroid of the tricuspid valve to the centroid of the mitral valve as the x-axis of the spatial coordinate system, the vector from the centroid of the left ventricle to the apex as the y-axis of the spatial coordinate system, and the outer product of the vectors in the x-axis and y-axis directions as the z-axis of the spatial coordinate system, and automatically extracts the corresponding affine matrix according to the structural calibration. After calibrating the pose, the vertex coordinate values ​​of the point cloud data can be reduced to obtain standardized whole heart point cloud data;

[0066] (2) Standard section point cloud acquisition: Based on the standard sections involved in two-dimensional cardiac medical imaging, the standardized whole-heart point cloud data is segmented and acquired to obtain the standard section point cloud. Specifically, the plane equation is solved based on the three non-collinear key points of each standard section, and the standardized point cloud data is traversed to sample the whole-heart point cloud using the inequality of the plane standard equation. In particular, the section geometry formula can be used to sample the whole-heart point cloud to obtain the standardized whole-heart point cloud.

[0067] Taking the commonly used standard sections of ultrasound as an example, according to the ultrasound scanning guidelines, the key points of the apical four-chamber view are the centroid of the mitral valve, the centroid of the tricuspid valve, and the left ventricular apex; the key points of the parasternal long-axis view are the centroid of the left ventricle, the centroid of the aortic valve, and the centroid of the mitral valve; the key points of the apical two-chamber view are the centroid of the left ventricle, the left ventricular apex, and the centroid of the mitral valve.

[0068] After the above steps, the standardized full-center point cloud and the standardized section point cloud can be obtained.

[0069] The present invention further builds an innovative echocardiography scanning guidance network, and implements training and testing based on the above point cloud data. Specifically, it includes the following process:

[0070] 1. Build a convolutional neural network using a codec structure as an echocardiography scanning guidance network for 3D reconstruction of cardiac structure and view positioning; the details are as follows:

[0071] In the radar data processing process of existing autonomous driving technology, the point cloud completion network is often used to complete the point cloud of a single-angle scanned surface. However, the point cloud completion involved in the present invention is different from completing the topological structure of the three-dimensional spherical surface homeomorphism from a single angle, but infers the spatial position of the section and the structure of the entire heart based on the section information. Therefore, the point cloud completion network of the prior art cannot be directly transferred to the present invention.

[0072] The present invention proposes an innovative point cloud completion network, which is a convolutional neural network with a codec structure. In the network, the PCN structure is used as an encoder for feature extraction, the point cloud data is encoded into a 1024-dimensional feature vector, and a parallel multi-decoder is connected after the PCN structure to perform a variety of different point cloud completion tasks.

[0073] The whole heart point cloud involved in the present invention is composed of six components, including the left ventricle, left atrium, myocardium, right ventricle, right atrium and aorta. When considering a single structure, the myocardium exhibits a significant concave feature. When considering the overall structure, the structures near the valve ring are coupled. In order to accurately depict the structure of each component, the present invention uses a component decoder and a local generation block for structural reconstruction. The component decoder generates point clouds for six heart structures, while the rough shape decoder directly generates a new shape point cloud. After processing by the local generation block, the whole heart structure point cloud can be characterized from two perspectives.

[0074] The present invention also uses a view reconstruction branch to enhance the network's understanding of the input slice. Since the input slice is cropped according to the echocardiographic sector and window, the branch used for slice positioning in the network actually assumes the functions of slice positioning and slice structure completion at the same time. In addition, the cropping mode of the slice is related to the acoustic window and contains the slice information. Therefore, the present invention designs a slice reconstruction branch to enhance the stability of the slice shape and make the network more sensitive to structural defects related to the slice angle, so that the acoustic window and slice angle can be more accurately located.

[0075] Based on the above innovative means, the parallel multi-decoder structure adopted in the present invention is mainly divided into four parts: the view completion and rotation decoder is used to generate the three-dimensional point cloud of the section plane where the echocardiogram is located; the component decoders respectively generate the component point clouds of the six parts of the heart, and the rough shape decoder directly generates the rough point cloud of the entire heart shape. The results of these two parts of the decoder are finally used to generate the full heart shape point cloud through the local generation module; the view reconstruction decoder is used to reconstruct the section to enhance the stability of the section shape and make the network more sensitive to structural defects related to the section angle, thereby achieving more accurate acoustic window and section angle positioning.

[0076] 2. Divide the standardized full-center point cloud and standardized section point cloud to be processed into a training set, a validation set, and a test set;

[0077] In order to avoid different samples of the same case appearing in the training set and test set respectively, it is necessary to separate the case samples according to their sources, and divide the to-be-processed standardized whole-center point cloud and standardized section point cloud into training set, validation set and test set.

[0078] 3. Using the standardized whole heart point cloud and the three-dimensional standardized section point cloud as the target point cloud and the two-dimensional standardized section point cloud as the input point cloud, the echocardiography scanning guidance network is trained to obtain a network model for reconstructing the three-dimensional heart structure and view positioning;

[0079] (1) Resample the training set to the size that is input into the echocardiography scanning guidance network;

[0080] (2) Initializing the echocardiographic scanning guidance network;

[0081] (3) Setting the pre-hyperparameters of the echocardiography scanning guidance network. Specifically, increasing the weight of the loss function for the rough point cloud branch in the early stage, and increasing the weight of the loss function for the fine point cloud part in the later stage.

[0082] Furthermore, the loss function L compare for:

[0083] L total =L coarse +αL fine +αβL compare

[0084] Among them, α and β are hyperparameters; the value of α increases with the progress of training, and is 0.01 when the number of training steps is less than 100, and is 0.1 when the number of training steps is more than 200; the value of β is fixed at 0.1; L coarse It includes the chamfer loss of the sparse point cloud of the six output structures, the chamfer loss of the reconstructed and rotated cross-sections, and the loss of the overall shape branch; L fine is the chamfer loss of dense point cloud; L compare To compare the reconstruction loss, the difference between the six heart structure point clouds generated separately and the full heart shape point cloud output by the local generation module is measured, so that the two branches can better promote each other. The calculation of these three parts of the error is based on the chamfer distance under the L1 norm. Calculate the output point cloud S output and the real data point cloud S gt Chamfer distance L CD (S output ,S gt ) is:

[0085]

[0086] Loss function L compare The three-part loss function L in coarse , L fine and L compare The calculation formulas are:

[0087] L coarse =∑ x∈Ω L CD (x,x gt )+L CD (Y coarse ,Y gtShape )+L CD (Y View ,X)+

[0088] L CD (Y Rotate ,Y gtView)

[0089] L fine =L CD (Y Shape ,Y gtShape )+L CD (Y Component ,Y gtShape )

[0090] L compare =L CD (Y Component ,Y Shape )

[0091] Where Ω is the coarse component set {S lv ,S la ,S rv ,S ra ,S myo ,S aorta}, x gt is the real data point cloud corresponding to x.

[0092] (4) The Adam optimizer is used to optimize the echocardiography scanning guidance network, and the network parameters with the best performance on the validation set are selected for storage; in particular, the present invention uses the chamfer distance between two point clouds as an indicator to measure performance, namely:

[0093]

[0094] Among them, d CD (S1, S2) is the chamfer distance between two point clouds; S1 and S2 are two point clouds for calculating the chamfer distance, which are the predicted point cloud inferred by the network in the verification dataset and the actual heart structure point cloud and section positioning point cloud in this applicable scenario; x is a point in the S1 point cloud, and y is a point in the S2 point cloud.

[0095] 4. Obtain test samples, use the trained echocardiography scan-guided network model to reconstruct the heart model, predict the spatial location of the echocardiogram, and verify the accuracy of the point cloud section to infer the echocardiography view position. Specifically include:

[0096] (1) Obtaining test samples, and using the model to predict the cardiac structure point cloud and view point cloud after the echocardiogram scanning guidance network converges, to obtain the cardiac structure and the specific location of the echocardiogram in the heart;

[0097] (2) Calculate the L1 chamfer distance, L2 chamfer distance and FScore between the predicted point cloud and the real point cloud.

[0098] The acquired three-dimensional point cloud is compared with the point cloud of the actual medical image data, and the error between the actual inferred heart model and the actual model and the error between the inferred echocardiogram position and the actual position are calculated.

[0099] The present invention further provides an echocardiography scanning guidance network system based on point cloud, which includes a section point cloud acquisition module, a heart three-dimensional shape reconstruction module, an echocardiography view positioning module, an efficiency evaluation module and a drawing module; wherein,

[0100] (1) A section point cloud acquisition module is used to acquire a set of two-dimensional matrices to be processed of image data of a medical image, and pre-process the set of matrices to be processed to obtain the region of interest of the medical image and the category label corresponding to the region of interest, sample and acquire the contour point cloud of the region of interest, and affine transform the contour point cloud of the region of interest into three-dimensional space coordinates through the average plane equation and centroid coordinates of the relevant standard sections in the standardized point cloud data set obtained during training to form a three-dimensional section point cloud; based on the standard sections involved in two-dimensional medical imaging of the heart, the standardized whole heart point cloud data is segmented and collected to obtain a two-dimensional standard section point cloud;

[0101] (2) A heart 3D shape reconstruction module, which inputs the acquired 2D section point cloud into a trained echocardiography scanning guidance network to obtain an inferred heart 3D shape point cloud;

[0102] (3) an echocardiographic view localization module, which inputs the acquired two-dimensional section point cloud into a trained echocardiographic scanning guidance network to obtain a three-dimensional point cloud of the inferred section plane of the echocardiogram;

[0103] (4) Performance evaluation module, which compares the acquired 3D point cloud with the actual medical image data, and uses a variety of indicators to evaluate the performance of the model in the shape reconstruction task and the view localization task, including chamfer distance based on L1 norm, chamfer distance based on L2 norm, and FScore with a threshold of 2 mm. The view localization function of the model is evaluated using degree error and center distance error of the view plane;

[0104] (5) A drawing module that draws an image based on the three-dimensional shape point cloud and view point cloud of the heart generated by each two-dimensional section.

[0105] The present invention also discloses a cardiac point cloud reconstruction system for an echocardiogram, comprising:

[0106] An acquisition module, which acquires three-dimensional cardiac structure data and pre-processes the three-dimensional cardiac structure data;

[0107] The reconstruction module reads the voxel data of the three-dimensional cardiac structure data and converts the voxel data into point cloud to form voxel reconstruction point cloud data;

[0108] The segmentation module performs standardization processing on the voxel-reconstructed point cloud data to obtain the whole-center point cloud data, and segments and collects the whole-center point cloud data to obtain the standard section point cloud data;

[0109] A network construction module, which establishes a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data;

[0110] The drawing module obtains the test sample, inputs the test section point cloud data in the test sample into the scanning guidance network, and outputs the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located.

[0111] Compared with the prior art, the above technical solution has the following beneficial effects:

[0112] 1. This invention innovatively introduces a new echocardiography scanning guidance network technology, which is based on the powerful ability of deep learning, and has achieved breakthrough results by introducing a point cloud-based, weakly supervised, single-view 3D echocardiography generation network and a comprehensive processing flow tailored for it;

[0113] 2. Point cloud data, as a collection of vectors in a three-dimensional coordinate system, is widely used in fields such as autonomous driving. Its completion technology is an important generation and estimation problem, which has a profound impact on the field of 3D computer vision. Although the existing technology (such as the completion technology for radar data in the field of autonomous driving) can usually only complete the point cloud of a single-angle scanned surface, the present invention specifically proposes a new type of weakly supervised three-dimensional generation network based on point cloud for echocardiography. The network automatically performs three-dimensional heart inference based on two-dimensional echocardiography and section tracking;

[0114] 3. To further improve accuracy, a self-supervised learning branch is integrated into the model framework, which introduces multi-structure reconstruction loss and an overall reconstruction loss designed specifically for cardiac structure completion. In addition, the present invention also constructs a comparison branch to enhance the accuracy of the network in inferring cardiac structure, thereby improving the authenticity of the generated 3D model;

[0115] 4. It can perform 3D heart modeling in real time and stably, and does not rely on paired data, thereby promoting the progress of echocardiography digital twin heart research. Based on the above innovative ideas, the present invention proposes an economical, efficient and easy-to-operate echocardiography digital twin heart. Figure 3 The 3D spatial positioning and dynamic cardiac structure reconstruction solutions realize fully automatic, fast and highly robust 3D inference of cardiac structure, can generate real-time 3D cardiac models, and perform section positioning during scanning, providing strong support for clinical decision-making;

[0116] 5. The present invention uses contrast reconstruction loss to enhance the accuracy of the network in inferring cardiac structure, while achieving the decoupling of the multi-structure point cloud of the heart. Compared with other models, the present invention achieves the best performance on the test set;

[0117] 6. The present invention solves the problem in the prior art that it is difficult to confirm the relative position in space during the echocardiography scanning process, and is helpful for the three-dimensional reconstruction of the heart structure without magnetic positioning and the calibration of the heart structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 A schematic diagram of a process flow of a cardiac point cloud reconstruction method for echocardiography in accordance with a preferred embodiment of the present invention;

[0119] Figure 2a A schematic diagram of voxel data of three-dimensional cardiac structure data in accordance with a preferred embodiment of the present invention;

[0120] Figure 2b A three-dimensional voxel mask of a heart component in accordance with a preferred embodiment of the present invention;

[0121] Figure 2c A point cloud image of three-dimensional cardiac structure data in accordance with a preferred embodiment of the present invention;

[0122] Figure 2d A point cloud image of a two-dimensional point cloud that conforms to standard section point cloud data in a preferred embodiment of the present invention;

[0123] Figure 3 A network structure diagram of an echocardiographic scanning guidance network in accordance with a preferred embodiment of the present invention;

[0124] Figure 4 A schematic diagram of three-dimensional cardiac structure data of a test sample and cardiac point cloud data of a section where test section point cloud data is located in accordance with a preferred embodiment of the present invention, the schematic diagram includes images of section point cloud, inferred section position point cloud, real section position point cloud, inferred cardiac structure point cloud and real cardiac point cloud; DETAILED DESCRIPTION

[0125] The advantages of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments.

[0126] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0127] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0128] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0129] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0130] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0131] In the following description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module" and "component" can be used interchangeably.

[0132] See also Figure 1 , is a flow chart of a cardiac point cloud reconstruction method for echocardiography in accordance with a preferred embodiment of the present invention. In this embodiment, the cardiac point cloud reconstruction method includes the following steps:

[0133] S100: Acquire three-dimensional cardiac structure data, and pre-process the three-dimensional cardiac structure data;

[0134] Acquiring 3D cardiac structure data and preprocessing it is one of the important steps in conducting heart-related research. 3D cardiac structure data can be obtained through medical imaging technology (such as MRI and CT), providing detailed anatomical structure and tissue information about the heart.

[0135] First, obtaining three-dimensional cardiac structure data usually involves medical imaging scans of patients. MRI technology can provide high-resolution, multi-sequence cardiac images, showing details such as the heart's soft tissue structure, heart chambers, and heart walls. CT technology can provide high-contrast cardiovascular images, including anatomical information of coronary arteries and other blood vessels.

[0136] Once the 3D cardiac structure data is obtained, it needs to be preprocessed to obtain high-quality data that can be used for analysis. The goal of preprocessing is to remove noise, enhance image contrast, perform image registration and other operations to improve the quality and consistency of the data.

[0137] First of all, image denoising is one of the key steps in preprocessing. Since medical image data is often affected by artifacts, motion artifacts and noise, the application of denoising algorithms such as Gaussian filtering, median filtering or wavelet denoising can reduce the interference of noise on image quality and analysis results.

[0138] Secondly, image enhancement can be achieved through techniques such as histogram equalization, contrast stretching, filter design, etc. These methods can increase the contrast and details of the image, making the cardiac structure more clearly visible.

[0139] In addition, image registration is also an important part of preprocessing. Image registration can align cardiac images from different scanning sequences or at different time points to achieve a consistent spatial reference. Common registration methods include feature point-based registration, mutual information-based registration, and elastic deformation models.

[0140] Finally, during the preprocessing process, image segmentation and 3D reconstruction can be performed to further analyze specific structures and regions of the heart. These operations can be implemented using traditional methods such as thresholding and edge detection, or with the help of deep learning image segmentation networks.

[0141] In short, acquiring 3D cardiac structure data and preprocessing it is an important step in conducting cardiac-related research and applications. Preprocessing techniques such as denoising, image enhancement, and image registration can improve the quality and usability of data, providing a reliable basis for subsequent cardiac analysis, diagnosis, and treatment.

[0142] S200: reading voxel data of the three-dimensional heart structure data, and converting the voxel data into point cloud to form voxel reconstruction point cloud data;

[0143] Reading the voxel data of three-dimensional cardiac structure data and converting it into point cloud is one of the key steps for subsequent analysis and processing. Voxel data is a data representation that divides the three-dimensional space into small cubes (voxels) and stores density or eigenvalues ​​in each voxel.

[0144] First, the voxel data of the 3D cardiac structure data needs to be parsed and read according to the data format. Common data formats include DICOM (Digital Imaging and Communications in Medicine) format, NIfTI (Neuroimaging Informatics Technology Initiative) format, etc. Specialized libraries or tools can be used to read these data, such as the pydicom library in Python, ITK (Insight Segmentation and Registration Toolkit), etc.

[0145] Once the voxel data is read, it can be converted into point cloud data for subsequent analysis and visualization. Point cloud data is a collection of three-dimensional coordinate points, each of which carries information about the spatial location. Converting voxel data into point cloud can provide a more flexible and intuitive data representation.

[0146] There are many ways to convert voxel points into point clouds. One method is to traverse each voxel and add the coordinates of the voxel center point to the point cloud data. This can create a point cloud composed of voxel center points. Another method is to determine whether to add the voxel center point to the point cloud data based on the density or feature value of the voxel. For example, you can filter based on the grayscale value or gradient value in the voxel and only select the voxel center points with specific characteristics.

[0147] The point clouding process can also perform downsampling or reconstruction operations as needed to reduce the amount of data or increase the density of the point cloud. Downsampling can be achieved through random sampling, grid sampling, or distance-based sampling. Reconstruction can use interpolation techniques such as nearest neighbor interpolation or trilinear interpolation to improve the resolution and smoothness of the point cloud.

[0148] By converting voxel data into point cloud, we can obtain point cloud data of three-dimensional cardiac structure, which will provide the basis for subsequent analysis, visualization and model building. The form of point cloud data makes it possible to apply various point cloud processing algorithms, such as point cloud segmentation, point cloud registration or point cloud reconstruction, to further analyze and process cardiac structure data.

[0149] S300: performing standardization processing on the voxel reconstructed point cloud data to obtain full-center point cloud data, and segmenting and collecting the full-center point cloud data to obtain standard section point cloud data;

[0150] The purpose of standardizing the point cloud data reconstructed by voxels is to obtain complete point cloud data of the entire heart for further analysis and processing. Standardization can include steps such as removing noise, normalizing coordinates, and unifying point cloud density to ensure the consistency and comparability of point cloud data.

[0151] First of all, noise removal is one of the key steps in standardization. In the process of voxel reconstruction, there may be some noise points or outliers, which may interfere with subsequent analysis and processing. Noise points can be removed by filtering algorithms (such as Gaussian filtering, median filtering) or denoising algorithms (such as statistical denoising algorithms, machine learning-based denoising algorithms), thereby improving the quality and accuracy of point cloud data.

[0152] Secondly, normalized coordinates are to map the coordinate range of point cloud data to a unified range to ensure data consistency and comparability. The coordinate range of the point cloud can be mapped to a standard coordinate space through linear transformation or scaling operations, such as mapping the range of each coordinate axis to the range of [-1, 1] or [0, 1].

[0153] In addition, unifying the point cloud density is also an important task in the standardization process. The point density in the point cloud data may vary, and some areas may have a higher point density while other areas may have a lower point density. In order to ensure the uniformity of the data, the point cloud data can be sampled or interpolated so that the point cloud has a similar point density throughout the heart area.

[0154] Once the normalization process is completed, complete point cloud data of the entire heart can be obtained. These point cloud data contain global morphological and structural information about the heart and can be used for applications such as whole heart analysis, cardiac registration, and morphological statistics.

[0155] In addition, in order to obtain standard section point cloud data, the whole heart point cloud data can be segmented and collected. Segmentation acquisition usually involves dividing the whole heart point cloud data into multiple sections, each of which contains point cloud data of a specific plane. Segmentation acquisition can be achieved by specifying the position, direction and size of the section. In this way, a series of standard section point cloud data can be obtained for applications such as section analysis, coronary artery analysis or cardiac function assessment.

[0156] In summary, standardization of voxel-reconstructed point cloud data can obtain complete point cloud data of the entire heart, and standard section point cloud data can be obtained by segmentation acquisition. Standardization processing includes steps such as noise removal, normalization of coordinates, and uniform point cloud density to ensure data consistency and comparability, providing an accurate and reliable data basis for subsequent analysis and processing.

[0157] S400: establishing a scanning guidance network based on the heart structure data to be processed and the standard section point cloud data;

[0158] A scanning guidance network is established based on the cardiac structure data to be processed and the standard section point cloud data to automatically guide the operations during the cardiac scanning process, thereby improving the scanning quality and efficiency.

[0159] First, establishing a scan guidance network usually requires the use of deep learning techniques, such as convolutional neural networks (CNNs) or generative adversarial networks (GANs). These networks can be trained from a large amount of cardiac structure data and standard section point cloud data to learn the patterns and laws of the relationship between cardiac structure and section.

[0160] During the training process of the network, the cardiac structure data can be used as input, and the standard section point cloud data can be used as output. By learning the mapping relationship between input and output, the network can predict the standard section point cloud data under specific circumstances. In this way, when given new cardiac structure data, the network can generate the corresponding standard section point cloud data.

[0161] In order to improve the accuracy and robustness of the network, several strategies can be adopted. First, data augmentation can be applied to the training data to increase the diversity of the data through transformation operations such as rotation, translation, and scaling, thereby improving the generalization ability of the network. Second, regularization terms or loss functions can be introduced to constrain the output of the network to ensure that the generated standard section point cloud data is as close to the actual section as possible. In addition, supervised learning methods can be introduced to guide the training and adjustment of the network by comparing with manually annotated standard sections.

[0162] Establishing a scan guidance network can bring multiple benefits. First, it can reduce human errors in the scanning process and improve the consistency and repeatability of the scan. Second, through the prediction of the network, the scanning time and radiation dose can be reduced, and the scanning efficiency and safety can be improved. In addition, the network can automatically identify and guide important sections during the scanning process, improving image quality and diagnostic accuracy.

[0163] In summary, establishing a scan guidance network based on the cardiac structure data to be processed and the standard section point cloud data is an effective method for automatically guiding the operation during the cardiac scanning process. Through deep learning technology and a large amount of training data, the network can predict the standard section point cloud data in specific situations and guide the doctor to perform operations in actual scanning, thereby improving the quality and efficiency of scanning.

[0164] S500: Acquire a test sample, input the test section point cloud data in the test sample into a scanning guidance network, and output the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located.

[0165] In order to evaluate the performance and effect of the scan-guided network, a set of test samples is required. The test samples contain cardiac structure data and corresponding test section point cloud data. These test samples can come from independent cardiac datasets and can be new data that has not been trained by the network.

[0166] First, the test section point cloud data in the test sample is input into the trained scan guidance network. The network will predict the three-dimensional point cloud data of the heart structure and the heart point cloud data of the section where the test section point cloud data is located based on the input test section point cloud data.

[0167] By inputting the test section point cloud data into the network, two key output results can be obtained. First, the network will generate 3D cardiac structure data of the test sample, which represents the 3D morphology and anatomical structure of the whole heart. These data can be used for further cardiac analysis applications.

[0168] Secondly, the network will also generate heart point cloud data of the slice where the test slice point cloud data is located. These data represent the heart structure of a specific slice, which contains the anatomical features and detailed information of the slice. These data are very useful for tasks such as slice analysis, hemodynamic assessment or abnormality detection.

[0169] By obtaining the 3D cardiac structure data and section point cloud data of the test sample, the performance and effectiveness of the scan guidance network can be evaluated. The difference between the cardiac structure data generated by the network and the real cardiac structure data can be compared to evaluate the accuracy and stability of the network. At the same time, the generated section point cloud data can be used for section analysis and other related tasks to verify the guidance effect and reliability of the network for a specific section.

[0170] In summary, obtaining test samples and inputting them into the scanning guidance network is a key step in evaluating the performance and effect of the network. By inputting the test section point cloud data, the network will generate the three-dimensional heart structure data of the test sample and the heart point cloud data of the specific section. These output results can be used to evaluate the performance of the network, verify the guidance effect, and provide a basis for further cardiac analysis and application.

[0171] In a preferred embodiment, step S100 includes:

[0172] S110: Collect cardiac volume data files and clinical data medical record information from multiple hospital units as basic data;

[0173] Sufficient cardiac volume data files and clinical data and medical records are collected from multiple hospitals at different levels and in different regions. For example, in this embodiment, cardiac CTA is used as the volume data to be processed to obtain its dicom file.

[0174] The cardiac volume data file can be three-dimensional intensity data obtained by radiological scanning or three-dimensional ultrasound technology, or three-dimensional volume data manually spliced ​​according to a certain thickness using two-dimensional imaging, or volume data processed from the cavity edge data reconstructed from the inside by intracavitary ultrasound, or even structural volume data reconstructed using cardiac perfusion specimens. In actual application scenarios, there are many ways to obtain cardiac volume data files, which are not limited here. For example, in medical data such as CTA, the cardiac volume data file is structural volume data with the same resolution as CTA that has been annotated and processed. The specific form is a three-dimensional array, and different component structures are distinguished by matrices of the same size composed of structural labels.

[0175] S120: After preprocessing the basic data, a basic database is established;

[0176] Preprocessing includes steps such as data cleaning, renaming, and format conversion. After preprocessing, the basic data is selected into the basic database after passing the review. Since the sources of cardiac volume data files are diverse, they need to be sorted out from different sources when necessary. Among them, the qualified data can be volume data with clear boundaries that meet quality standards (such as CTA data), or it can be volume data generated by boundary calibration information (such as intracardiac ultrasound volume data). In this embodiment, cardiac CTA is used as an example.

[0177] S130: Use a priori three-dimensional convolutional neural network to segment the three-dimensional heart structure in the heart volume data file in the basic database, and label the segmented heart area.

[0178] See also Figure 2a-2d , the data was automatically segmented using a priori 3D convolutional neural networks (previously validated 3D convolutional neural networks) to label various areas of the heart. To ensure the quality of the 3D segmentation results, a senior radiologist performed a visual inspection to ensure that the data labels met the standards.

[0179] Data labeling includes steps such as labeling process confirmation, labeling object and labeling task confirmation, labeling rule clarification, labeling quality control and evaluation, and data statistics. Specifically, it includes the following steps:

[0180] Annotation process: clarify the annotation object, personnel division of labor, annotation steps, result review, unify the annotation rules, confirm the meaning, form and output format of the annotation results, and describe the decision-making mechanism;

[0181] Clarify the labeled objects and labeling tasks. In particular, the present invention automatically segments CTA images by utilizing a previously verified three-dimensional convolutional neural network. The network provides ten labels including the left atrium, left ventricle, right ventricle, right atrium, left ventricular myocardium, aorta, and left and right pulmonary arteries. The outline results of senior physicians whose professional level meets the recognized standards in the industry are used as the gold standard, and they must be reviewed and confirmed by the reviewers before proceeding to the next step. After labeling with the outline software, export using the nii format;

[0182] Among them, the cardiac chamber structure needs to have a clear boundary definition. Exemplarily, in the embodiment of the present invention, multiple cardiac chambers and arteriovenous data are outlined based on voxels in CTA, and the structures of interest include the left and right atria, the left and right ventricles, the left ventricular wall, the aortic root, the pulmonary artery root, the left and right pulmonary veins, and the left atrial appendage. The junction of the atrioventricular junction is defined as the valve annulus, including the aortic valve, the mitral valve, and the tricuspid valve.

[0183] Annotation quality control: During the annotation process, the annotation quality of the annotators should be supervised, and both consistency and accuracy indicators should be considered. Senior radiologists further visually checked the quality of the 3D segmentation results to ensure that they met strict standards. After annotation, the accuracy, consistency, comprehensibility, accessibility, data security, and traceability were evaluated.

[0184] Further, step S200 includes:

[0185] S210: reading voxel data of the three-dimensional cardiac structure data, converting the voxel data into a three-dimensional array form, and performing an affine transformation on the voxel data in the three-dimensional array form to adjust it to a unified coordinate system;

[0186] The present invention first reads the nii data of the marked structure and converts it into a three-dimensional array form. Then, the three-dimensional array is affine transformed according to the stored affine matrix to adjust the data to a unified coordinate system, providing a basis for subsequent processing.

[0187] S220: for the annotated cardiac chamber structure and arteriovenous structure in the voxel reconstruction point cloud data, traverse the voxels in the eight neighborhoods of the cardiac chamber structure and the arteriovenous structure in an iterative manner to determine the cardiac chamber boundary point cloud of the cardiac chamber structure and the arteriovenous boundary point cloud of the arteriovenous structure, and extract the boundary information of the cardiac chamber structure and the arteriovenous structure;

[0188] The present invention adopts an iterative method, treating each structure as a positive sample in turn, and traversing all voxels in its eight neighborhoods. By identifying the coordinates of voxels with negative labels, the boundary point cloud of each structure is determined, so as to accurately extract the boundary information between structures, and different structures are stored as different color information to achieve distinction. In particular, the valve point in the embodiment of the present invention is located at the junction of the chamber annotations, wherein the mitral valve takes the junction point of the left ventricle and the left atrium, the tricuspid valve takes the junction point of the right ventricle and the right atrium, and the aortic valve takes the junction point of the left ventricle and the aorta.

[0189] Exemplarily, this embodiment samples the following structures according to voxel annotation: left ventricle, left ventricular wall, right ventricle, right atrium, left atrium, aorta, mitral valve, tricuspid valve and aortic valve.

[0190] S230: scanning the eight neighborhoods of each pixel in the voxel reconstruction point cloud data to identify the inner point cloud and the outer point cloud relative to the current structure of interest, and defining an initial simulation normal vector, and performing smoothing processing on the initial simulation normal vector;

[0191] In order to improve the quality of point cloud data, the present invention introduces the normal generation and correction steps. Specifically, firstly, based on the current voxel data, by scanning the eight-connected domain of each pixel point, the inner point and outer point relative to the current structure of interest are identified. Then, the direction from the centroid of the inner point cloud to the centroid of the outer point cloud is defined as the initial simulated normal vector. Furthermore, the minimum spanning tree algorithm is used to smooth the simulated normal vector to obtain more accurate normal vector information;

[0192] For example, in this embodiment, for a coordinate point (i, j, k)∈V in a voxel, when it is a boundary point, the 9 voxels in the cube [(i-1):(i+1),(j-1):(j+1),(k-1):(k+1)] are considered to be inside or outside the current structure, and the centroid of the point set pointing from the centroid of the point set inside the structure to the centroid of the point set outside the structure is used as the normal vector of the current vertex. In fact, the generation of the intersection point can be parallel to the generation of the normal.

[0193] S240: gridding the voxel data using the inner point coordinates and the outer point coordinates of the inner point cloud and the outer point cloud in the unified coordinate system, respectively, and the initial simulation normal vector to obtain gridded data, and retaining the original voxel data;

[0194] The point cloud (voxel data) is meshed using the obtained point cloud coordinates and their corrected simulated normal vectors. This process is performed one by one in a structural order to ensure that the reconstructed mesh not only conforms to the geometric characteristics of the original voxel data, but also retains rich detail information. Exemplarily, the Poisson reconstruction algorithm is used in the preferred embodiment of the present invention. At the same time, in order to facilitate subsequent processing, the present invention also retains the original point cloud data.

[0195] S250: for the gridded data, using curvature-based point cloud resampling to standardize the number of cardiac chamber structure points and the number of arteriovenous structure points in the gridded data to obtain voxel-reconstructed point cloud data;

[0196] Based on the grid data, a suitable point cloud resampling strategy is implemented to standardize the number of points of each structure to obtain voxel-reconstructed point cloud data. This method can automatically adjust the sampling density according to the curvature change of the grid surface, thereby effectively reducing data redundancy and improving processing efficiency while maintaining important geometric features. Specifically, in an embodiment of the present invention, the structures included in the resampled data are: left ventricle, left ventricular wall, left atrium, mitral valve, right ventricle, right atrium, tricuspid valve, aorta, and aortic valve;

[0197] Among them, the resampling method can be based on voxel resampling or curvature-based resampling. Exemplarily, the preferred embodiment of the present invention uses curvature-based point cloud resampling to standardize the number of points of each structure, and the standardized points are 4038 points for the left ventricle, 4660 points for the left ventricular wall, 1299 points for the left atrium, 1000 points for the mitral valve, 3814 points for the right ventricle, 1480 points for the right atrium, 1000 points for the tricuspid valve, 1139 points for the aorta, and 1000 points for the aortic valve.

[0198] Preferably or optionally, step S300 includes:

[0199] S310: performing point cloud posture correction and point cloud size correction on the voxel-reconstructed point cloud data to obtain standardized full-center point cloud data, so as to reduce the vertex coordinate values ​​of the voxel-reconstructed point cloud data;

[0200] Point cloud standardization includes point cloud pose correction and point cloud size correction; point cloud pose correction involves the regulation and unification of the spatial coordinate system, and point cloud size correction involves strengthening the mean consistency of the coordinates of points in different sample point clouds, which can reduce the difficulty of echocardiography scanning guidance network fitting. In particular, in the embodiment of the present invention, the centroid of the whole heart structure is used as the origin of the spatial coordinate system, the direction from the centroid of the tricuspid valve to the centroid of the mitral valve is used as the x-axis of the spatial coordinate system, the vector from the centroid of the left ventricle to the apex is used as the y-axis of the spatial coordinate system, and the outer product of the vector in the x-axis and y-axis direction is used as the z-axis of the spatial coordinate system. The corresponding affine matrix is ​​automatically extracted according to the structural calibration. After calibrating the pose, the vertex coordinate values ​​of the point cloud data are reduced to 1 / 300 of the original value to obtain standardized whole heart point cloud data.

[0201] S320: Based on the standard sections involved in the cardiac two-dimensional medical imaging standard, the whole heart point cloud data is segmented and collected to obtain standard section point cloud data.

[0202] Specifically, according to the three non-collinear key points of each standard section, the plane equation is solved, and the standardized point cloud data is traversed, and the inequality of the plane standard equation is used to sample the full-center point cloud data. In particular, in the embodiment of the present invention, the section geometry formula with a thickness of 0.01 unit length is used to sample the full-center point cloud data;

[0203] The standard section can be an ultrasound standard section or a CMR standard section. For example, taking the commonly used standard sections of ultrasound as an example, according to the ultrasound scanning guide, the key points of the apical four-chamber section are the mitral valve centroid, the tricuspid valve centroid, and the left ventricular apex; the key points of the parasternal long axis section are the left ventricular centroid, the aortic valve centroid, and the mitral valve centroid; the key points of the apical two-chamber section are the left ventricular centroid, the left ventricular apex, and the mitral valve centroid.

[0204] Examples of the label voxel data, standardized whole heart point cloud data, standardized valve point cloud data, and standardized section point cloud data involved in this embodiment are as follows: Figure 2a-2d The standardized section point cloud includes the apical four-chamber section, the apical two-chamber section, and the parasternal long axis section.

[0205] Preferably or optionally, step S400 is performed by a valve space inference device, which may be configured in a server and includes:

[0206] S410: Based on the PCN network, a multi-branch completion network is established by combining self-supervised learning and multi-branch reconstruction loss;

[0207] S420: dividing the cardiac structure data to be processed and the standard section point cloud data into a training set, a validation set, and a test set;

[0208] Considering that the same case may include cardiac structure data of different phases, a standardized point cloud is obtained. The present invention uses case level for diversity to avoid different samples of the same case appearing in the training set and the test set respectively. Exemplarily, this embodiment uses 7:1:2 as the ratio of the number of cases in the training set, validation set and test set. The sample size is 1007 pairs of point clouds for the training set, 145 pairs of point clouds for the validation set, and 288 pairs of point clouds for the test set.

[0209] S430: Using the three-dimensional heart structure data and the section point cloud where the standard section point cloud data is located as the target point cloud, and using the two-dimensional point cloud of the standard section point cloud data as the input point cloud, the multi-branch completion network is trained to obtain a scanning guidance network

[0210] See also Figure 3Exemplarily, the ultrasound scanning guidance network in the preferred embodiment of the present invention is a convolutional neural network using a codec structure; in the network, the PCN structure is used as an encoder for feature extraction, and the three-dimensional heart structure data and the section point cloud where the standard section point cloud data are located are converted into a 1024-dimensional feature vector, and a parallel multi-decoder is connected after the PCN structure; the parallel multi-decoder is mainly divided into four parts: the view completion and rotation decoder is used to generate the three-dimensional point cloud of the section plane where the ultrasound cardiogram is located; the component decoders generate three-dimensional point clouds of the six parts of the heart respectively, and the rough shape decoder directly generates the point cloud of the entire heart shape. The results of these two parts of the decoders are finally generated through the local generation module to generate the whole heart structure point cloud; the view reconstruction decoder is used to reconstruct the section to enhance the stability of the section shape and make the network more sensitive to structural defects related to the section angle, thereby achieving more accurate acoustic window and section angle positioning.

[0211] Preferably, step S430 includes:

[0212] S431: resampling the three-dimensional cardiac structure data and the standard section point cloud data, and arranging them into sizes received by the multi-branch completion network;

[0213] S432: Initialize the multi-branch completion network;

[0214] The initialization method can be random initialization or pre-trained parameter initialization.

[0215] S433: setting the pre-hyperparameters of the multi-branch completion network, wherein the pre-hyperparameters include pre-weights and post-weights, wherein the pre-weights increase the weight of the loss function for the coarse point cloud branch of the standard section point cloud data, and the post-weights increase the weight of the loss function for the fine point cloud branch of the standard section point cloud data;

[0216] Specifically, the loss function L compare for:

[0217] L total =L coarse +αL fine +αβL compare

[0218] Among them, α and β are pre-hyperparameters; the value of α increases with the progress of training, and is 0.01 when the number of training steps is less than 100, and is 0.1 when the number of training steps is more than 200; the value of β is fixed at 0.1; L coarse It includes the chamfer loss of the sparse point cloud of the six cardiac output structures, the chamfer loss of the reconstructed and rotated cross-sections, and the loss of the overall shape branch; L fine is the chamfer loss of dense point cloud; L compareTo compare the reconstruction loss, it is used to measure the difference between the point clouds of the six three-dimensional cardiac structure data generated separately and the whole heart point cloud data output by the local generation module.

[0219] The loss function L compare The three-part loss function L in coarse , L fine and L compare The calculation formulas are:

[0220] L coarse =∑ x∈Ω L CD (x,x gt )+L CD (Y coarse ,Y gtShape )+L CD (Y View ,X)+

[0221] L CD (Y Rotate ,Y gtView )

[0222] L fine =L CD (Y Shape ,Y gtShape )+L CD (Y Component ,Y gtShape )

[0223] L compare =L CD (Y Component ,Y Shape )

[0224] Where Ω is the rough point cloud branch set {S lv ,S la ,S rv ,S ra ,S myo ,S aorta}, x gt is the real data point cloud corresponding to x, and

[0225] L coarse , L fine and L compare The calculations are based on the chamfer distance under the L1 norm, where the output point cloud S is calculated. output and the real data point cloud S gt Chamfer distance L CD (S output ,S gt ) is:

[0226]

[0227] S434: Use the Adam optimizer to optimize the multi-branch completion network, and select the network parameters of the multi-branch completion network with the smallest L1 chamfer distance on the validation set data for storage.

[0228] The L1 chamfer distance is calculated by the following formula:

[0229]

[0230] Among them, S1 and S2 are two point clouds for calculating the chamfer distance. In this applicable scenario, they are the predicted point cloud inferred by the network for the verification data set and the actual heart structure point cloud and section positioning point cloud.

[0231] Preferably or optionally, step S500 includes:

[0232] S510: Acquire a test sample, input the test section point cloud data in the test sample into the scan guidance network, output the three-dimensional heart structure data of the test sample (use the model to predict the heart structure point cloud and view point cloud after the echocardiography scan guidance network converges), and locate the test section in the three-dimensional heart structure to obtain the structure of the heart and the specific position of the echocardiogram in the heart, and verify the accuracy of the test section point cloud data inferring the position of the heart valve;

[0233] Exemplarily, this embodiment includes 288 pairs of test samples.

[0234] S520: Evaluate the performance of the scan-guided network in the shape reconstruction task and the view localization task using evaluation criteria, wherein the performance includes chamfer distance based on L1 norm, chamfer distance based on L2 norm, and FScore with a threshold of 2 mm.

[0235] The evaluation criteria can be an indicator reflecting the overall distance between point clouds or a local distance indicator to the key structures of the heart. For example, in this embodiment, L1 chamfer distance, L2 chamfer distance and FScore are used as accuracy indicators. The L2 chamfer distance replaces the 1-norm of the L1 chamfer distance with the square of the 2-norm, and the calculation formula of FScore is as follows:

[0236]

[0237] With a certain distance threshold as the limit, when the minimum L1 distance between the predicted point cloud and the actual point cloud is less than a certain value, it is considered as a positive sample relative to the actual point cloud. Precision refers to the prediction result, which means the probability of actually being a positive sample among all samples predicted to be positive. Recall refers to the original sample, which means the probability of being predicted as a positive sample among samples that are actually positive. The distance threshold in this embodiment is 0.01 unit length.

[0238] Specifically, in the test set consisting of 288 samples, the average L1 chamfer distance obtained by the cardiac reconstruction method is 9.69×1e-3 unit lengths, the average L2 chamfer distance is 2.89×1e-4 unit lengths, and the average FScore is 65.22; the average L1 chamfer distance of each cardiac component point cloud output by the cardiac component branch is 7.75×1e-3 unit lengths, the average L2 chamfer distance is 1.86×1e-4 unit lengths, and the average FScore is 79.55. The section positioning method is accurate and stable, with a total angle error of 4.83±4.45 degrees. Among them, the angle error of the apical four-chamber section is the smallest, which is 2.79±4.84 degrees. The angle errors of the apical two-chamber section, the apical five-chamber section, and the parasternal left ventricular long axis section are 5.75±4.13 degrees, 4.98±4.22 degrees, and 5.79±3.89 degrees, respectively. Figure 4 Schematic diagram of section point cloud, inferred section position point cloud, true section position point cloud, inferred heart structure point cloud and true heart point cloud.

[0239] The present invention also discloses a cardiac point cloud reconstruction system for an echocardiogram, comprising:

[0240] An acquisition module, which acquires three-dimensional cardiac structure data and pre-processes the three-dimensional cardiac structure data;

[0241] The reconstruction module reads the voxel data of the three-dimensional cardiac structure data and converts the voxel data into point cloud to form voxel reconstruction point cloud data;

[0242] The segmentation module performs standardization processing on the voxel-reconstructed point cloud data to obtain the whole-center point cloud data, and segments and collects the whole-center point cloud data to obtain the standard section point cloud data;

[0243] A network construction module, which establishes a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data;

[0244] The drawing module obtains the test sample, inputs the test section point cloud data in the test sample into the scanning guidance network, and outputs the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located.

[0245] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A cardiac point cloud reconstruction method for echocardiography, characterized in that: The following steps are involved: Acquiring three-dimensional cardiac structure data, and preprocessing the three-dimensional cardiac structure data; Reading voxel data of the three-dimensional cardiac structure data, and converting the voxel data into point cloud to form voxel reconstruction point cloud data; Standardizing the voxel-reconstructed point cloud data to obtain full-center point cloud data, and segmenting and collecting the full-center point cloud data to obtain standard section point cloud data; Establishing a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data; A test sample is obtained, and the test section point cloud data in the test sample is input into the scanning guidance network, and the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located are output.

2. The cardiac point cloud reconstruction method according to claim 1, characterized in that: The steps of acquiring three-dimensional cardiac structure data and preprocessing the three-dimensional cardiac structure data include: Collect cardiac volume data files and clinical data and medical records from multiple hospital units as basic data; After preprocessing the basic data, a basic database is established; A priori three-dimensional convolutional neural network is used to segment the three-dimensional cardiac structure in the cardiac volume data file in the basic database, and the segmented cardiac region is labeled.

3. The cardiac point cloud reconstruction method according to claim 2, characterized in that: The steps of reading the voxel data of the three-dimensional cardiac structure data and converting the voxel data into point cloud to form voxel reconstruction point cloud data include: Reading voxel data of the three-dimensional cardiac structure data, converting the voxel data into a three-dimensional array form, and affine transforming the voxel data in the three-dimensional array form to adjust it to a unified coordinate system; For the annotated cardiac chamber structures and arteriovenous structures in the voxel reconstruction point cloud data, the voxels in the eight neighborhoods of the cardiac chamber structures and arteriovenous structures are iterated to determine the cardiac chamber boundary point cloud of the cardiac chamber structure and the arteriovenous boundary point cloud of the arteriovenous structure, and to extract the boundary information of the cardiac chamber structure and the arteriovenous structure; Scanning the eight neighborhoods of each pixel in the voxel reconstruction point cloud data to identify the inner point cloud and the outer point cloud relative to the current structure of interest, and defining an initial simulation normal vector, and smoothing the initial simulation normal vector; Using the inner point coordinates and outer point coordinates of the inner point cloud and the outer point cloud in a unified coordinate system, respectively, and the initial simulation normal vector, the voxel data is gridded to obtain gridded data, and the original voxel data is retained; For the gridded data, curvature-based point cloud resampling is used to normalize the number of cardiac chamber structure points and arteriovenous structure points in the gridded data to obtain voxel-reconstructed point cloud data.

4. The cardiac point cloud reconstruction method according to claim 1, characterized in that: The steps of standardizing the voxel reconstructed point cloud data to obtain full-center point cloud data, and segmenting and collecting the full-center point cloud data to obtain standard section point cloud data include: Performing point cloud posture correction and point cloud size correction on the voxel-reconstructed point cloud data to obtain standardized full-center point cloud data, so as to reduce the vertex coordinate values ​​of the voxel-reconstructed point cloud data; Based on the standard sections involved in the cardiac two-dimensional medical imaging standard, the whole heart point cloud data is segmented and collected to obtain standard section point cloud data.

5. The cardiac point cloud reconstruction method according to claim 1, characterized in that: The steps of establishing a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data include: Based on the PCN network, a multi-branch completion network is established by combining self-supervised learning and multi-branch reconstruction loss; The cardiac structure data to be processed and the standard section point cloud data are divided into a training set, a validation set and a test set; The three-dimensional heart structure data and the section point cloud where the standard section point cloud data are located are used as the target point cloud, and the two-dimensional point cloud of the standard section point cloud data is used as the input point cloud. The multi-branch completion network is trained to obtain the scanning guidance network.

6. The cardiac point cloud reconstruction method according to claim 5, characterized in that: The steps of training a multi-branch completion network by taking the three-dimensional heart structure data and the section point cloud where the standard section point cloud data is located as the target point cloud and the two-dimensional point cloud of the standard section point cloud data as the input point cloud to obtain the scanning guidance network include: Resample the 3D cardiac structure data and standard section point cloud data and organize them into sizes that can be received by the multi-branch completion network; Initialize the multi-branch completion network; Setting the pre-hyperparameters of the multi-branch completion network, wherein the pre-hyperparameters include pre-weights and post-weights, wherein the pre-weights increase the weight of the loss function for the coarse point cloud branch of the standard section point cloud data, and the post-weights increase the weight of the loss function for the fine point cloud branch of the standard section point cloud data; The multi-branch completion network is optimized using the Adam optimizer, and the network parameters of the multi-branch completion network with the smallest L1 chamfer distance on the validation set data are selected for storage.

7. The cardiac point cloud reconstruction method according to claim 6, characterized in that: The loss function L compare for: L total =L coarse +αL fine +αβL compare Among them, α and β are pre-hyperparameters; the value of α increases with the progress of training, and is 0.01 when the number of training steps is less than 100, and is 0.1 when the number of training steps is more than 200; the value of β is fixed at 0.1; L coarse It includes the chamfer loss of the sparse point cloud of the six cardiac output structures, the chamfer loss of the reconstructed and rotated cross-sections, and the loss of the overall shape branch; L fine is the chamfer loss of dense point cloud; L compare To compare the reconstruction loss, it is used to measure the difference between the point clouds of the six three-dimensional cardiac structure data generated separately and the whole heart point cloud data output by the local generation module.

8. The cardiac point cloud reconstruction method according to claim 7, characterized in that: Loss function L compare The three-part loss function L in coarse , L fine and L compare The calculation formulas are: L fine =L CD (AND Shape ,AND gtShape )+L CD (AND Component ,AND gtShape ) L compare =L CD (Y Component ,Y Shape ) Where Ω is the rough point cloud branch set {S lv ,S la ,S rv ,S ra ,S myo ,S aorta }, x gt is the real data point cloud corresponding to x, and L coarse , L fine and L compare The calculations are based on the chamfer distance under the L1 norm, where the output point cloud S is calculated. output and the real data point cloud S gt Chamfer distance L CD (S output ,S gt ) is: The L1 chamfer distance is calculated by the following formula: Among them, S1 and S2 are two point clouds for calculating the chamfer distance.

9. The cardiac point cloud reconstruction method according to claim 1, characterized in that: The steps of obtaining a test sample, inputting the test section point cloud data in the test sample into the scanning guidance network, and outputting the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located include: Acquire a test sample, input the test section point cloud data in the test sample into the scanning guidance network, output the three-dimensional heart structure data of the test sample, and locate the test section in the three-dimensional heart structure to verify the accuracy of the test section point cloud data inferring the position of the heart valve; Evaluation criteria were used to evaluate the performance of the scan-guided network in the shape reconstruction task and the view localization task, including the chamfer distance based on the L1 norm, the chamfer distance based on the L2 norm, and the FScore with a threshold of 2 mm.

10. A cardiac point cloud reconstruction system for echocardiography, characterized in that: include: An acquisition module, which acquires three-dimensional cardiac structure data and pre-processes the three-dimensional cardiac structure data; A reconstruction module, which reads the voxel data of the three-dimensional cardiac structure data and converts the voxel data into point cloud to form voxel reconstruction point cloud data; A segmentation module performs standardization processing on the voxel-reconstructed point cloud data to obtain full-center point cloud data, and segments and collects the full-center point cloud data to obtain standard section point cloud data; A network construction module, which establishes a scanning guidance network based on the cardiac structure data to be processed and the standard section point cloud data; The drawing module obtains a test sample, inputs the test section point cloud data in the test sample into the scanning guidance network, and outputs the three-dimensional heart structure data of the test sample and the heart point cloud data of the section where the test section point cloud data is located.

Images