Three-dimensional modeling method and device for heart based on point cloud generation

Abstract

The application discloses a kind of based on point cloud generation Heart three-dimensional modeling method and device, comprising: by the UNet network of pre-determined feature extraction operation is carried out to ultrasound volume image, obtain volume feature map;Based on volume feature map and pre-determined first processing operation, generate the feature cost body corresponding to volume feature map, and to feature cost body and pre-determined second processing operation, generate the feature probability body corresponding to volume feature map;Based on feature probability body and volume feature map, in combination with pre-determined depth map fusion algorithm, determine the depth estimation map corresponding to ultrasound volume image;According to pre-determined data processing operation and depth estimation map generate target point cloud data, and based on target point cloud data, generate the heart three-dimensional model corresponding to ultrasound volume image.It can be seen that the accuracy and reliability of the three-dimensional modeling of heart can be improved by implementing the present application, which is beneficial to improve the precision of the generated heart three-dimensional model.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a method and apparatus for 3D modeling of the heart based on point cloud generation. Background Technology

[0002] The heart, a muscular organ and one of the most vital organs in the human body, primarily functions to provide pressure for blood flow, propelling blood through the circulatory system and distributing it to all parts of the body. In clinical applications, the three-dimensional anatomical reconstruction of the entire four chambers of the heart is crucial for the clinical diagnosis of heart diseases. Most existing three-dimensional modeling methods for the heart utilize deep learning-based reconstruction techniques. However, this method has many limitations. For example, incorrect segmentation can result in broken areas in the generated 3D heart model, or limited segmentation resolution can lead to step artifacts, resulting in errors and low accuracy. Therefore, providing a new method for 3D heart modeling to improve its accuracy is of paramount importance. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and apparatus for three-dimensional modeling of the heart based on point cloud generation, which can improve the accuracy and reliability of three-dimensional modeling of the heart and is conducive to improving the accuracy of the generated three-dimensional heart model.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a method for three-dimensional modeling of the heart based on point cloud generation, the method comprising:

[0005] Feature extraction is performed on the ultrasound volume image using a pre-defined UNet network to obtain a volume feature map;

[0006] Based on the volume feature map and the predetermined first processing operation, a feature cost body corresponding to the volume feature map is generated, and a feature probability body corresponding to the volume feature map is generated based on the feature cost body and the predetermined second processing operation.

[0007] Based on the feature probability volume and the volume feature map, and combined with a pre-determined depth map fusion algorithm, the depth estimation map corresponding to the ultrasound volume image is determined.

[0008] Target point cloud data is generated based on the predetermined data processing operations and the depth estimation map, and a three-dimensional model of the heart corresponding to the ultrasound volume image is generated based on the target point cloud data.

[0009] As an optional implementation, in the first aspect of the present invention, the step of generating target point cloud data based on a predetermined data processing operation and the depth estimation map, and generating a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data, includes:

[0010] Based on the depth estimation map, generate backup point cloud data that matches the depth estimation map;

[0011] According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to obtain the target point cloud data; wherein, the target point cloud data includes grid point cloud data, and the grid point cloud data is used to construct a three-dimensional heart model and / or render a three-dimensional heart model;

[0012] Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated;

[0013] Determine the feature resolution information of the backup heart 3D model, and based on the feature resolution information, determine whether the resolution corresponding to the backup heart 3D model meets the preset model generation conditions;

[0014] When it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions, the model feature vector of the backup heart 3D model is extracted, and the fusion feature vector of the backup heart 3D model is determined. An update operation is performed on the backup heart 3D model based on the model feature vector and the fusion feature vector to obtain an updated backup heart 3D model. The operation of determining the feature resolution information of the backup heart 3D model and determining whether the resolution of the backup heart 3D model meets the preset model generation conditions based on the feature resolution information are triggered again.

[0015] When it is determined that the resolution of the backup heart 3D model meets the preset model generation conditions, the backup heart 3D model is identified as the heart 3D model corresponding to the ultrasound volume image.

[0016] As an optional implementation, in the first aspect of the present invention, the UNet network includes at least one convolutional layer and at least one residual block, wherein the step of performing feature extraction on the ultrasound volume image using the pre-determined UNet network to obtain a volume feature map includes:

[0017] The ultrasound volume image is subjected to feature extraction operation by each of the convolutional layers in the pre-determined UNet network to obtain at least one depth feature map;

[0018] For each depth feature map, the depth feature map is input into the residual block to obtain the output feature map corresponding to the depth feature map. Then, a feature addition operation is performed based on the depth feature map and the output feature map corresponding to the depth feature map to obtain the feature processing result corresponding to the depth feature map.

[0019] Based on the feature processing results corresponding to all the aforementioned depth feature maps, a volume feature map is generated.

[0020] As an optional implementation, in the first aspect of the present invention, generating the feature cost body corresponding to the volume feature map based on the volume feature map and a predetermined first processing operation includes:

[0021] Based on a predetermined first processing operation, a homography transformation operation is performed on the volume feature map to obtain a feature mapping result, wherein the feature mapping result includes the homography matrix corresponding to the volume feature map and the feature depth corresponding to the volume feature map;

[0022] Based on the feature mapping results and the pre-determined camera parameters, the feature cost volume corresponding to the volume feature map is determined;

[0023] The first processing operation, which is predetermined, includes the homography transformation operation.

[0024] As an optional implementation, in the first aspect of the present invention, generating the feature probability volume corresponding to the volume feature map from the feature cost volume and the predetermined second processing operation includes:

[0025] Based on the predetermined second processing operation, a regularization operation is performed on the feature cost body to remove the noise information contained in the feature cost body, thereby obtaining the target feature cost body;

[0026] Based on the target feature cost volume, a feature probability volume corresponding to the volume feature map is generated;

[0027] The second processing operation, which is predetermined, includes the regularization processing operation.

[0028] As an optional implementation, in the first aspect of the present invention, determining the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm, includes:

[0029] Based on the feature probability volume, the volume feature map, and a pre-determined depth map fusion algorithm, the depth probability corresponding to each feature pixel in the volume feature map is determined.

[0030] The depth estimation map corresponding to the ultrasound volume image is determined based on the depth probability corresponding to each feature pixel.

[0031] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0032] Determine a reference depth map corresponding to the ultrasound volume image; wherein, the reference depth map includes ultrasound image pixels of the ultrasound volume image and pixel coordinate information of the ultrasound volume image;

[0033] Calculate the image difference parameter between the reference depth map and the depth estimation map, wherein the image difference parameter includes the mean absolute error value between the reference depth map and the depth estimation map;

[0034] Determine whether the image difference parameters meet the preset image difference conditions;

[0035] When it is determined that the image difference parameters do not meet the preset image difference conditions, the parameters to be adjusted in the depth map fusion algorithm are determined, and based on the parameters to be adjusted, a backpropagation adjustment operation is performed on the depth map fusion algorithm to update the depth map fusion algorithm.

[0036] A second aspect of the present invention discloses a three-dimensional modeling device for the heart based on point cloud generation, the device comprising:

[0037] The extraction module is used to perform feature extraction operations on the ultrasound volume image through a pre-determined UNet network to obtain a volume feature map;

[0038] The generation module is used to generate a feature cost body corresponding to the volume feature map based on the volume feature map and a pre-determined first processing operation, and to generate a feature probability body corresponding to the volume feature map based on the feature cost body and a pre-determined second processing operation.

[0039] The determining module is also used to determine the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm;

[0040] The generation module is further configured to generate target point cloud data based on predetermined data processing operations and the depth estimation map, and generate a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data.

[0041] As an optional implementation, in the second aspect of the present invention, the specific method by which the generation module generates target point cloud data based on predetermined data processing operations and the depth estimation map, and generates a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data, includes:

[0042] Based on the depth estimation map, generate backup point cloud data that matches the depth estimation map;

[0043] According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to obtain the target point cloud data; wherein, the target point cloud data includes grid point cloud data, and the grid point cloud data is used to construct a three-dimensional heart model and / or render a three-dimensional heart model;

[0044] Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated;

[0045] Determine the feature resolution information of the backup heart 3D model, and based on the feature resolution information, determine whether the resolution corresponding to the backup heart 3D model meets the preset model generation conditions;

[0046] When it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions, the model feature vector of the backup heart 3D model is extracted, and the fusion feature vector of the backup heart 3D model is determined. An update operation is performed on the backup heart 3D model based on the model feature vector and the fusion feature vector to obtain an updated backup heart 3D model. The operation of determining the feature resolution information of the backup heart 3D model and determining whether the resolution of the backup heart 3D model meets the preset model generation conditions based on the feature resolution information are triggered again.

[0047] When it is determined that the resolution of the backup heart 3D model meets the preset model generation conditions, the backup heart 3D model is identified as the heart 3D model corresponding to the ultrasound volume image.

[0048] As an optional implementation, in a second aspect of the present invention, the UNet network includes at least one convolutional layer and at least one residual block, wherein the extraction module performs feature extraction on the ultrasound volume image using a pre-determined UNet network to obtain a volume feature map in the following specific manner:

[0049] The ultrasound volume image is subjected to feature extraction operation by each of the convolutional layers in the pre-determined UNet network to obtain at least one depth feature map;

[0050] For each depth feature map, the depth feature map is input into the residual block to obtain the output feature map corresponding to the depth feature map. Then, a feature addition operation is performed based on the depth feature map and the output feature map corresponding to the depth feature map to obtain the feature processing result corresponding to the depth feature map.

[0051] Based on the feature processing results corresponding to all the aforementioned depth feature maps, a volume feature map is generated.

[0052] As an optional implementation, in a second aspect of the present invention, the specific method by which the generation module generates the feature cost body corresponding to the volume feature map based on the volume feature map and a predetermined first processing operation includes:

[0053] Based on a predetermined first processing operation, a homography transformation operation is performed on the volume feature map to obtain a feature mapping result, wherein the feature mapping result includes the homography matrix corresponding to the volume feature map and the feature depth corresponding to the volume feature map;

[0054] Based on the feature mapping results and the pre-determined camera parameters, the feature cost volume corresponding to the volume feature map is determined;

[0055] The first processing operation, which is predetermined, includes the homography transformation operation.

[0056] As an optional implementation, in a second aspect of the present invention, the specific method by which the generation module generates the feature probability volume corresponding to the volume feature map based on the feature cost volume and the predetermined second processing operation includes:

[0057] Based on the predetermined second processing operation, a regularization operation is performed on the feature cost body to remove the noise information contained in the feature cost body, thereby obtaining the target feature cost body;

[0058] Based on the target feature cost volume, a feature probability volume corresponding to the volume feature map is generated;

[0059] The second processing operation, which is predetermined, includes the regularization processing operation.

[0060] As an optional implementation, in a second aspect of the present invention, the specific method by which the determining module determines the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm, includes:

[0061] Based on the feature probability volume, the volume feature map, and a pre-determined depth map fusion algorithm, the depth probability corresponding to each feature pixel in the volume feature map is determined.

[0062] The depth estimation map corresponding to the ultrasound volume image is determined based on the depth probability corresponding to each feature pixel.

[0063] As an optional implementation, in a second aspect of the present invention, the determining module is further configured to determine a reference depth map corresponding to the ultrasound volume image; wherein the reference depth map includes ultrasound image pixels of the ultrasound volume image and pixel coordinate information of the ultrasound volume image;

[0064] The device further includes:

[0065] The calculation module is used to calculate the image difference parameters between the reference depth map and the depth estimation map, wherein the image difference parameters include the mean absolute error value between the reference depth map and the depth estimation map;

[0066] The judgment module is used to determine whether the image difference parameters meet the preset image difference conditions;

[0067] The determining module is further configured to determine the parameters to be adjusted in the depth map fusion algorithm when the judging module determines that the image difference parameters do not meet the preset image difference conditions;

[0068] An adjustment module is used to perform a backpropagation adjustment operation on the depth map fusion algorithm based on the parameters to be adjusted, so as to update the depth map fusion algorithm.

[0069] A third aspect of the present invention discloses another device for three-dimensional modeling of the heart based on point cloud generation, the device comprising:

[0070] Memory containing executable program code;

[0071] A processor coupled to the memory;

[0072] The processor calls the executable program code stored in the memory to execute the heart 3D modeling method based on point cloud generation disclosed in the first aspect of the present invention.

[0073] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the heart 3D modeling method based on point cloud generation disclosed in the first aspect of the present invention.

[0074] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0075] In this embodiment of the invention, a feature extraction operation is performed on an ultrasound volume image using a pre-determined UNet network to obtain a volume feature map. Based on the volume feature map and a pre-determined first processing operation, a feature cost volume corresponding to the volume feature map is generated. Then, based on the feature cost volume and a pre-determined second processing operation, a feature probability volume corresponding to the volume feature map is generated. Based on the feature probability volume and the volume feature map, and combined with a pre-determined depth map fusion algorithm, a depth estimation map corresponding to the ultrasound volume image is determined. Target point cloud data is generated according to pre-determined data processing operations and the depth estimation map. Based on the target point cloud data, a three-dimensional model of the heart corresponding to the ultrasound volume image is generated. Therefore, implementing this invention can improve the accuracy and reliability of three-dimensional modeling of the heart, and is beneficial to improving the accuracy of the generated three-dimensional heart model. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 This is a flowchart illustrating a method for three-dimensional modeling of the heart based on point cloud generation, as disclosed in an embodiment of the present invention.

[0078] Figure 2 This is a flowchart illustrating another method for three-dimensional modeling of the heart based on point cloud generation disclosed in an embodiment of the present invention;

[0079] Figure 3 This is a schematic diagram of the structure of a heart 3D modeling device based on point cloud generation disclosed in an embodiment of the present invention;

[0080] Figure 4 This is a schematic diagram of another heart 3D modeling device based on point cloud generation disclosed in an embodiment of the present invention;

[0081] Figure 5 This is a schematic diagram of another heart 3D modeling device based on point cloud generation disclosed in an embodiment of the present invention. Detailed Implementation

[0082] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0084] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0085] This invention discloses a method and apparatus for three-dimensional modeling of the heart based on point cloud generation, which can improve the accuracy and reliability of three-dimensional modeling of the heart and is beneficial to improving the precision of the generated three-dimensional heart model. Detailed descriptions follow.

[0086] Example 1

[0087] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for three-dimensional modeling of the heart based on point cloud generation, as disclosed in an embodiment of the present invention. Figure 1 The described point cloud-based 3D heart modeling method can be applied to point cloud-based 3D heart modeling devices, as well as to cloud servers or local servers for point cloud-based 3D heart modeling. This invention does not limit the application of this method. Figure 1 As shown, the point cloud-based 3D modeling method for the heart can include the following operations:

[0088] 101. Perform feature extraction on the ultrasound volume image using a pre-defined UNet network to obtain a volume feature map.

[0089] In this embodiment of the invention, the UNet network can optionally be a 3DUNet network, wherein the UNet network belongs to a fully convolutional neural network model. The UNet network combines shallow and deep features through a U-shaped network structure with jumps, which is used for the final semantic segmentation map generation. The UNet network combines shallow and deep features in a splicing manner.

[0090] In this embodiment of the invention, optionally, the convolutional layers of the UNet network are used to perform multi-level feature extraction on the ultrasound volume image to obtain a multi-resolution volume feature map; wherein, the volume feature map is a multi-resolution volume feature map.

[0091] 102. Based on the volume feature map and the predetermined first processing operation, generate the feature cost body corresponding to the volume feature map, and generate the feature probability body corresponding to the volume feature map based on the feature cost body and the predetermined second processing operation.

[0092] In this embodiment of the invention, optionally, the first processing operation includes a homography transformation operation, and the second processing operation includes a cost volume regularization operation.

[0093] In this embodiment of the invention, optionally, the feature cost body includes a matching cost body used to reflect the similarity between candidate matching pixel pairs. Therefore, the true matching pixel pair should have the minimum cost, which reflects that it should be a unimodal distribution near the ground truth disparity.

[0094] In this embodiment of the invention, optionally, the feature probability volume corresponding to the volume feature map includes the value of each feature pixel included in the volume feature map. The value of each point represents the validity of the depth estimation at that location. The higher the probability, the more likely it is to be a correct depth estimation. For the probability space, the probability sum of every four neighborhoods is calculated along the depth dimension, and then the maximum probability sum is taken along the depth dimension. Further optionally, the expected depth is calculated by multiplying the probability distribution of each depth by the actual depth.

[0095] 103. Based on the feature probability volume and volume feature map, and combined with a pre-determined depth map fusion algorithm, determine the depth estimation map corresponding to the ultrasound volume image.

[0096] In this embodiment of the invention, optionally, the depth estimation map includes the depth probability corresponding to each pixel in the volume feature map.

[0097] 104. Generate target point cloud data based on the predetermined data processing operations and depth estimation map, and generate a three-dimensional model of the heart corresponding to the ultrasound volume image based on the target point cloud data.

[0098] In this embodiment of the invention, optionally, the target point cloud data is rendered and / or used to construct a three-dimensional model of the heart.

[0099] In this embodiment of the invention, optionally, the generated three-dimensional heart model includes at least two three-dimensional heart models with different resolutions, and the three-dimensional heart model is a three-dimensional model of the entire heart with four chambers.

[0100] It is evident that implementation Figure 1 The described point cloud-based 3D cardiac modeling method can perform feature extraction on ultrasound volumetric images using a UNet network to obtain volumetric feature maps. A first processing operation is then performed on the volumetric feature maps to obtain feature cost volumes, followed by a second processing operation to generate feature probability volumes. Based on the feature probability volumes and volumetric feature maps, a depth estimation map corresponding to the ultrasound volumetric image is determined using a depth map fusion algorithm, and target point cloud data is generated. Finally, a 3D cardiac model corresponding to the ultrasound volumetric image is generated based on the target point cloud data. This method can generate multi-resolution 3D cardiac models through a progressive processing approach, which improves the accuracy and reliability of the generated 3D cardiac models, as well as their intelligence and efficiency. Furthermore, it facilitates further improvement in the generation and reconstruction accuracy of 3D cardiac models by extracting more semantic information about cardiac structures.

[0101] In an optional embodiment, target point cloud data is generated based on predetermined data processing operations and a depth estimation map, and a three-dimensional cardiac model corresponding to the ultrasound volume image is generated based on the target point cloud data, including:

[0102] Based on the depth estimation map, generate backup point cloud data that matches the depth estimation map;

[0103] According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to obtain the target point cloud data; wherein, the target point cloud data includes grid point cloud data, and the grid point cloud data is used to construct a three-dimensional heart model and / or render a three-dimensional heart model;

[0104] Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated.

[0105] Determine the feature resolution information of the three-dimensional model of the spare heart, and based on the feature resolution information, determine whether the resolution of the three-dimensional model of the spare heart meets the preset model generation conditions.

[0106] When it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions, the model feature vector of the backup heart 3D model is extracted, and the fusion feature vector of the backup heart 3D model is determined. The backup heart 3D model is updated based on the model feature vector and the fusion feature vector to obtain the updated backup heart 3D model. The operation of determining the feature resolution information of the backup heart 3D model and the operation of determining whether the resolution of the backup heart 3D model meets the preset model generation conditions based on the feature resolution information are triggered again.

[0107] When it is determined that the resolution of the backup heart 3D model meets the preset model generation conditions, the backup heart 3D model is identified as the heart 3D model corresponding to the ultrasound volume image.

[0108] In this optional embodiment, optionally, a data transformation operation is performed on the backup point cloud data according to a predetermined data processing operation to obtain the target point cloud data, including:

[0109] According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to transform the backup point cloud data into grid point cloud data, and the target point cloud data is determined based on the grid point cloud data.

[0110] Among them, grid point cloud data is used for rendering and visualization of scene 3D models, enabling 3D modeling based on volumetric images.

[0111] In this optional embodiment, the feature resolution information of the backup heart 3D model may include one or more of the following: resolution quantity information corresponding to the backup heart 3D model, resolution value corresponding to the backup heart 3D model, and resolution hierarchy information corresponding to the backup heart 3D model.

[0112] In this optional embodiment, the data processing operation may include a data transformation operation.

[0113] In this optional embodiment, optionally, determining whether the resolution corresponding to the three-dimensional model of the spare heart meets the preset model generation conditions based on the feature resolution information includes:

[0114] Based on the feature resolution information, determine the number of resolution levels in the three-dimensional model of the spare heart.

[0115] Determine whether the number of resolution layers in the 3D model of the spare heart is greater than or equal to a preset threshold.

[0116] When it is determined that the number of resolution levels of the backup heart 3D model is greater than or equal to the preset number threshold, the resolution of the backup heart 3D model is determined to meet the preset model generation conditions.

[0117] When it is determined that the number of resolution levels of the backup heart 3D model is less than the preset threshold, it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions.

[0118] In this optional embodiment, the fusion feature vector corresponding to the backup heart 3D model may include the fusion feature vector used by the 3D model in the next generation stage. For example, if the resolution corresponding to the backup heart 3D model is low, then the fusion feature vector used by the 3D model in the next generation stage is the fusion feature vector corresponding to high resolution. Furthermore, the above 3D modeling process is repeated to complete the generation of the progressive 3D model. Further optionally, the resolution of the fusion feature vector corresponding to the 3D model is different in each generation stage.

[0119] In this optional embodiment, optionally, an update operation is performed on the backup heart 3D model based on the model feature vector and the fused feature vector to obtain an updated backup heart 3D model, including:

[0120] Based on the model feature vector and the fused feature vector, the model update parameters of the standby 3D model are determined, including resolution update parameters and hierarchy update parameters.

[0121] Based on the model update parameters of the backup 3D model, an update operation is performed on the backup heart 3D model to obtain the updated backup heart 3D model.

[0122] In this optional embodiment, for example, the three-dimensional heart model may include five levels. This is achieved by processing the data using five residual blocks to obtain five feature maps of different scales and levels. The extracted low-level image features have higher resolution and contain more detailed information about the heart structure, but their semantics are lower and they contain more noise. As the network deepens, the extracted high-level volumetric feature maps become increasingly abstract, possessing stronger semantic information, but their spatial resolution decreases, and their ability to perceive details is also poorer. Therefore, we use high-level, low-resolution feature maps to generate a coarse structure of the heart; for detailed heart structures, we use low-level, high-resolution image features. Furthermore, the residual blocks used in this invention fuse high-level and low-level feature maps to increase the number of feature map channels describing image information. This allows for the addition of more detailed information about the heart structure by fusing low-level features while retaining high-level semantic information, thus mapping to obtain a higher-quality three-dimensional model of the entire four chambers of the heart.

[0123] As can be seen, implementing this optional embodiment can generate backup point cloud data based on the depth estimation map and perform data transformation operations to obtain target point cloud data. Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated. The feature resolution information of the backup three-dimensional model of the heart is determined, and it is judged whether it meets the preset model generation conditions. If it meets the conditions, the backup three-dimensional model of the heart is determined to be a three-dimensional model of the heart. If it does not meet the conditions, the model feature vector of the backup three-dimensional model of the heart is extracted, and the corresponding fusion feature vector of the backup three-dimensional model of the heart is determined. The backup three-dimensional model of the heart is updated according to the model feature vector and the fusion feature vector, and the backup three-dimensional model of the heart is re-judged to see if it meets the model generation conditions. By continuously adding detailed information of the heart anatomical structure, the heart can be gradually approximated to the real heart anatomical structure through progressive fine-tuning. This is beneficial to improving the accuracy and reliability of the generated three-dimensional model of the heart, as well as the intelligence and efficiency of the generated three-dimensional model of the heart. Furthermore, by generating the three-dimensional model of the heart through progressive point cloud data, the accuracy and reliability of the generated three-dimensional model of the heart can be further improved, and the precision of the generated three-dimensional model of the heart can be improved.

[0124] In another optional embodiment, the UNet network includes at least one convolutional layer and at least one residual block, wherein feature extraction is performed on the ultrasound volume image using a pre-determined UNet network to obtain a volume feature map, including:

[0125] By performing feature extraction operations on each convolutional layer in a pre-defined UNet network, at least one depth feature map is obtained from the ultrasound volume image.

[0126] For each depth feature map, the depth feature map is input into the residual block to obtain the output feature map corresponding to the depth feature map. Then, a feature addition operation is performed based on the depth feature map and the output feature map corresponding to the depth feature map to obtain the feature processing result corresponding to the depth feature map.

[0127] A volumetric feature map is generated based on the feature processing results corresponding to all depth feature maps.

[0128] In this optional embodiment, the UNet network may optionally include at least one residual block.

[0129] In this optional embodiment, the ultrasound volume image is optionally composed of N different cross-sectional images from N viewpoints, and N depth feature maps can be extracted by the UNet network.

[0130] In this optional embodiment, each convolutional layer is optionally a 3×3×3 convolutional layer with a stride of 2, and each residual block consists of two 3×3×3 convolutional layers and a dropout layer. The input feature map and the output feature map are summed through a shortcut mapping, and finally, a feature pool for mapping the three-dimensional model of the heart is obtained.

[0131] In this optional embodiment, for example, residual blocks can be introduced into the UNet network to promote gradient propagation during training, solve the gradient vanishing problem that occurs as the network deepens, improve generalization ability, and enable the network to learn deeper and more abstract features and acquire stronger semantic information. First, each feature extraction layer inputs the feature map into a 3×3×3 convolutional layer with a stride of 2 to reduce the resolution of the feature map and improve the network's ability to process large volume image data. Next, the feature map is input into a residual block for feature extraction. Each residual block consists of two 3×3×3 convolutional layers and a dropout layer, and the input and output feature maps are summed using a shortcut mapping. Finally, a feature pool is obtained for mapping the 3D model of the heart. Among them, the feature map input residual block can extract deeper features and semantic information of heart structure through convolution calculation; the dropout layer can suppress overfitting and improve the model's generalization ability; the equal mapping enables each channel feature map to acquire more image features and avoids gradient vanishing due to the increase in network depth, so that image features can be passed to deeper network layers.

[0132] As can be seen, implementing this optional embodiment can perform feature extraction operations on ultrasound volume images through convolutional layers in the UNet network to obtain depth feature maps, and obtain output feature maps corresponding to the depth feature maps through residual blocks. Based on the depth feature maps and the corresponding output feature maps, a feature addition operation is performed to obtain the feature processing results corresponding to each depth feature map, and a volume feature map is generated based on the feature processing results. This allows for multi-level feature extraction of ultrasound volume images through the UNet network, resulting in multi-resolution volume feature maps. This enriches the data on which the subsequent heart model is based, which is beneficial to improving the accuracy and multi-level nature of the subsequently generated multi-resolution, multi-level 3D heart model. By introducing residual blocks into the UNet network, gradient propagation and generalization ability are promoted during training, enabling the network to learn deeper abstract features and obtain stronger semantic information. This is beneficial to improving the accuracy and multi-level nature of the subsequently generated multi-resolution, multi-level 3D heart model. Furthermore, by obtaining multi-dimensional semantic information about the heart structure, the accuracy and reliability of the subsequently generated 3D heart model are further improved, as well as the precision and intelligence of the subsequently generated 3D heart model.

[0133] In another optional embodiment, based on the volume feature map and a pre-determined first processing operation, a feature cost body corresponding to the volume feature map is generated, including:

[0134] Based on the predetermined first processing operation, a homography transformation operation is performed on the volume feature map to obtain the feature mapping result, wherein the feature mapping result includes the homography matrix corresponding to the volume feature map and the feature depth corresponding to the volume feature map;

[0135] Based on the feature mapping results and the pre-determined camera parameters, the feature cost volume corresponding to the volume feature map is determined;

[0136] The first processing operation, which is predetermined, includes a homography transformation operation.

[0137] In this optional embodiment, the homography transformation operation can be performed through homography matrix transformation, whereby 2D image features can be transformed into feature volumes based on camera frustums, and multiple feature volumes based on variance can be merged into a single cost volume. This enables the encoding of camera geometric information into the network and allows it to adapt to different numbers of multi-view image inputs, ensuring end-to-end training in deep learning.

[0138] In this optional embodiment, the feature map coordinates can be mapped first using a homography transformation: F′ i ~H i (d i )·F i Among them, H i (d) is a single ′ The adaptability matrix, where d is the feature map depth. That is, it represents the feature map F at depth d. i Mapping to reference feature map F i Assuming the depth interval is [d1, d2] and the resolution is Δd, then there are D = (d2 - d1) / Δd planes. The input 576-channel feature map is obtained by feature fusion of the highest-layer 384-channel feature map and the lowest-layer 192-channel feature map. The feature map undergoes the homography transformation described above to obtain the 3D cost volume, which is further used to obtain the feature cost volume corresponding to the volume feature map.

[0139] In this optional embodiment, the reference view is optionally labeled I1, and the source view image is optionally labeled as I2. Camera internal parameters are The rotation parameters and translation parameters (extrinsic parameters) are respectively and First, a differentiable homography transformation is performed. All extracted feature maps need to be mapped onto different parallel planes from the reference camera's viewpoint, forming N feature maps. The feature cost volume of depth d.

[0140] As can be seen, implementing this optional embodiment can perform a homography transformation operation on the volume feature map based on a first processing operation including a homography transformation operation to obtain a feature mapping result. Based on the feature mapping result and camera parameters, the feature cost body corresponding to the volume feature map is determined. It can combine all extracted feature maps with camera parameters and map them to different parallel planes under the reference camera viewpoint to form feature maps through differentiable homography transformation. This is beneficial to improving the accuracy and reliability of the generated feature cost body, as well as improving the accuracy of the generated feature cost body. Furthermore, by generating the cost body, it is possible to encode the set information in the camera parameters into the network and adapt to different numbers of multi-view image inputs. This is beneficial to improving the accuracy and reliability of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, as well as improving the intelligence and efficiency of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, and further improving the model accuracy of the generated three-dimensional heart model.

[0141] In another optional embodiment, generating a feature probability volume corresponding to the volume feature map based on the feature cost volume and the predetermined second processing operation includes:

[0142] Based on the predetermined second processing operation, a regularization operation is performed on the feature cost body to remove the noise information contained in the feature cost body, thereby obtaining the target feature cost body.

[0143] Based on the target feature cost volume, generate the feature probability volume corresponding to the volume feature map;

[0144] The pre-determined second processing operation includes a regularization operation.

[0145] In this optional embodiment, optionally, based on a predetermined second processing operation, a regularization operation is performed on the feature cost body to remove noise information contained in the feature cost body, thereby obtaining the target feature cost body, including:

[0146] Perform a noise information detection operation on the feature cost body to obtain at least one noise information contained in the feature cost body;

[0147] For each piece of noise information contained in the feature cost body, a regularization operation is performed on the noise information to remove the noise information from the feature cost body;

[0148] The target feature cost body is generated based on the feature cost body after removing noise information.

[0149] In this optional embodiment, for example, regularization can be performed using 3Dconv, wherein the noise contained in the initial volume calculated from the image feature map may be due to the presence of non-Lambertian surfaces or line-of-sight occlusion. Smoothing is also required to predict the depth map. The regularization step is to optimize the feature cost volume by removing noise to obtain the feature probability volume.

[0150] In this optional embodiment, the feature probability volume can be used for pixel-by-pixel depth estimation in an image, and can also be used to measure the estimation confidence level.

[0151] As can be seen, implementing this optional embodiment can perform regularization processing on the feature cost body based on the second processing operation including regularization processing to remove noise information contained in the feature cost body to obtain the target feature cost body, and generate the feature probability body corresponding to the volume feature map based on the target feature cost body. It can remove the noise information contained in the feature cost body through regularization processing, which is beneficial to improving the accuracy and reliability of the generated target feature cost body, preventing errors in the feature cost body due to noise interference. Furthermore, generating the feature probability body corresponding to the volume feature map based on the target feature cost body after removing noise information can reduce the amount of noise information contained in the generated feature probability body, which is beneficial to improving the accuracy of the generated feature probability body, as well as improving the accuracy and intelligence of the generated feature probability body. This, in turn, is beneficial to improving the accuracy and reliability of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, as well as improving the intelligence and efficiency of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, and further beneficial to improving the model accuracy of the generated three-dimensional heart model.

[0152] In another optional embodiment, based on the feature probability volume and volume feature map, and combined with a pre-determined depth map fusion algorithm, a depth estimation map corresponding to the ultrasound volume image is determined, including:

[0153] Based on the feature probability volume, volume feature map and pre-determined depth map fusion algorithm, the depth probability corresponding to each feature pixel point included in the volume feature map is determined;

[0154] The depth estimation map corresponding to the ultrasound volume image is determined based on the depth probability corresponding to each feature pixel.

[0155] In this optional embodiment, the depth estimation map generated by the feature probability volume can be generated by the winnner-take-all strategy. The expected depth is calculated by multiplying the probability distribution of each depth by the actual depth, and then the depth estimation map is generated. This helps to improve the accuracy and precision of the generated depth estimation map.

[0156] In this optional embodiment, the depth map fusion algorithm may include one or more of the following: multi-modal image fusion algorithm, digital photographic image fusion algorithm, and remote sensing image fusion algorithm. This embodiment of the invention does not impose specific limitations.

[0157] In this optional embodiment, by defining the accuracy of the estimated depth as the probability that the true depth is within a small range near the estimated depth, and by analyzing the probability distribution of the estimated depth to determine the accuracy of the estimated depth, the accuracy and reliability of the depth probability corresponding to each feature pixel can be improved, as can the intelligence and precision of the depth probability corresponding to each feature pixel can be improved.

[0158] In this optional embodiment, the depth probability corresponding to each feature pixel is further optionally used to represent the estimation validity of the depth at the location of that feature pixel. A higher probability indicates a more likely correct depth estimate. For the probability space, the probability sum of every four neighborhoods along the depth dimension is calculated, and then the maximum probability sum along the depth dimension is taken to obtain the depth probability corresponding to each feature point. Furthermore, the depth estimation map can be improved. Since the large receptive field during the regularization process leads to overly smooth reconstructed edges, a residual learning network for the depth map is used to connect the initial depth map and the reference map to form a four-channel feature map. This feature map is then convolved to obtain the final probability estimation result.

[0159] As can be seen, implementing this optional embodiment can determine the depth probability corresponding to each feature pixel in the volume feature map based on the feature probability volume and the volume feature map combined with the depth map fusion algorithm, and determine the depth estimation map corresponding to the ultrasound volume image based on the depth probability corresponding to each feature pixel. The accuracy of the estimated depth can be defined as the probability that the true depth is within a small range near the estimated depth, and the accuracy of the estimated depth can be determined by analyzing the probability distribution of the estimated depth. This can improve the accuracy and reliability of the determined depth probability corresponding to each feature pixel, as well as the intelligence and accuracy of the determined depth probability corresponding to each feature pixel. This is beneficial to improving the accuracy and reliability of the subsequent generation of multi-dimensional and multi-level three-dimensional cardiac models, as well as the intelligence and efficiency of the subsequent generation of multi-dimensional and multi-level three-dimensional cardiac models, and further beneficial to improving the model accuracy of the generated three-dimensional cardiac models.

[0160] Example 2

[0161] Please see Figure 2 , Figure 2 This is a flowchart illustrating another method for three-dimensional cardiac modeling based on point cloud generation disclosed in an embodiment of the present invention. Figure 2The described point cloud-based 3D heart modeling method can be applied to point cloud-based 3D heart modeling devices, as well as to cloud servers or local servers for point cloud-based 3D heart modeling. This invention does not limit the application of this method. Figure 2 As shown, the point cloud-based 3D modeling method for the heart can include the following operations:

[0162] 201. Perform feature extraction on the ultrasound volume image using a pre-defined UNet network to obtain a volume feature map.

[0163] 202. Based on the volume feature map and the predetermined first processing operation, generate the feature cost body corresponding to the volume feature map, and generate the feature probability body corresponding to the volume feature map based on the feature cost body and the predetermined second processing operation.

[0164] 203. Based on the feature probability volume and volume feature map, and combined with a pre-determined depth map fusion algorithm, determine the depth estimation map corresponding to the ultrasound volume image.

[0165] 204. Generate target point cloud data based on the predetermined data processing operations and depth estimation map, and generate a three-dimensional model of the heart corresponding to the ultrasound volume image based on the target point cloud data.

[0166] In this embodiment of the invention, for a detailed description of steps 201-204, please refer to the other descriptions of steps 101-104 in Embodiment 1. This embodiment of the invention will not repeat them.

[0167] 205. Determine the reference depth map corresponding to the ultrasound volume image.

[0168] In this embodiment of the invention, the reference depth map includes ultrasound image pixels of the ultrasound volume image and pixel coordinate information of the ultrasound volume image.

[0169] In this embodiment of the invention, optionally, the reference depth map can be a pre-determined reference depth map corresponding to the ultrasound volumetric image. Further optionally, the number of reference depth maps corresponding to the ultrasound volumetric image can be one or more; this embodiment of the invention does not impose a specific limitation.

[0170] 206. Calculate the image difference parameters between the reference depth map and the depth estimation map.

[0171] In this embodiment of the invention, the image difference parameters include the average absolute error value between the reference depth map and the depth estimation map. Further, the image difference parameters may also include the difference parameters between ultrasound image pixels in the reference depth map and ultrasound image pixels in the depth estimation map, and the difference parameters between pixel coordinate information in the reference depth map and pixel coordinate information in the depth estimation map.

[0172] In this embodiment of the invention, optionally, the image difference parameter between the reference depth map and the depth estimation map can be calculated using a preset image difference calculation formula, wherein the image difference calculation formula may include:

[0173]

[0174] Among them, P valid d(p) represents the valid set of correct pixels (ground truth pixels); d(p) represents the actual depth of pixel p; This represents the initial estimated depth of pixel p; This represents the precise estimated depth of pixel p; the value of parameter λ is 1.

[0175] 207. Determine whether the image difference parameters meet the preset image difference conditions.

[0176] In this embodiment of the invention, optionally, determining whether the image difference parameters meet preset image difference conditions may include:

[0177] Based on the image difference parameters, determine the image difference value, and based on the preset image difference conditions, determine the image difference threshold;

[0178] Determine whether the image difference value is greater than or equal to a preset image difference threshold;

[0179] When it is determined that the image difference value is greater than or equal to the preset image difference threshold, it is determined that the image difference parameter does not meet the preset image difference condition.

[0180] When the image difference value is determined to be less than the preset image difference threshold, the image difference parameter is determined to meet the preset image difference condition.

[0181] 208. When it is determined that the image difference parameters do not meet the preset image difference conditions, the parameters to be adjusted in the depth map fusion algorithm are determined, and based on the parameters to be adjusted, the backpropagation adjustment operation is performed on the depth map fusion algorithm to update the depth map fusion algorithm.

[0182] In this embodiment of the invention, optionally, the process can be terminated when it is determined that the image difference parameters meet the preset image difference conditions.

[0183] In this embodiment of the invention, optionally, the number of parameters to be adjusted can be one or more, and this embodiment of the invention does not make a specific limitation.

[0184] In this embodiment of the invention, optionally, the backpropagation adjustment operation may include a gradient backpropagation adjustment operation. Further optionally, the chain rule is used to update the parameters of each layer during backpropagation until the optimal weights are obtained.

[0185] As can be seen, implementing this optional embodiment can determine the reference depth map corresponding to the ultrasound volume image, calculate the image difference parameters between the reference depth map and the depth estimation map, and determine whether the image difference parameters meet the preset image difference conditions. If not, it determines the parameters to be adjusted in the depth map fusion algorithm and performs backpropagation adjustment operation on the depth map fusion algorithm to update the depth map fusion algorithm. It can calculate the loss of the initial depth map and the optimized depth map respectively, and backpropagate to adjust the network parameters, optimize the network performance, and continuously fine-tune the model performance. Finally, it can generate a three-dimensional model of the whole heart with four chambers that approximates the real heart anatomy, which is beneficial to improving the accuracy and reliability of the generated three-dimensional heart model, and thus beneficial to improving the accuracy and reliability of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, as well as the intelligence and efficiency of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, and further beneficial to improving the model accuracy of the generated three-dimensional heart model.

[0186] Example 3

[0187] Please see Figure 3 , Figure 3 This is a schematic diagram of a heart 3D modeling device based on point cloud generation disclosed in an embodiment of the present invention. Figure 3 As shown, the point cloud-based 3D heart modeling device may include:

[0188] Extraction module 301 is used to perform feature extraction operation on ultrasound volume images through a pre-determined UNet network to obtain volume feature maps;

[0189] The generation module 302 is used to generate a feature cost body corresponding to the volume feature map based on the volume feature map and a predetermined first processing operation, and to generate a feature probability body corresponding to the volume feature map based on the feature cost body and a predetermined second processing operation.

[0190] The determination module 303 is also used to determine the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm.

[0191] The generation module 302 is also used to generate target point cloud data based on predetermined data processing operations and depth estimation maps, and to generate a three-dimensional model of the heart corresponding to the ultrasound volume image based on the target point cloud data.

[0192] It is evident that implementation Figure 3The described device can perform feature extraction operations on ultrasound volumetric images through a UNet network to obtain a volumetric feature map, perform a first processing operation on the volumetric feature map to obtain a feature cost volume, perform a second processing operation on the feature cost volume to generate a feature probability volume, determine the depth estimation map corresponding to the ultrasound volumetric image based on the feature probability volume and the volumetric feature map combined with a depth map fusion algorithm and generate target point cloud data, and generate a three-dimensional heart model corresponding to the ultrasound volumetric image based on the target point cloud data. It can generate multi-resolution three-dimensional heart models based on a layer-by-layer progressive processing operation, which is beneficial to improving the accuracy and reliability of the generated three-dimensional heart models, as well as the intelligence and efficiency of the generated three-dimensional heart models. In addition, it is beneficial to further improve the generation and reconstruction accuracy of the three-dimensional heart models by extracting more semantic information of the heart structure.

[0193] In an optional embodiment, the generation module 302 generates target point cloud data based on predetermined data processing operations and a depth estimation map, and generates a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data in the following specific ways:

[0194] Based on the depth estimation map, generate backup point cloud data that matches the depth estimation map;

[0195] According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to obtain the target point cloud data; wherein, the target point cloud data includes grid point cloud data, and the grid point cloud data is used to construct a three-dimensional heart model and / or render a three-dimensional heart model;

[0196] Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated.

[0197] Determine the feature resolution information of the three-dimensional model of the spare heart, and based on the feature resolution information, determine whether the resolution of the three-dimensional model of the spare heart meets the preset model generation conditions.

[0198] When it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions, the model feature vector of the backup heart 3D model is extracted, and the fusion feature vector of the backup heart 3D model is determined. The backup heart 3D model is updated based on the model feature vector and the fusion feature vector to obtain the updated backup heart 3D model. The operation of determining the feature resolution information of the backup heart 3D model and the operation of determining whether the resolution of the backup heart 3D model meets the preset model generation conditions based on the feature resolution information are triggered again.

[0199] When it is determined that the resolution of the backup heart 3D model meets the preset model generation conditions, the backup heart 3D model is identified as the heart 3D model corresponding to the ultrasound volume image.

[0200] It is evident that implementation Figure 3 The described device can generate backup point cloud data based on a depth estimation map and perform data transformation operations to obtain target point cloud data. Based on the target point cloud data, it generates a backup 3D heart model corresponding to an ultrasound volumetric image. It determines the feature resolution information of the backup 3D heart model and judges whether it meets preset model generation conditions. If it does, the backup 3D heart model is determined to be a 3D heart model; otherwise, it extracts the model feature vector of the backup 3D heart model and determines the corresponding fusion feature vector. Based on the model feature vector and the fusion feature vector, the backup 3D heart model is updated, and the model generation conditions are re-evaluated. By continuously adding detailed information about the cardiac anatomy, it gradually approximates the real cardiac anatomy through progressive fine-tuning, which improves the accuracy and reliability of the generated 3D heart model, as well as its intelligence and efficiency. Furthermore, by generating the 3D heart model from progressive point cloud data, it further improves the accuracy and reliability of the generated 3D heart model and enhances its precision.

[0201] In another optional embodiment, the UNet network includes at least one convolutional layer and at least one residual block, wherein the extraction module 301 performs feature extraction operations on the ultrasound volume image through the pre-determined UNet network to obtain the volume feature map in the following specific ways:

[0202] By performing feature extraction operations on each convolutional layer in a pre-defined UNet network, at least one depth feature map is obtained from the ultrasound volume image.

[0203] For each depth feature map, the depth feature map is input into the residual block to obtain the output feature map corresponding to the depth feature map. Then, a feature addition operation is performed based on the depth feature map and the output feature map corresponding to the depth feature map to obtain the feature processing result corresponding to the depth feature map.

[0204] A volumetric feature map is generated based on the feature processing results corresponding to all depth feature maps.

[0205] It is evident that implementation Figure 3The described device can perform feature extraction operations on ultrasound volume images through convolutional layers in a UNet network to obtain depth feature maps, and obtain output feature maps corresponding to the depth feature maps through residual blocks. Based on the depth feature maps and the corresponding output feature maps, a feature addition operation is performed to obtain the feature processing results corresponding to each depth feature map, and a volume feature map is generated based on the feature processing results. It can perform multi-level feature extraction on ultrasound volume images through the UNet network, and obtain multi-resolution volume feature maps, which can enrich the data on which subsequent heart models are generated. This is beneficial to improving the accuracy and multi-level nature of the subsequently generated multi-resolution, multi-level 3D heart models. By introducing residual blocks into the UNet network, gradient propagation and generalization ability are promoted during training, enabling the network to learn deeper abstract features and obtain stronger semantic information. This is beneficial to improving the accuracy and multi-level nature of the subsequently generated multi-resolution, multi-level 3D heart models. Furthermore, by obtaining multi-dimensional semantic information about the heart structure, the accuracy and reliability of the subsequently generated 3D heart models are further improved, as well as the precision and intelligence of the subsequently generated 3D heart models.

[0206] In another optional embodiment, the generation module 302 generates the feature cost body corresponding to the volume feature map based on the volume feature map and the predetermined first processing operation in the following specific ways:

[0207] Based on the predetermined first processing operation, a homography transformation operation is performed on the volume feature map to obtain the feature mapping result, wherein the feature mapping result includes the homography matrix corresponding to the volume feature map and the feature depth corresponding to the volume feature map;

[0208] Based on the feature mapping results and the pre-determined camera parameters, the feature cost volume corresponding to the volume feature map is determined;

[0209] The first processing operation, which is predetermined, includes a homography transformation operation.

[0210] It is evident that implementation Figure 3The described apparatus can perform a homography transformation operation on a volumetric feature map based on a first processing operation including a homography transformation operation to obtain a feature mapping result. Based on the feature mapping result and camera parameters, it determines the feature cost body corresponding to the volumetric feature map. It can combine all extracted feature maps with camera parameters and map them to different parallel planes under the reference camera viewpoint to form feature maps through differentiable homography transformation. This is beneficial to improving the accuracy and reliability of the generated feature cost body, as well as the accuracy of the generated feature cost body. Furthermore, by generating the cost body, it can encode the set information in the camera parameters into the network and adapt to different numbers of multi-view image inputs. This is beneficial to improving the accuracy and reliability of subsequent generation of multi-dimensional and multi-level three-dimensional heart models, as well as the intelligence and efficiency of subsequent generation of multi-dimensional and multi-level three-dimensional heart models, and further improves the model accuracy of the generated three-dimensional heart model.

[0211] In yet another optional embodiment, the generation module 302 generates the feature probability volume corresponding to the volume feature map by processing the feature cost volume and the predetermined second processing operation in the following specific ways:

[0212] Based on the predetermined second processing operation, a regularization operation is performed on the feature cost body to remove the noise information contained in the feature cost body, thereby obtaining the target feature cost body.

[0213] Based on the target feature cost volume, generate the feature probability volume corresponding to the volume feature map;

[0214] The pre-determined second processing operation includes a regularization operation.

[0215] It is evident that implementation Figure 3 The described apparatus can perform regularization processing on the feature cost body based on a second processing operation including regularization processing to remove noise information contained in the feature cost body to obtain a target feature cost body, and generate a feature probability body corresponding to the volume feature map based on the target feature cost body. Regularization processing can remove noise information contained in the feature cost body, which is beneficial to improving the accuracy and reliability of the generated target feature cost body, preventing errors in the feature cost body due to noise interference. Furthermore, generating a feature probability body corresponding to the volume feature map based on the noise-removed target feature cost body can reduce the amount of noise information contained in the generated feature probability body, which is beneficial to improving the accuracy, precision, and intelligence of the generated feature probability body. This, in turn, is beneficial to improving the accuracy and reliability of subsequent generation of multi-dimensional, multi-level 3D heart models, as well as the intelligence and efficiency of subsequent generation of multi-dimensional, multi-level 3D heart models, and further beneficial to improving the model accuracy of the generated 3D heart model.

[0216] In another optional embodiment, the specific method by which the determining module 303 determines the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm, includes:

[0217] Based on the feature probability volume, volume feature map and pre-determined depth map fusion algorithm, the depth probability corresponding to each feature pixel point included in the volume feature map is determined;

[0218] The depth estimation map corresponding to the ultrasound volume image is determined based on the depth probability corresponding to each feature pixel.

[0219] It is evident that implementation Figure 3 The described device can determine the depth probability corresponding to each feature pixel in the volume feature map based on the feature probability volume and the volume feature map combined with the depth map fusion algorithm, and determine the depth estimation map corresponding to the ultrasound volume image based on the depth probability corresponding to each feature pixel. The accuracy of the estimated depth can be defined as the probability that the true depth is within a small range near the estimated depth, and the accuracy of the estimated depth can be determined by analyzing the probability distribution of the estimated depth. This can improve the accuracy and reliability of the determined depth probability corresponding to each feature pixel, as well as the intelligence and accuracy of the determined depth probability corresponding to each feature pixel. This is beneficial to improving the accuracy and reliability of the subsequent generation of multi-dimensional and multi-level three-dimensional cardiac models, as well as the intelligence and efficiency of the subsequent generation of multi-dimensional and multi-level three-dimensional cardiac models, and further improves the model accuracy of the generated three-dimensional cardiac models.

[0220] In yet another alternative embodiment, such as Figure 4 As shown, the determining module 303 is also used to determine a reference depth map corresponding to the ultrasound volume image; wherein, the reference depth map includes ultrasound image pixels of the ultrasound volume image and pixel coordinate information of the ultrasound volume image.

[0221] The device also includes:

[0222] The calculation module 304 is used to calculate the image difference parameters between the reference depth map and the depth estimation map, wherein the image difference parameters include the mean absolute error value between the reference depth map and the depth estimation map;

[0223] The judgment module 305 is used to determine whether the image difference parameters meet the preset image difference conditions.

[0224] The determining module 303 is also used to determine the parameters to be adjusted in the depth map fusion algorithm when the judging module 305 determines that the image difference parameters do not meet the preset image difference conditions;

[0225] The adjustment module 306 is used to perform a backpropagation adjustment operation on the depth map fusion algorithm based on the parameters to be adjusted, so as to update the depth map fusion algorithm.

[0226] It is evident that implementation Figure 4 The described device can determine the reference depth map corresponding to the ultrasound volumetric image, calculate the image difference parameters between the reference depth map and the depth estimation map, and determine whether the image difference parameters meet the preset image difference conditions. If not, it determines the parameters to be adjusted in the depth map fusion algorithm and performs a backpropagation adjustment operation on the depth map fusion algorithm to update it. It can calculate the loss of the initial depth map and the optimized depth map respectively, and backpropagate to adjust the network parameters, optimize the network performance, and continuously fine-tune the model performance. Finally, it can generate a three-dimensional model of the whole heart with four chambers that approximates the real heart anatomy, which is beneficial to improving the accuracy and reliability of the generated three-dimensional heart model, and thus to improving the accuracy and reliability of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, as well as the intelligence and efficiency of the subsequent generation of multi-dimensional and multi-level three-dimensional heart models, and further improve the model accuracy of the generated three-dimensional heart model.

[0227] Example 4

[0228] Please see Figure 5 , Figure 5 This is a schematic diagram of another heart 3D modeling device based on point cloud generation disclosed in an embodiment of the present invention. For example... Figure 5 As shown, the point cloud-based 3D heart modeling device may include:

[0229] Memory 401 storing executable program code;

[0230] Processor 402 coupled to memory 401;

[0231] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the heart 3D modeling method based on point cloud generation described in Embodiment 1 or Embodiment 2 of the present invention.

[0232] Example 5

[0233] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the point cloud-based three-dimensional heart modeling method described in Embodiment 1 or Embodiment 2 of this invention.

[0234] Example 6

[0235] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the point cloud-based three-dimensional modeling method for the heart described in Embodiment 1 or Embodiment 2.

[0236] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0237] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0238] Finally, it should be noted that the method and apparatus for three-dimensional modeling of the heart based on point cloud generation disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for 3D modeling of the heart based on point cloud generation, characterized in that, The method includes: Feature extraction is performed on the ultrasound volume image using a pre-defined UNet network to obtain a volume feature map; Based on the volume feature map and the predetermined first processing operation, a feature cost body corresponding to the volume feature map is generated, and a feature probability body corresponding to the volume feature map is generated based on the feature cost body and the predetermined second processing operation. Based on the feature probability volume and the volume feature map, and combined with a pre-determined depth map fusion algorithm, the depth estimation map corresponding to the ultrasound volume image is determined. Target point cloud data is generated based on the predetermined data processing operations and the depth estimation map, and a three-dimensional model of the heart corresponding to the ultrasound volume image is generated based on the target point cloud data. The step of generating target point cloud data based on predetermined data processing operations and the depth estimation map, and generating a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data, includes: Based on the depth estimation map, generate backup point cloud data that matches the depth estimation map; According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to obtain the target point cloud data; wherein, the target point cloud data includes grid point cloud data, and the grid point cloud data is used to construct a three-dimensional heart model and / or render a three-dimensional heart model; Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated; Determine the feature resolution information of the backup heart 3D model, and based on the feature resolution information, determine whether the resolution corresponding to the backup heart 3D model meets the preset model generation conditions; When it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions, the model feature vector of the backup heart 3D model is extracted, and the fusion feature vector of the backup heart 3D model is determined. An update operation is performed on the backup heart 3D model based on the model feature vector and the fusion feature vector to obtain an updated backup heart 3D model. The operation of determining the feature resolution information of the backup heart 3D model and determining whether the resolution of the backup heart 3D model meets the preset model generation conditions based on the feature resolution information are triggered again. When it is determined that the resolution of the backup heart 3D model meets the preset model generation conditions, the backup heart 3D model is identified as the heart 3D model corresponding to the ultrasound volume image. The UNet network includes at least one convolutional layer and at least one residual block, wherein the feature extraction operation on the ultrasound volume image using the pre-determined UNet network to obtain a volume feature map includes: The ultrasound volume image is subjected to feature extraction operation by each of the convolutional layers in the pre-determined UNet network to obtain at least one depth feature map; For each depth feature map, the depth feature map is input into the residual block to obtain the output feature map corresponding to the depth feature map. Then, a feature addition operation is performed based on the depth feature map and the output feature map corresponding to the depth feature map to obtain the feature processing result corresponding to the depth feature map. Based on the feature processing results corresponding to all the depth feature maps, a volume feature map is generated; The step of generating the feature cost body corresponding to the volume feature map based on the volume feature map and the pre-determined first processing operation includes: Based on a predetermined first processing operation, a homography transformation operation is performed on the volume feature map to obtain a feature mapping result, wherein the feature mapping result includes the homography matrix corresponding to the volume feature map and the feature depth corresponding to the volume feature map; Based on the feature mapping results and the pre-determined camera parameters, the feature cost volume corresponding to the volume feature map is determined; The first processing operation, which is predetermined, includes the homography transformation operation; The step of generating the feature probability volume corresponding to the volume feature map by processing the feature cost volume and the pre-determined second processing operation includes: Based on the predetermined second processing operation, a regularization operation is performed on the feature cost body to remove the noise information contained in the feature cost body, thereby obtaining the target feature cost body; Based on the target feature cost volume, a feature probability volume corresponding to the volume feature map is generated; The second processing operation, which is predetermined, includes the regularization processing operation.

2. The method for three-dimensional modeling of the heart based on point cloud generation according to claim 1, characterized in that, The step of determining the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm, includes: Based on the feature probability volume, the volume feature map, and a pre-determined depth map fusion algorithm, the depth probability corresponding to each feature pixel in the volume feature map is determined. The depth estimation map corresponding to the ultrasound volume image is determined based on the depth probability corresponding to each feature pixel.

3. The method for three-dimensional modeling of the heart based on point cloud generation according to any one of claims 1-2, characterized in that, The method further includes: Determine a reference depth map corresponding to the ultrasound volume image; wherein, the reference depth map includes ultrasound image pixels of the ultrasound volume image and pixel coordinate information of the ultrasound volume image; Calculate the image difference parameter between the reference depth map and the depth estimation map, wherein the image difference parameter includes the mean absolute error value between the reference depth map and the depth estimation map; Determine whether the image difference parameters meet the preset image difference conditions; When it is determined that the image difference parameters do not meet the preset image difference conditions, the parameters to be adjusted in the depth map fusion algorithm are determined, and based on the parameters to be adjusted, a backpropagation adjustment operation is performed on the depth map fusion algorithm to update the depth map fusion algorithm.

4. A three-dimensional modeling device for the heart based on point cloud generation, characterized in that, The device includes: The extraction module is used to perform feature extraction operations on the ultrasound volume image through a pre-determined UNet network to obtain a volume feature map; The generation module is used to generate a feature cost body corresponding to the volume feature map based on the volume feature map and a pre-determined first processing operation, and to generate a feature probability body corresponding to the volume feature map based on the feature cost body and a pre-determined second processing operation. The determining module is also used to determine the depth estimation map corresponding to the ultrasound volume image based on the feature probability volume and the volume feature map, combined with a pre-determined depth map fusion algorithm; The generation module is further configured to generate target point cloud data based on the predetermined data processing operations and the depth estimation map, and generate a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data. The generation module generates target point cloud data based on predetermined data processing operations and the depth estimation map, and generates a three-dimensional cardiac model corresponding to the ultrasound volume image based on the target point cloud data. The specific methods for generating this model include: Based on the depth estimation map, generate backup point cloud data that matches the depth estimation map; According to the predetermined data processing operations, a data transformation operation is performed on the backup point cloud data to obtain the target point cloud data; wherein, the target point cloud data includes grid point cloud data, and the grid point cloud data is used to construct a three-dimensional heart model and / or render a three-dimensional heart model; Based on the target point cloud data, a backup three-dimensional model of the heart corresponding to the ultrasound volume image is generated; Determine the feature resolution information of the backup heart 3D model, and based on the feature resolution information, determine whether the resolution corresponding to the backup heart 3D model meets the preset model generation conditions; When it is determined that the resolution of the backup heart 3D model does not meet the preset model generation conditions, the model feature vector of the backup heart 3D model is extracted, and the fusion feature vector of the backup heart 3D model is determined. An update operation is performed on the backup heart 3D model based on the model feature vector and the fusion feature vector to obtain an updated backup heart 3D model. The operation of determining the feature resolution information of the backup heart 3D model and determining whether the resolution of the backup heart 3D model meets the preset model generation conditions based on the feature resolution information are triggered again. When it is determined that the resolution of the backup heart 3D model meets the preset model generation conditions, the backup heart 3D model is identified as the heart 3D model corresponding to the ultrasound volume image. The UNet network includes at least one convolutional layer and at least one residual block. The extraction module performs feature extraction on the ultrasound volume image using the pre-determined UNet network to obtain the volume feature map. The specific methods for this are as follows: The ultrasound volume image is subjected to feature extraction operation by each of the convolutional layers in the pre-determined UNet network to obtain at least one depth feature map; For each depth feature map, the depth feature map is input into the residual block to obtain the output feature map corresponding to the depth feature map. Then, a feature addition operation is performed based on the depth feature map and the output feature map corresponding to the depth feature map to obtain the feature processing result corresponding to the depth feature map. Based on the feature processing results corresponding to all the depth feature maps, a volume feature map is generated; The specific method by which the generation module generates the feature cost body corresponding to the volume feature map based on the volume feature map and the pre-determined first processing operation includes: Based on a predetermined first processing operation, a homography transformation operation is performed on the volume feature map to obtain a feature mapping result, wherein the feature mapping result includes the homography matrix corresponding to the volume feature map and the feature depth corresponding to the volume feature map; Based on the feature mapping results and the pre-determined camera parameters, the feature cost volume corresponding to the volume feature map is determined; The first processing operation, which is predetermined, includes the homography transformation operation; The specific method by which the generation module generates the feature probability volume corresponding to the volume feature map based on the feature cost volume and the pre-determined second processing operation includes: Based on the predetermined second processing operation, a regularization operation is performed on the feature cost body to remove the noise information contained in the feature cost body, thereby obtaining the target feature cost body; Based on the target feature cost volume, a feature probability volume corresponding to the volume feature map is generated; The second processing operation, which is predetermined, includes the regularization processing operation.

5. A three-dimensional modeling device for the heart based on point cloud generation, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the heart 3D modeling method based on point cloud generation as described in any one of claims 1-3.

6. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute a method for three-dimensional modeling of the heart based on point cloud generation as described in any one of claims 1-3.

Images