A three-dimensional reconstruction method and related device for the right ventricle based on echocardiogram

Abstract

The present application discloses a method and related device for three-dimensional reconstruction of the right ventricle based on echocardiogram, which relates to the fields of deep learning and three-dimensional reconstruction. The method includes: obtaining contour annotation data of the right ventricle of the heart in the echocardiogram; generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data; inputting the contour point voxel data and the dense convex hull voxel data into a trained fully convolutional neural network model to obtain a three-dimensional voxel model of the right ventricle of the heart. The present invention can accurately and quickly construct a three-dimensional model of the right ventricle based on echocardiogram.

Description

Technical Field

[0001] This application relates to the fields of deep learning and three-dimensional reconstruction, and particularly to a method, device, equipment, medium and product for three-dimensional reconstruction of the right ventricle based on echocardiogram. Background Art

[0002] Echocardiogram has the advantages of high real-time performance, harmlessness to the human body, low cost, high portability, etc., and is the most common diagnostic method for cardiovascular diseases. Using echocardiogram to reconstruct the three-dimensional model of the heart ventricle can, on the one hand, help reproduce the three-dimensional model of the ventricle at low cost and conveniently to assist surgery, and at the same time help detect ventricular diseases and cardiac function indexes. For example, it can help measure the most common parameter when evaluating cardiac function: ejection fraction (EF). When calculating the EF value, it is necessary to measure the volume of the ventricle at the end of systole and the end of diastole. However, due to the irregular shape of the right ventricle, its volume is usually difficult to be directly measured. Therefore, three-dimensional reconstruction using the right ventricle contour in ultrasonic images is of great significance for measuring three-dimensional indexes of cardiac function and reproducing the three-dimensional model to assist surgery.

[0003] Echocardiogram data has the characteristics of parallel image sections and sparse data. Echocardiogram is a sectional view of the heart detected by a doctor holding a probe at different positions and angles on the patient's chest. Therefore, the sectional views of echocardiogram images are first non-parallel sectional data with random geometric relationships. In addition, due to conditions such as rib occlusion, the sections of echocardiogram images are generally sparse. Therefore, three-dimensional reconstruction of the right ventricle based on echocardiogram contour is of great difficulty.

[0004] Currently, the process of three-dimensional reconstruction of the right ventricle using echocardiogram is mainly completed by the knowledge-based method (KBR). Doctors need to mark the coordinate values of key points representing each anatomical structure of the right ventricle on the echocardiogram, and then the KBR algorithm performs three-dimensional reconstruction by deforming a template model according to the key points of the anatomical structure. However, such a process is time-consuming and laborious, requires operators to have certain professional knowledge, and the accuracy is uncontrollable. Currently, there is no automated method that can quickly and accurately perform three-dimensional reconstruction on the contour image of ultrasound.

[0005] The development of artificial intelligence has simultaneously promoted the development of computer vision applications. Using deep learning to solve the problem of three-dimensional reconstruction, the most common method currently is to use a deep learning model for multi-view three-dimensional reconstruction. However, when performing three-dimensional reconstruction using marked contours on echocardiograms, due to the data characteristics of echocardiograms, it is impossible to directly use the multi-view three-dimensional reconstruction model. Multi-view three-dimensional reconstruction requires inputting images of an object from different perspectives to perform three-dimensional reconstruction on the object. Obviously, the contour data of echocardiograms does not meet the input requirements of multi-view reconstruction. Therefore, in order to utilize the powerful three-dimensional reconstruction ability of deep learning, it is necessary to design a brand-new algorithm framework suitable for three-dimensional reconstruction of the right ventricle from echocardiogram contours. Summary of the Invention

[0006] The purpose of this application is to provide a method and related device for three-dimensional reconstruction of the right ventricle based on echocardiograms, which can accurately and quickly construct a three-dimensional model of the right ventricle based on echocardiograms.

[0007] To achieve the above purpose, this application provides the following solutions:

[0008] In the first aspect, this application provides a method for three-dimensional reconstruction of the right ventricle based on echocardiograms, including:

[0009] Obtain the contour annotation data of the right ventricle of the heart in the echocardiogram;

[0010] Generate contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data;

[0011] Input the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model to obtain the three-dimensional voxel model of the right ventricle of the heart.

[0012] In the second aspect, this application provides a device for three-dimensional reconstruction of the right ventricle based on echocardiograms, including:

[0013] A data acquisition unit for obtaining the contour annotation data of the right ventricle of the heart in the echocardiogram;

[0014] A three-dimensional voxel data generation unit for generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data;

[0015] A three-dimensional reconstruction unit for inputting the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model to obtain the three-dimensional voxel model of the right ventricle of the heart.

[0016] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method for three-dimensional reconstruction of the right ventricle based on echocardiogram.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for three-dimensional reconstruction of the right ventricle based on echocardiogram.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method for three-dimensional reconstruction of the right ventricle based on echocardiogram.

[0019] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0020] The present application provides a method for three-dimensional reconstruction of the right ventricle based on echocardiogram and related devices. According to the contour annotation data, contour point voxel data and dense convex hull voxel data of three-dimensional contour points are generated in the three-dimensional voxel space, and the contour point voxel data and dense convex hull voxel data are input into the trained fully convolutional neural network model to obtain the three-dimensional voxel model of the right ventricle of the heart. Among them, dual-channel data sampling is used to construct voxel data according to the characteristics of echocardiogram images, so that echocardiogram data with any number of cross-sections and cross-sectional geometric relationships can be input into the fully convolutional neural network model; the algorithm still has good robustness for sparse echocardiogram data, and can still ensure high algorithm accuracy. In practical applications, the three-dimensional reconstruction automation solution of the present invention has the advantages of fast speed and accuracy, can greatly reduce the workload and time of manual calculation, and can accurately and quickly construct the three-dimensional model of the right ventricle. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is an application environment diagram of a method for three-dimensional reconstruction of the right ventricle based on echocardiogram in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a method for three-dimensional reconstruction of the right ventricle based on echocardiogram provided by an embodiment of the present application;

[0024] Figure 3 A schematic diagram of an example structure of a fully convolutional neural network model provided by an embodiment of the present application;

[0025] Figure 4 A schematic diagram for comparing the three-dimensional prediction structure of the right ventricle of the fully convolutional neural network model provided by an embodiment of the present application with the three-dimensional true structure of the right ventricle;

[0026] Figure 5 A schematic diagram of the functional modules of a three-dimensional right ventricle reconstruction device based on echocardiogram provided by an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present application. Specific embodiments

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0029] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] The three-dimensional right ventricle reconstruction method based on echocardiogram provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set separately, integrated on the server 104, placed on the cloud or other servers. The terminal 102 can send the contour annotation data of the right ventricle of the heart in the echocardiogram to the server 104. After receiving the contour annotation data of the right ventricle of the heart in the echocardiogram, for the contour annotation data of the right ventricle of the heart in the echocardiogram, the server 104 generates contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data;

[0031] Input the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model to obtain a three-dimensional voxel model of the right ventricle of the heart. The server 104 can feedback the obtained three-dimensional voxel model of the right ventricle of the heart to the terminal 102. In addition, in some embodiments, the method for three-dimensional reconstruction of the right ventricle based on echocardiogram can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform three-dimensional reconstruction of the right ventricle of the heart on the contour annotation data of the right ventricle of the heart in the echocardiogram, or the server 104 can obtain the contour annotation data of the right ventricle of the heart in the echocardiogram from the data storage system and perform three-dimensional reconstruction of the right ventricle of the heart on the contour annotation data of the right ventricle of the heart in the echocardiogram.

[0032] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smartphones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0033] In an exemplary embodiment, as Figure 2 shown, a method for three-dimensional reconstruction of the right ventricle based on echocardiogram is provided. This method is executed by a computer device, and specifically can be executed alone by a computer device such as a terminal or a server, or can be jointly executed by a terminal and a server. In the embodiments of the present application, taking this method applied to Figure 1 the server 104 in it as an example for illustration, it includes the following steps 201 to step 203.

[0034] Step 201, obtain the contour annotation data of the right ventricle of the heart in the echocardiogram.

[0035] The contour annotation data of the echocardiogram includes the echocardiogram data of the ventricle and the surface contour of the ventricle marked in the echocardiogram data, where the ventricle contour is the ventricle contour line manually marked in the echocardiogram.

[0036] In this step, the three-dimensional parameters of the echocardiogram acquisition device can also be obtained, including 3D transformation parameters and calibration parameters.

[0037] Step 202, generate contour point voxel data (corresponding to Figure 3 contours in it) and dense convex hull voxel data of three-dimensional contour points (corresponding to Figure 3 convex hull in it) in the three-dimensional voxel space according to the contour annotation data.

[0038] Step 203: Input the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model (corresponding to the deep learning network in Figure 3 ) to obtain a three-dimensional voxel model of the right ventricle of the heart.

[0039] Implementing the above steps 201 to 203 provides an automated method framework for directly obtaining a three-dimensional model of the right ventricle based on echocardiogram images. Different from existing methods (such as the knowledge-based reconstruction method KBR), the present invention uses dual-channel data sampling and deep neural network technology, and can input echocardiogram data with any number of cross-sections and cross-sectional geometric relationships; the algorithm still has good robustness for sparse echocardiogram data and can still ensure high algorithm accuracy. In practical applications, this automated solution has the advantages of being fast and accurate, can greatly reduce the workload and time of manual calculation, and greatly improve the efficiency of three-dimensional reconstruction.

[0040] In another exemplary embodiment of the present application, point sampling is first performed on the contour of the echocardiogram, and then three-dimensional spatial matrix transformation is performed on these sampled two-dimensional points using the three-dimensional parameters of the ultrasound acquisition device to obtain a set of points of these two-dimensional points in three-dimensional space. That is, a set of points is sampled from the contour of the echocardiogram so that these points are evenly distributed on the contour. In this way, a coordinate set of two-dimensional points is obtained. Next, the dimension of these two-dimensional point coordinates is expanded to convert them into a coordinate set of three-dimensional points. In order to map the two-dimensional point coordinates into three-dimensional space, a three-dimensional transformation matrix is generated using the information in the ultrasound acquisition device, and then the three-dimensional point coordinates are multiplied by the three-dimensional spatial matrix to obtain a three-dimensional coordinate set of the contour sampling points, that is, a three-dimensional contour sampling point set is obtained. In the dual-channel processing, two channels are used to process the three-dimensional contour point data. The first channel performs point sampling on the three-dimensional contour points. Specifically, a voxel grid is created in a three-dimensional voxel space centered at the origin. For example, the size of the grid is 128*128*128, and the side length of each voxel is a fixed value (for example, 0.016). Then, the three-dimensional contour sampling points are sampled in this voxel grid (the voxel value corresponding to the three-dimensional contour sampling points is 1, and the voxel value of non-three-dimensional contour sampling points is 0) to obtain a voxel data representing the distribution of the three-dimensional contour sampling points, that is, contour point voxel data. At the same time, the other channel performs convex hull solution on the three-dimensional contour sampling points and performs dense convex hull sampling in the same three-dimensional voxel space (the value of the three-dimensional dense convex hull in the voxel space is represented as 1 (the voxel values of the convex hull surface and internal points are 1), and the voxel value of non-dense convex hull points is 0). Specifically, a convex hull operation is performed on the three-dimensional contour sampling points (implemented using an existing convex hull algorithm, such as the quickhull convex hull algorithm) to obtain voxel data representing the convex hull shape of the three-dimensional contour sampling points, that is, dense convex hull voxel data, and its size is the same as that of the voxel grid, which is 128*128*128. Finally, by fusing the data of the two channels, the input data required for the neural network model in the subsequent steps is obtained, and the size is 2*128*128*128. Therefore, in the present invention, step 202, generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in a three-dimensional voxel space according to the contour annotation data specifically includes:

[0041] (1) Uniformly distributed point sampling is performed on the marked contour of the right ventricle of the heart in the echocardiogram to obtain contour sampling points.

[0042] (2) Three-dimensional spatial matrix transformation is performed on the contour sampling points using the 3D transformation parameters and calibration parameters of the echocardiogram data to obtain three-dimensional contour sampling points.

[0043] For the sake of consistency, the coordinates of the three-dimensional contour sampling points are normalized to ensure that their coordinate ranges are within a fixed range (-1 to 1). Therefore, after obtaining the three-dimensional contour sampling points, it further includes normalizing the three-dimensional contour sampling points to obtain normalized three-dimensional contour sampling points and recording the normalization scaling ratio.

[0044] (3) Represent the three-dimensional contour sampling points in the three-dimensional voxel space to obtain contour point voxel data.

[0045] (4) Determine the dense convex hull of the three-dimensional contour points according to the three-dimensional contour sampling points.

[0046] (5) Voxelize the dense convex hull of the three-dimensional contour points in the three-dimensional voxel space to obtain the dense convex hull voxel data.

[0047] In another exemplary embodiment of the present application, the fully convolutional neural network model designed by the present invention has an encoder-decoder structure, and the encoder and the decoder together form a U-shaped network with an encoding and decoding structure. The encoder is responsible for feature extraction and downsampling of the input data; the decoder is responsible for restoring the feature map extracted by the encoder to the original resolution and generating the final three-dimensional voxel reconstruction result. The encoder and the decoder are connected by skip connections for feature fusion. The input three-dimensional data size of the fully convolutional neural network model is voxel data of C*D*H*W, where C represents the number of channels, D represents the depth, H represents the height, and W represents the width; the output size of the decoder in the fully convolutional neural network model is voxel data of D*H*W.

[0048] As Figure 3 shown, the fully convolutional neural network model includes an encoder and a decoder.

[0049] The encoder includes an input convolution module, a plurality of encoding convolution modules, and a plurality of downsampling modules; the decoder includes a plurality of decoding convolution modules, a plurality of skip connection modules, a plurality of upsampling modules, and an output convolution module. Figure 3 In, an example is given where 4 encoding convolution modules, downsampling modules, decoding convolution modules, skip connection modules, and upsampling modules are used. Of course, other values can also be selected according to requirements, and no limitations are made here.

[0050] According to the data flow direction from the encoder to the decoder, a plurality of encoding convolution modules are defined as the first encoding convolution module,..., the nth encoding convolution module,..., the Nth encoding convolution module, and a plurality of downsampling modules are defined as the first downsampling module,..., the nth downsampling module,..., the Nth downsampling module.

[0051] Define several decoding convolutional modules as the first decoding convolutional module, ..., the n-th decoding convolutional module, ..., the N-th decoding convolutional module in the reverse direction of the data flow from the encoder to the decoder, several skip connection modules as the first skip connection module, ..., the n-th skip connection module, ..., the N-th skip connection module, and several upsampling modules as the first upsampling module, ..., the n-th upsampling module, ..., the N-th upsampling module.

[0052] The input of the input convolutional module is the contour point voxel data and the dense convex hull voxel data; the output of the input convolutional module is connected to the input of the first encoding convolutional module and the input of the first skip connection module.

[0053] The output of the first encoding convolutional module is connected to the input of the first downsampling module, and the output of the first downsampling module is respectively connected to the input of the second encoding convolutional module and the input of the second skip connection module.

[0054] The output of the n-th encoding convolutional module is connected to the input of the n-th downsampling module, and the output of the n-th downsampling module is respectively connected to the input of the (n + 1)-th encoding convolutional module and the input of the (n + 1)-th skip connection module; n = 2, ..., N - 1.

[0055] The output of the N-th encoding convolutional module is connected to the input of the N-th downsampling module; the output of the N-th downsampling module is connected to the input of the N-th decoding convolutional module.

[0056] The output of the N-th decoding convolutional module is connected to the input of the N-th upsampling module, the output of the N-th upsampling module is connected to the input of the N-th skip connection module, and the output of the N-th skip connection module is connected to the input of the (N - 1)-th decoding convolutional module.

[0057] The output of the n-th decoding convolutional module is connected to the input of the n-th upsampling module, the output of the n-th upsampling module is connected to the input of the n-th skip connection module, and the output of the n-th skip connection module is connected to the input of the (n - 1)-th decoding convolutional module.

[0058] The output of the first decoding convolutional module is connected to the input of the first upsampling module, the output of the first upsampling module is connected to the input of the first skip connection module, and the output of the first skip connection module is connected to the input of the output convolutional module. The output of the output convolutional module is the three-dimensional voxel model of the right ventricle of the heart.

[0059] In another exemplary embodiment of the present application, as Figure 3 shown, after performing step 203 "input the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model to obtain the three-dimensional voxel model of the right ventricle of the heart", it further includes:

[0060] Restore the three-dimensional voxel model of the right ventricle of the heart according to the normalized scaling ratio; calculate the volume of the restored three-dimensional voxel model; calculate the ejection fraction according to the volumes at the diastolic and systolic ends of the restored three-dimensional voxel model.

[0061] Perform a simple proportional reduction post-processing operation on the prediction result to obtain the volume of the corresponding right ventricle in a specific cycle. Finally, select the ventricular volume results at the diastolic and systolic ends of its right ventricle and calculate its ejection fraction index.

[0062] In another exemplary embodiment of the present application, before performing step 203 "input the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model", it further includes: training the fully convolutional neural network model, configuring a suitable software and hardware environment to train the fully convolutional neural network model. When training, the optimizer uses Adam for iteration, the initial learning rate is 1e-6, the learning rate adjustment strategy adopts the cosine annealing strategy, and the batchsize is 24. Use a mixed function of binary cross-entropy and Dice coefficient as the loss function for training.

[0063] The specific training process is as follows:

[0064] (a) Obtain the contour annotation sample data of the right ventricle of the heart in echocardiogram.

[0065] Obtain echocardiogram images of the hearts of multiple heart patients at different cycles, as well as the corresponding right ventricle contour annotations, 3D transformation parameters and calibration parameters of the echocardiogram data.

[0066] (b) Generate contour point voxel sample data and dense convex hull voxel sample data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation sample data.

[0067] (c) Input the contour point voxel sample data and the dense convex hull voxel sample data into the fully convolutional neural network model until the loss error of the model converges or the number of training iterations reaches the preset number of iterations, and obtain the trained fully convolutional neural network model.

[0068] After training is completed, further verify the model effect using the test set. Input the contour point voxel sample data and the dense convex hull voxel sample data in the test set into the fully convolutional neural network model to obtain the three-dimensional voxel model of the right ventricle predicted by the test samples. After restoring according to the normalized scaling ratio, compare it with the real three-dimensional model of the right ventricle. The comparison results are as Figure 4 shown. It can be seen that the present invention can accurately and efficiently predict the three-dimensional model of the right ventricle based on the echocardiogram contour.

[0069] In this embodiment, dual-channel data sampling is used to construct voxel data according to the characteristics of echocardiogram images, and then prediction is performed in the three-dimensional voxel space through a deep learning network, and finally a right ventricular model of three-dimensional data is obtained. First, dual-channel point sampling and dense convex hull sampling voxel data are generated from echocardiogram images annotated with the right ventricular contour as the input of the neural network; then, a fully convolutional neural network based on encoding-decoding is used to reconstruct the three-dimensional right ventricular model; then a series of post-processing measurement operations are performed on the prediction results, and finally a three-dimensional right ventricular model and three-dimensional cardiac function index data are obtained.

[0070] The present application also provides an application scenario that applies the above-mentioned method for three-dimensional reconstruction of the right ventricle based on echocardiogram. Specifically: The method for three-dimensional reconstruction of the right ventricle based on echocardiogram provided in this embodiment can be applied to the three-dimensional reconstruction scenario of the right ventricle. The three-dimensional reconstruction scenario of the right ventricle includes a data acquisition link and a three-dimensional reconstruction link; the echocardiogram contour annotation data enters the three-dimensional reconstruction link from the data acquisition link. In the method for three-dimensional reconstruction of the right ventricle based on echocardiogram provided in this embodiment, obtaining the contour annotation data of the right ventricle of the heart in the echocardiogram belongs to the data acquisition link, generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data, and inputting the contour point voxel data and dense convex hull voxel data into the trained fully convolutional neural network model to obtain the three-dimensional voxel model of the right ventricle of the heart belongs to the three-dimensional reconstruction link.

[0071] Based on the same inventive concept, an embodiment of the present application also provides a device for implementing the above-mentioned three-dimensional reconstruction of the right ventricle based on echocardiogram. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the device for three-dimensional reconstruction of the right ventricle based on echocardiogram provided below can refer to the limitations on the method for three-dimensional reconstruction of the right ventricle based on echocardiogram in the above text, and will not be repeated here.

[0072] In an exemplary embodiment, as Figure 5 shown, a device for three-dimensional reconstruction of the right ventricle based on echocardiogram includes:

[0073] A data acquisition unit M1 for obtaining the contour annotation data of the right ventricle of the heart in the echocardiogram.

[0074] A three-dimensional voxel data generation unit M2 for generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data.

[0075] A three-dimensional reconstruction unit M3 is configured to input the contour point voxel data and the dense convex hull voxel data into a trained fully convolutional neural network model to obtain a three-dimensional voxel model of the right ventricle of the heart.

[0076] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structural diagram can be as shown in Figure 6 the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video tag processing data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for three-dimensional reconstruction of the right ventricle based on echocardiography.

[0077] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0078] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0079] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0080] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0082] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0083] The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0084] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0085] In this article, specific examples are used to elaborate on the principles and implementation modes of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation modes and application scopes. To sum up, the content of this specification should not be construed as a limitation to this application.

Claims

1. A three-dimensional reconstruction method of the right ventricle based on echocardiogram, characterized in that, The method for three-dimensional reconstruction of the right ventricle based on echocardiogram includes: Obtaining the contour annotation data of the right ventricle of the heart in the echocardiogram; Generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data; Inputting the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model to obtain a three-dimensional voxel model of the right ventricle of the heart; Among them, generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data specifically includes: Performing point sampling at evenly distributed points on the marked contour of the right ventricle of the heart in the echocardiogram to obtain contour sampling points; Performing three-dimensional space matrix transformation on the contour sampling points by using the 3D transformation parameters and calibration parameters of the echocardiogram data to obtain three-dimensional contour sampling points; Representing the three-dimensional contour sampling points in the three-dimensional voxel space to obtain contour point voxel data; Determining the dense convex hull of the three-dimensional contour points according to the three-dimensional contour sampling points; Voxelizing the dense convex hull of the three-dimensional contour points in the three-dimensional voxel space to obtain the dense convex hull voxel data.

2. The three-dimensional reconstruction method of the right ventricle based on echocardiogram according to claim 1, characterized in that Before performing the step of "representing the three-dimensional contour sampling points in the three-dimensional voxel space to obtain contour point voxel data", the method for three-dimensional reconstruction of the right ventricle based on echocardiogram further includes: Performing normalization processing on the three-dimensional contour sampling points to obtain normalized three-dimensional contour sampling points and recording the normalization scaling ratio.

3. The three-dimensional right ventricle reconstruction method based on echocardiogram according to claim 1, characterized in that, The fully convolutional neural network model includes an encoder and a decoder; The encoder includes an input convolutional module, a plurality of encoding convolutional modules, and a plurality of downsampling modules; the decoder includes a plurality of decoding convolutional modules, a plurality of skip connection modules, a plurality of upsampling modules, and an output convolutional module; According to the data flow direction from the encoder to the decoder, a plurality of encoding convolutional modules are defined as the first encoding convolutional module,..., the nth encoding convolutional module,..., the Nth encoding convolutional module, and a plurality of downsampling modules are defined as the first downsampling module,..., the nth downsampling module,..., the Nth downsampling module; According to the reverse data flow direction from the encoder to the decoder, a plurality of decoding convolutional modules are defined as the first decoding convolutional module,..., the nth decoding convolutional module,..., the Nth decoding convolutional module, a plurality of skip connection modules are defined as the first skip connection module,..., the nth skip connection module,..., the Nth skip connection module, and a plurality of upsampling modules are defined as the first upsampling module,..., the nth upsampling module,..., the Nth upsampling module; The input of the input convolutional module is the contour point voxel data and the dense convex hull voxel data; the output of the input convolutional module is connected to the input of the first encoding convolutional module and the input of the first skip connection module; The output of the first encoding convolutional module is connected to the input of the first downsampling module, and the output of the first downsampling module is respectively connected to the input of the second encoding convolutional module and the input of the second skip connection module; The output of the n-th encoded convolutional module is connected to the input of the n-th downsampling module, and the output of the n-th downsampling module is respectively connected to the input of the (n + 1)-th encoded convolutional module and the input of the (n + 1)-th skip connection module; n = 2,..., N - 1; The output of the N-th encoded convolutional module is connected to the input of the N-th downsampling module; the output of the N-th downsampling module is connected to the input of the N-th decoded convolutional module; The output of the N-th decoded convolutional module is connected to the input of the N-th upsampling module, the output of the N-th upsampling module is connected to the input of the N-th skip connection module, and the output of the N-th skip connection module is connected to the input of the (N - 1)-th decoded convolutional module; The output of the n-th decoded convolutional module is connected to the input of the n-th upsampling module, the output of the n-th upsampling module is connected to the input of the n-th skip connection module, and the output of the n-th skip connection module is connected to the input of the (n - 1)-th decoded convolutional module; The output of the first decoded convolutional module is connected to the input of the first upsampling module, the output of the first upsampling module is connected to the input of the first skip connection module, and the output of the first skip connection module is connected to the input of the output convolutional module, and the output of the output convolutional module is a three-dimensional voxel model of the right ventricle of the heart.

4. The three-dimensional right ventricle reconstruction method based on echocardiogram according to claim 2, characterized in that After performing the step of "inputting the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model to obtain a three-dimensional voxel model of the right ventricle of the heart", the three-dimensional reconstruction method of the right ventricle based on echocardiography further includes: Restoring the three-dimensional voxel model of the right ventricle of the heart according to the normalized scaling ratio; Calculating the volume of the restored three-dimensional voxel model; Calculating the ejection fraction according to the volumes of the restored three-dimensional voxel model at the end-diastolic and end-systolic phases.

5. The three-dimensional reconstruction method of the right ventricle based on echocardiogram according to claim 1, characterized in that, Before performing the step of "inputting the contour point voxel data and the dense convex hull voxel data into the trained fully convolutional neural network model", it further includes: training the fully convolutional neural network model, specifically including: Obtaining contour annotation sample data of the right ventricle of the heart in the echocardiogram; Generating contour point voxel sample data and dense convex hull voxel sample data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation sample data; Inputting the contour point voxel sample data and the dense convex hull voxel sample data into the fully convolutional neural network model until the loss error of the model converges or the number of training iterations reaches a preset number of iterations, to obtain the trained fully convolutional neural network model.

6. A three-dimensional reconstruction device for the right ventricle based on echocardiogram, characterized in that, The three-dimensional reconstruction device of the right ventricle based on echocardiography includes: A data acquisition unit for acquiring contour annotation data of the right ventricle of the heart in the echocardiogram; A three-dimensional voxel data generation unit for generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data; Among them, generating contour point voxel data and dense convex hull voxel data of three-dimensional contour points in the three-dimensional voxel space according to the contour annotation data specifically includes: Performing uniformly distributed point sampling on the marked contour of the right ventricle of the heart in the echocardiogram to obtain contour sampling points; Performing a three-dimensional spatial matrix transformation on the contour sampling points by using 3D transformation parameters and calibration parameters of echocardiogram data to obtain three-dimensional contour sampling points; Representing the three-dimensional contour sampling points in a three-dimensional voxel space to obtain contour point voxel data; Determining a dense convex hull of the three-dimensional contour points according to the three-dimensional contour sampling points; Voxelizing the dense convex hull of the three-dimensional contour points in the three-dimensional voxel space to obtain the dense convex hull voxel data; A three-dimensional reconstruction unit, configured to input the contour point voxel data and the dense convex hull voxel data into a trained fully convolutional neural network model to obtain a three-dimensional voxel model of the right ventricle of the heart.

7. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the echocardiogram-based right ventricle three-dimensional reconstruction method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the echocardiogram-based right ventricle three-dimensional reconstruction method according to any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the echocardiogram-based right ventricle three-dimensional reconstruction method according to any one of claims 1-5.

Images