Three-dimensional ultrasonic modeling method and system, electronic device and readable storage medium

Abstract

The present invention provides a three-dimensional ultrasonic modeling method and system, an electronic device and a readable storage medium. The method comprises: using a first generation model to carry out domain conversion on each frame of ultrasonic image in an acquired target organ ultrasonic image sequence, so as to acquire a corresponding pseudo target domain image sequence, wherein the target organ ultrasonic image sequence comprises multiple frames of ultrasonic images collected by an ultrasonic probe in the cavity of a target organ at different poses and different depths; on the basis of the pseudo target domain image sequence, acquiring a sparse pseudo target domain three-dimensional image, and on the basis of the sparse pseudo target domain three-dimensional image, acquiring a dense pseudo target domain three-dimensional image; carrying out target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to acquire a target organ structure image; and on the basis of the target organ structure image, carrying out three-dimensional reconstruction to acquire a target organ three-dimensional model. The present invention can acquire a finer and clearer target organ three-dimensional model, such that a doctor can accurately analyze tiny lesions and abnormal areas.

Description

Three-dimensional ultrasound modeling method, system, electronic device and readable storage medium Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional ultrasound modeling method, system, electronic equipment and readable storage medium. Background Art

[0002] Atrial fibrillation (AF) is one of the most common clinically diagnosed cardiac arrhythmias. Driven by an aging population, chronic heart disease, and other factors, the incidence of AF has increased dramatically worldwide, with my country being the hardest hit. Currently, the only effective treatment for AF is catheter ablation. This involves three-dimensional reconstruction of the cardiac structure using an interventional catheter. Under the guidance of a three-dimensional mapping system, energy is applied to the lesion, thereby blocking the pathway of abnormal electrical signals or destroying abnormal tissue to eliminate the abnormal signal, achieving the desired treatment.

[0003] By reconstructing the cardiac electroanatomical map through a three-dimensional mapping system and leveraging its navigation capabilities to accurately select ablation targets, the success rate of atrial fibrillation treatment can be significantly improved. Based on high-precision magnetic positioning technology, the accuracy of positioning mapping can be ensured. A positioning catheter can be used to quickly construct a real-time three-dimensional cardiac mapping map. As the number of mapping points increases, it is continuously corrected to improve accuracy. The three-dimensional image is reconstructed and refreshed in real time as the catheter moves, better guiding the surgeon to complete the radiofrequency ablation procedure. However, for patients with complex cardiac structures or special anatomical characteristics, the information provided by traditional three-dimensional mapping reconstruction methods is not accurate and complete enough.

[0004] Ultrasound imaging technology is currently widely used in the medical field for clinical diagnosis and treatment. Ultrasound imaging, which transmits and receives ultrasound waves to capture images of human tissue, is non-invasive, real-time, and radiation-free, making it a crucial imaging tool. Intracardiac 3D ultrasound technology provides a powerful guide for accurately understanding cardiac and vascular structures and precisely positioning surgical instruments during interventional procedures.

[0005] However, conventional ultrasound imaging has some limitations, such as:

[0006] (1) Resolution limitation: The resolution of traditional ultrasound imaging is limited by the frequency of sound waves and the performance of the sensor, resulting in the inability to clearly display tiny structures or lesions in some scenarios.

[0007] (2) Image quality: Ultrasound imaging is affected by physical factors such as tissue attenuation and scattering, which may cause noise, artifacts, or image blur, reducing diagnostic accuracy.

[0008] (3) Lack of comprehensive cardiac information: Traditional ultrasound imaging often only provides information on the surface cardiac structure and has certain limitations in the evaluation of deep structures, hemodynamics, and cardiac function, which affects the accurate diagnosis and treatment decision-making.

[0009] (4) Human factors: Due to differences in individual experience, individuals may make different judgments based on the same ultrasound image, resulting in some differences that cannot be unified.

[0010] It should be noted that the information disclosed in the background technology section of the invention is only intended to deepen the understanding of the general background technology of the invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

[0011] Summary of the Invention

[0012] The purpose of the present invention is to provide a three-dimensional ultrasound modeling method, system, electronic device and readable storage medium, which can obtain a more detailed and clear three-dimensional model of the target organ, helping doctors to accurately analyze small lesions and abnormal areas.

[0013] To achieve the above object, the present invention provides a three-dimensional ultrasound modeling method, comprising:

[0014] Performing domain conversion on each frame of an acquired target organ ultrasound image sequence using a first generative model to obtain a corresponding pseudo target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images acquired by an ultrasound probe at different positions and depths within a cavity of the target organ;

[0015] Acquire a sparse pseudo target domain three-dimensional image according to the pseudo target domain image sequence, and acquire a dense pseudo target domain three-dimensional image according to the sparse pseudo target domain three-dimensional image;

[0016] performing target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image;

[0017] Three-dimensional reconstruction is performed based on the target organ structure image to obtain a three-dimensional model of the target organ.

[0018] Optionally, the first generation model is obtained by the following steps:

[0019] Acquire multiple first training samples, wherein each of the first training samples includes a sample ultrasound image, a sample target domain image, and a corresponding target organ label image, wherein the target organ label image is obtained by segmenting and labeling the sample target domain image;

[0020] The first training sample is used to train a pre-built neural network model, wherein the neural network model includes a first generating subnetwork, a first discriminating subnetwork, a first segmenting subnetwork, a second generating subnetwork, a second discriminating subnetwork, and a second segmenting subnetwork, wherein the first generating subnetwork is used to convert the image it receives from the ultrasound domain to the target domain, the second generating subnetwork is used to convert the image it receives from the target domain to the ultrasound domain, the first discriminating subnetwork is used to judge the authenticity of the image it receives in the ultrasound domain, the second discriminating subnetwork is used to judge the authenticity of the image it receives in the target domain, the first segmenting subnetwork is used to perform anatomical structure segmentation of the target organ in the image it receives in the ultrasound domain, and the second segmenting subnetwork is used to perform anatomical structure segmentation of the target organ in the image it receives in the target domain;

[0021] The first generation subnetwork in the trained neural network model is used as the first generation model.

[0022] Optionally, the using the first training sample to train a pre-built neural network model includes:

[0023] In each iterative training, the first generation sub-network, the first segmentation sub-network, the second generation sub-network and the second segmentation sub-network are first trained using the first training sample, and then the first discriminant sub-network and the second discriminant sub-network are trained using the first training sample.

[0024] Optionally, the neural network model is trained according to the following loss function: L = λ cycle *L cycle +λ shape *L shape +λ gan *L gan L cycle =||φ2(φ1(x)-x)||1+||φ1(φ2(y)-y)||1 L gan =logψ1(x)+log(1-ψ1(φ2(y)))+logψ2(y)+log(1-ψ2(φ1(x)))

[0025] Among them, L cycle is the cycle consistency loss, L shape is the segmentation loss, L gan is the adversarial loss, λ cycle For L cycle The weight, λ shape For L shape The weight, λ gan For L gan, x is a sample ultrasound image, y is a sample target domain image, s is a target organ label image, φ1(x) represents the first pseudo target domain image generated by the first generation subnetwork performing domain conversion on the sample ultrasound image, φ2(y) represents the first pseudo ultrasound image generated by the second generation subnetwork performing domain conversion on the sample target domain image, φ2(φ1(x)) represents the second pseudo ultrasound image generated by the second generation subnetwork performing domain conversion on the first pseudo target domain image, φ1(φ2(y)) represents the second pseudo target domain image generated by the first generation subnetwork performing domain conversion on the first pseudo ultrasound image, represents the segmentation result of the second segmentation sub-network performing target organ anatomical structure segmentation on the sample target domain image, represents the segmentation result of the first segmentation sub-network performing target organ anatomical structure segmentation on the first pseudo ultrasound image, represents the segmentation result of the first segmentation sub-network for segmenting the target organ anatomical structure of the sample ultrasound image, represents the segmentation result of the second segmentation sub-network for the target organ anatomical structure segmentation of the first pseudo target domain image, ψ1(x) represents the true or false judgment result of the first discriminant sub-network for the sample ultrasound image, ψ1(φ2(y)) represents the true or false judgment result of the first discriminant sub-network for the first pseudo ultrasound image, ψ2(y) represents the true or false judgment result of the second discriminant sub-network for the sample target domain image, and ψ2(φ1(x)) represents the true or false judgment result of the second discriminant sub-network for the first pseudo target domain image.

[0026] Optionally, acquiring a sparse pseudo target domain three-dimensional image according to the pseudo target domain image sequence includes:

[0027] For each frame of the pseudo target domain image in the pseudo target domain image sequence, based on the position information and depth information of the ultrasound probe corresponding to the frame of the pseudo target domain image, the frame of the pseudo target domain image is converted into a three-dimensional space coordinate system;

[0028] An interpolation process is performed on the target domain image sequence converted into the three-dimensional space coordinate system to obtain a sparse pseudo target domain three-dimensional image.

[0029] Optionally, the acquiring of a dense pseudo target domain three-dimensional image according to the sparse pseudo target domain three-dimensional image includes:

[0030] A completion model is used to complete the image content of the sparse pseudo target domain three-dimensional image to obtain the dense pseudo target domain three-dimensional image, wherein the completion model is trained using the second training sample.

[0031] Optionally, the second training sample is obtained by the following steps:

[0032] Acquire multiple real target domain 3D medical images and multiple 3D white noise images;

[0033] For each of the three-dimensional medical images in the real target domain, performing noise processing on the three-dimensional medical image in the real target domain to obtain a corresponding noise sample image;

[0034] For each of the real target domain three-dimensional medical images, the real target domain three-dimensional medical image is used as a label and the corresponding noise sample image is used to form a third training sample;

[0035] Training the pre-created second generation model according to the third training sample to obtain a trained second generation model;

[0036] Using the second generative model to perform noise reduction processing on each of the three-dimensional white noise images to obtain a corresponding expanded target domain three-dimensional medical image;

[0037] The real target domain three-dimensional medical image and the expanded target domain three-dimensional medical image are both used as sample target domain three-dimensional medical images;

[0038] For each of the sample target domain three-dimensional medical images, downsampling the sample target domain three-dimensional medical image to obtain a corresponding sample sparse target domain three-dimensional medical image;

[0039] For each of the sample target domain three-dimensional medical images, the sample target domain three-dimensional medical image is used as a label to form a second training sample with its corresponding sample sparse target domain three-dimensional medical image.

[0040] Optionally, performing three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ includes:

[0041] A surface reconstruction algorithm is used to perform three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ.

[0042] Optionally, performing target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image includes:

[0043] A pre-trained segmentation model is used to segment the target organ anatomical structure of the dense pseudo target domain three-dimensional image to obtain the target organ structure image.

[0044] To achieve the above object, the present invention further provides a three-dimensional ultrasound modeling system, comprising:

[0045] a generation module configured to perform domain conversion on each frame of an acquired target organ ultrasound image sequence using a first generation model to acquire a corresponding pseudo target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images acquired by an ultrasound probe at different positions and depths within a cavity of the target organ;

[0046] a restoration module configured to acquire a sparse pseudo target domain 3D image according to the pseudo target domain image sequence, and acquire a dense pseudo target domain 3D image according to the sparse pseudo target domain 3D image;

[0047] a segmentation module configured to perform target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image; and

[0048] A modeling module is configured to perform three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ.

[0049] Optionally, the restoration module is configured to obtain a sparse pseudo target domain three-dimensional image through the following steps:

[0050] For each frame of the pseudo target domain image in the pseudo target domain image sequence, based on the position information and depth information of the ultrasound probe corresponding to the frame of the pseudo target domain image, the frame of the pseudo target domain image is converted into a three-dimensional space coordinate system;

[0051] An interpolation process is performed on the target domain image sequence converted into the three-dimensional space coordinate system to obtain a sparse pseudo target domain three-dimensional image.

[0052] Optionally, the restoration module is configured to obtain a dense pseudo target domain three-dimensional image through the following steps:

[0053] Obtaining a dense pseudo target domain 3D image according to a sparse pseudo target domain 3D image, including:

[0054] A completion model is used to complete the image content of the sparse pseudo target domain three-dimensional image to obtain the dense pseudo target domain three-dimensional image, wherein the completion model is trained using the second training sample.

[0055] Optionally, the segmentation module is configured to use a pre-trained segmentation model to perform target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain the target organ structure image.

[0056] Optionally, the modeling module is configured to perform three-dimensional reconstruction based on the target organ structure image using a surface reconstruction algorithm to obtain a three-dimensional model of the target organ.

[0057] To achieve the above object, the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the three-dimensional ultrasound modeling method described above is implemented.

[0058] To achieve the above-mentioned object, the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional ultrasound modeling method described above is implemented.

[0059] Compared with the prior art, the three-dimensional ultrasound modeling method, system, electronic device, and readable storage medium provided by the present invention have the following beneficial effects:

[0060] The three-dimensional ultrasound modeling method provided by the present invention first uses a first generation model to perform domain conversion on each frame of the acquired target organ ultrasound image sequence to obtain a corresponding pseudo-target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images collected by the ultrasound probe in different postures and depths in the cavity of the target organ; then a sparse pseudo-target domain three-dimensional image is obtained based on the pseudo-target domain image sequence, and a dense pseudo-target domain three-dimensional image is obtained based on the sparse pseudo-target domain three-dimensional image; then the dense pseudo-target domain three-dimensional image is segmented to obtain the target organ anatomical structure image to obtain the target organ structure image; finally, a three-dimensional reconstruction is performed based on the target organ structure image to obtain the target organ three-dimensional model. Therefore, the present invention combines the high spatial resolution characteristics of three-dimensional medical images in target domains such as CT and MRI to process the ultrasound image sequence into a dense target domain (such as CT domain) three-dimensional image, which can provide a more detailed and clear target organ structure (such as heart structure) image for the subsequent reconstruction of the target organ three-dimensional model (such as the heart three-dimensional model), which helps doctors accurately analyze small lesions and abnormal areas, reduce the impact of image quality on diagnostic efficiency, and improve diagnosis and treatment efficiency. In addition, the present invention can obtain the intracavitary conditions of the target organ (such as the heart) in real time by using three-dimensional ultrasound technology, and because ultrasound technology is non-radioactive, it can be used throughout the operation, which can effectively improve the efficiency of the operation, reduce surgical complications, and greatly improve the safety of the operation. In addition, since the target organ ultrasound image sequence in the present invention is obtained by using an ultrasound probe to collect multiple frames of ultrasound images in different positions and depths in the cavity of the target organ, the proximal structure of the target organ (such as the heart) close to the ultrasound probe (i.e., the structure at a shallow depth) can be refined to improve the accuracy of the three-dimensional modeling. At the same time, by obtaining relevant information about the distal structure of the target organ (such as the heart) away from the ultrasound probe (i.e., the structure at a deeper depth), it can play an auxiliary role to improve the overall accuracy of the three-dimensional modeling. In addition, based on ultrasound image processing and three-dimensional reconstruction, the present invention can quickly complete the reconstruction of the anatomical structure of the target organ during the operation, especially the reconstruction of the complex cardiac cavity structure. By displaying the reconstruction results in the three-dimensional mapping system, the entire surgical process can be clearly visualized, which helps to assist doctors to perform atrial septal puncture more safely, complete ablation, and effectively monitor adverse events such as thrombosis and pericardial tamponade.

[0061] Since the three-dimensional ultrasound modeling system, electronic device and readable storage medium provided by the present invention belong to the same inventive concept as the three-dimensional ultrasound modeling method provided by the present invention, the three-dimensional ultrasound modeling system, electronic device and readable storage medium provided by the present invention have at least all the beneficial effects of the three-dimensional ultrasound modeling method provided by the present invention. For details, please refer to the relevant description of the beneficial effects of the three-dimensional ultrasound modeling method provided by the present invention in the above text. Therefore, the beneficial effects of the three-dimensional ultrasound modeling system, electronic device and readable storage medium provided by the present invention will not be described one by one here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] FIG1 is a flow chart of a three-dimensional ultrasound modeling method provided by one embodiment of the present invention;

[0063] FIG2 is a schematic diagram of the structure of a neural network model provided by one embodiment of the present invention;

[0064] FIG3 is a schematic diagram of image content completion using a completion model according to an embodiment of the present invention;

[0065] FIG4 is a schematic diagram of segmenting the anatomical structure of a target organ using a segmentation model according to one embodiment of the present invention;

[0066] FIG5 is a block diagram of a three-dimensional ultrasound modeling system provided in one embodiment of the present invention;

[0067] FIG6 is a block diagram of an electronic device according to an embodiment of the present invention.

[0068] The accompanying drawings are numerals as follows:

[0069] Generation module-110; Repair module-120; Segmentation module-130; Modeling module-140;

[0070] Processor 210 ; communication interface 220 ; memory 230 ; communication bus 240 . DETAILED DESCRIPTION

[0071] The following is a further detailed description of the three-dimensional ultrasound modeling method, system, electronic device and readable storage medium proposed in the present invention in conjunction with the accompanying drawings and specific embodiments. According to the following description, the advantages and features of the present invention will become clearer. It should be noted that the drawings are in a very simplified form and are not in precise proportions, which are only used to conveniently and clearly assist in explaining the purpose provided by the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed in the present invention when the effect produced by the present invention and the purpose achieved are the same or similar.

[0072] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element. The singular forms "a", "an" and "the" include plural objects, the term "or" is generally used in a sense including "and / or", the term "several" is generally used in a sense including "at least one", and the term "at least two" is generally used in a sense including "two or more". In addition, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance or implicitly indicating the number of the indicated technical features.

[0073] In addition, in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0074] The core idea of ​​the present invention is to provide a three-dimensional ultrasound modeling method, system, electronic device and readable storage medium, which can obtain a more detailed and clear three-dimensional model of the target organ, helping doctors to accurately analyze tiny lesions and abnormal areas.

[0075] It should be noted that the 3D ultrasound modeling method provided by the present invention can be applied to the 3D ultrasound modeling system provided by the present invention, and the 3D ultrasound modeling system provided by the present invention can be configured on an electronic device, wherein the electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a hardware device such as a mobile phone or a tablet computer with various operating systems. It should also be noted that, as those skilled in the art will appreciate, the 3D ultrasound modeling method and 3D ultrasound modeling system provided by the present invention can be used not only to construct a 3D heart model in real time during surgery, but also to construct 3D models of other organs in real time during surgery.

[0076] To achieve the above-mentioned concept, the present invention provides a three-dimensional ultrasound modeling method. Please refer to Figure 1, which schematically shows a flow chart of the three-dimensional ultrasound modeling method provided by one embodiment of the present invention. As shown in Figure 1, the three-dimensional ultrasound modeling method provided by the present invention includes the following steps:

[0077] Step S100: Using a first generative model to perform domain conversion on each frame of an acquired target organ ultrasound image sequence to obtain a corresponding pseudo target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images acquired by an ultrasound probe in different postures and at different depths in the cavity of the target organ.

[0078] Step S200: acquiring a sparse pseudo target domain 3D image according to the pseudo target domain image sequence, and acquiring a dense pseudo target domain 3D image according to the sparse pseudo target domain 3D image.

[0079] Step S300: performing target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image.

[0080] Step S400: performing three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ.

[0081] Therefore, the present invention combines the high spatial resolution characteristics of three-dimensional medical imaging in target domains such as CT and MRI to process ultrasound image sequences into dense target domain (e.g., CT domain) three-dimensional images. This can provide more detailed and clear images of the target organ structure (e.g., heart structure) for the subsequent reconstruction of a three-dimensional model of the target organ (e.g., a three-dimensional model of the heart), helping doctors accurately analyze small lesions and abnormal areas, reducing the impact of image quality on diagnostic efficiency, and improving diagnostic and treatment efficiency. In addition, the present invention uses three-dimensional ultrasound technology to obtain real-time intracavitary conditions of the target organ (e.g., the heart), and because ultrasound technology is non-radioactive, it can be used throughout the entire surgical process, effectively improving surgical efficiency and reducing surgical complications, greatly improving surgical safety. In addition, since the target organ ultrasound image sequence in the present invention is obtained by using an ultrasound probe to collect multiple frames of ultrasound images in different postures and depths in the cavity of the target organ, the proximal structure of the target organ (such as the heart) close to the ultrasound probe (i.e., the structure at a shallower depth) can be refined to improve the accuracy of the three-dimensional modeling. At the same time, by obtaining relevant information about the distal structure of the target organ (such as the heart) away from the ultrasound probe (i.e., the structure at a deeper depth), it can play an auxiliary role to improve the overall accuracy of the three-dimensional modeling. In addition, based on ultrasound image processing and three-dimensional reconstruction, the present invention can quickly complete the reconstruction of the anatomical structure of the target organ during surgery, especially the reconstruction of complex cardiac cavity structures. By displaying the reconstruction results in a three-dimensional mapping system, the entire surgical process can be clearly visualized, which helps assist doctors to perform atrial septal puncture more safely, complete ablation, and effectively monitor adverse events such as thrombosis and pericardial tamponade.

[0082] Specifically, the ultrasound probe is mounted on the distal end of an invasive medical catheter, which is also provided with a positioning sensor for measuring the distal end position of the invasive medical catheter (i.e., the position of the ultrasound probe). The invasive medical catheter may be, but is not limited to, an electrophysiological catheter, a catheter for ablation, or mapping. The positioning sensor may be, but is not limited to, a magnetic positioning sensor, an electric field positioning sensor, or an electromagnetic dual positioning sensor. It should be noted that, as those skilled in the art will appreciate, the target organ may be, but is not limited to, the heart, and the target organ anatomical structure may be, but is not limited to, the left atrium, the left inferior pulmonary vein, the left superior pulmonary vein, the right inferior pulmonary vein, the right superior pulmonary vein, etc. It should also be noted that, as those skilled in the art will appreciate, images of different imaging modalities or images of different imaging sequences in the same imaging modality in medical images can be considered as images of different domains. The target domain in the present invention may be, but is not limited to, the CT domain or the MRI domain. In addition, it should be noted that, as those skilled in the art will appreciate, ultrasound images of different depths correspond to different spatial resolutions. The deeper the detection depth of the ultrasound probe, the lower the spatial resolution of the acquired ultrasound image.

[0083] In some exemplary embodiments, the first generative model is obtained by the following steps:

[0084] Acquire multiple first training samples, wherein each of the first training samples includes a sample ultrasound image, a sample target domain image, and a corresponding target organ label image, wherein the target organ label image is obtained by segmenting and labeling the sample target domain image;

[0085] The first training sample is used to train a pre-built neural network model, wherein the neural network model includes a first generating subnetwork, a first discriminating subnetwork, a first segmenting subnetwork, a second generating subnetwork, a second discriminating subnetwork, and a second segmenting subnetwork, wherein the first generating subnetwork is used to convert the image it receives from the ultrasound domain to the target domain, the second generating subnetwork is used to convert the image it receives from the target domain to the ultrasound domain, the first discriminating subnetwork is used to judge the authenticity of the image it receives in the ultrasound domain, the second discriminating subnetwork is used to judge the authenticity of the image it receives in the target domain, the first segmenting subnetwork is used to perform anatomical structure segmentation of the target organ in the image it receives in the ultrasound domain, and the second segmenting subnetwork is used to perform anatomical structure segmentation of the target organ in the image it receives in the target domain;

[0086] The first generation subnetwork in the trained neural network model is used as the first generation model.

[0087] Specifically, please refer to Figure 2, which is a schematic diagram of the structure of the neural network model provided by one embodiment of the present invention. As shown in Figure 2, taking the target domain as the CT domain as an example, the role of the first generation subnetwork φ1 is to convert an ultrasound image x into a pseudo image y' that has the same content but looks similar to CT. A pseudo image refers to an image that does not exist in reality and is generated by technical means with a high degree of authenticity. Similarly, the role of the second generation subnetwork φ2 is to convert a CT image y into a pseudo image x' that has the same content but looks similar to ultrasound. The first segmentation subnetwork Responsible for extracting the left atrium and blood vessels and other cavity structures in the ultrasound image, the second segmentation sub-network Responsible for extracting the left atrium and vascular cavity structure in the CT domain image. It should be noted that the first segmentation sub-network and the second segmentation sub-network It is only responsible for segmenting the cavity structure and does not distinguish between each cavity structure, that is, all cavity structures have the same label. This segmentation method that does not require fine classification can reduce the segmentation difficulty and make the training of the neural network model easier. and the second segmentation sub-network The existence of can play a role in improving the quality of the images generated by the first generation sub-network φ1 and the second generation sub-network φ2. Therefore, the present invention can maintain the consistency of the anatomical structure while generating images by combining the image conversion task and the segmentation task, which is more conducive to outputting realistic images. The first discriminant sub-network ψ1 is used to judge whether the image it receives is a true ultrasound image, and outputs a probability value between 0 and 1. The greater the probability, the greater the possibility of judging it as a true ultrasound image; the second discriminant sub-network ψ2 is used to judge whether the image it receives is a true CT image, and outputs a probability value between 0 and 1. The greater the probability, the greater the possibility of judging it as a true CT image.

[0088] Furthermore, during actual surgery, ultrasound probes can be adjusted to different depths, so the acquired ultrasound images are also images of different poses and depths (different spatial resolutions). Therefore, when preparing CT images for training, we can use this characteristic of ultrasound to resample the CT images to different resolutions to enhance the robustness of the network.

[0089] It should be noted that, as those skilled in the art can understand, the present invention has the following characteristics: The second generation sub-network φ2, the second discrimination sub-network ψ2 and the second segmentation sub-network The specific network structure is not limited, the first generation sub-network φ1, the first segmentation sub-network The second generation sub-network φ2 and the second segmentation sub-network The first discrimination sub-network and the second discrimination sub-network may be, but are not limited to, U-Net. The first discrimination sub-network and the second discrimination sub-network may be, but are not limited to, PatchGan. The specific network structures and principles of U-Net and PatchGan can be referred to relevant technologies known to those skilled in the art and will not be elaborated here.

[0090] In some exemplary embodiments, the training of a pre-built neural network model using the first training sample includes:

[0091] In each iterative training, the weight parameters of the first discriminant subnetwork, the second discriminant subnetwork, the first segmentation subnetwork and the second segmentation subnetwork are first fixed, and the first training sample is used to train the first generation subnetwork and the second generation subnetwork to update the weight parameters of the first generation subnetwork and the second generation subnetwork; then the weight parameters of the first generation subnetwork and the second generation subnetwork are fixed, and the first training sample is used to train the first discriminant subnetwork, the second discriminant subnetwork, the first segmentation subnetwork and the second segmentation subnetwork to update the weight parameters of the first discriminant subnetwork, the second discriminant subnetwork, the first segmentation subnetwork and the second segmentation subnetwork.

[0092] Specifically, the training process of the neural network model involves generative adversarial training. Therefore, in each iteration, the generative subnetworks (i.e., the first generative subnetwork φ1 and the second generative subnetwork φ2) are trained first, and the first discriminative subnetwork ψ1, the second discriminative subnetwork ψ2, and the first segmentation subnetwork are fixed. and the second segmentation sub-network The weight parameters of the first generation sub-network φ1 and the second generation sub-network φ2 related to generation are updated. Specifically, the pseudo target domain image (e.g., pseudo CT image) generated by the first generation sub-network φ1 is input into the second discriminative sub-network ψ2, and the pseudo ultrasound image generated by the second generation sub-network ψ2 is input into the first discriminative sub-network ψ1. The adversarial loss is calculated based on the error between the output of the second discriminative sub-network ψ2 and 1 (judged as true) and the error between the output of the first discriminative sub-network ψ1 and 1 (judged as true). The gradient is further calculated by combining the cycle consistency loss and segmentation loss described below to update the weight parameters of the generation-related sub-networks (i.e., the first generation sub-network φ1 and the second generation sub-network φ2) through backpropagation. This step allows the first generation sub-network φ1 to generate more realistic pseudo target domain images (e.g., pseudo CT images) and the second generation sub-network φ2 to generate more realistic pseudo ultrasound images. The first discriminative sub-network ψ1, the second discriminative sub-network ψ2, and the first segmentation sub-network are then trained. and the second segmentation sub-network The weight parameters of the generation-related subnetworks (i.e., the first generation subnetwork φ1 and the second generation subnetwork φ2) are fixed, and only the first discriminant subnetwork ψ1, the second discriminant subnetwork ψ2, and the first segmentation subnetwork are updated. and the second segmentation sub-network The weight parameters of the first discriminant subnetwork ψ1 are input into the sample ultrasound image, and the sample target domain image is input into the second discriminant subnetwork ψ2. The error between the output of the first discriminant subnetwork ψ1 and 1 (judged to be true) and the error between the output of the second discriminant subnetwork ψ2 and 1 (judged to be true) are calculated. The pseudo target domain image (for example, a pseudo CT image) generated by the first generation subnetwork φ1 is input into the second discriminant subnetwork ψ2, and the pseudo ultrasound image generated by the second generation subnetwork φ2 is input into the first discriminant subnetwork ψ1. The error between the output of the first discriminant subnetwork ψ1 and 0 (judged to be false) and the error between the output of the second discriminant subnetwork ψ2 and 0 (judged to be false) are calculated. The gradient is calculated based on the sum of these errors to update the weight parameters of the first discriminant subnetwork ψ1 and the second discriminant subnetwork ψ2 through back propagation. That is, this step improves the ability of the first discriminant subnetwork ψ1 to recognize pseudo ultrasound images and the ability of the second discriminant subnetwork ψ2 to recognize pseudo target domain images (for example, pseudo CT images). By alternately repeating the above two steps, the capabilities of the generation part (the first generation sub-network φ1 and the second generation sub-network φ2) and the discrimination part (the first discrimination sub-network ψ1 and the second discrimination sub-network ψ2) of the neural network model can be made stronger and stronger, so that the first generation sub-network φ1 and the second generation sub-network φ2 can output realistic images. When the trained neural network model is applied, only the first generation sub-network φ1 is needed, and a set of ultrasound images (i.e., ultrasound image sequence) {x1, x2, ..., x2} of a target organ scan (e.g., a heart scan) is trained. n By inputting the first generative subnetwork φ1 (i.e., the first generative model) one by one, a set of pseudo target domain (e.g., pseudo CT) images {y′1, y′2, ..., y′ n}.

[0093] In some exemplary embodiments, the neural network model is trained according to the following loss function: L = λ cycle *L cycle +λ shape *L shape +λ gan *L gan L cycle =||φ2(φ1(x)-x)||1+||φ1(φ2(y)-y)||1 L gan =logψ1(x)+log(1-ψ1(φ2(y)))+logψ2(y)+log(1-ψ2(φ1(x)))

[0094] Among them, L cycle is the cycle consistency loss, L shape is the segmentation loss, Lgan is the adversarial loss, λ cycle For L cycle The weight, λ shape For L shape The weight, λ gan For L gan , x is a sample ultrasound image, y is a sample target domain image, s is a target organ label image, φ1(x) represents the first pseudo target domain image generated by the first generation subnetwork performing domain conversion on the sample ultrasound image, φ2(y) represents the first pseudo ultrasound image generated by the second generation subnetwork performing domain conversion on the sample target domain image, φ2(φ1(x)) represents the second pseudo ultrasound image generated by the second generation subnetwork performing domain conversion on the first pseudo target domain image, φ1(φ2(y)) represents the second pseudo target domain image generated by the first generation subnetwork performing domain conversion on the first pseudo ultrasound image, represents the segmentation result of the second segmentation sub-network performing target organ anatomical structure segmentation on the sample target domain image, represents the segmentation result of the first segmentation sub-network performing target organ anatomical structure segmentation on the first pseudo ultrasound image, represents the segmentation result of the first segmentation sub-network for segmenting the target organ anatomical structure of the sample ultrasound image, represents the segmentation result of the second segmentation sub-network for the target organ anatomical structure segmentation of the first pseudo target domain image, ψ1(x) represents the true or false judgment result of the first discriminant sub-network for the sample ultrasound image, ψ1(φ2(y)) represents the true or false judgment result of the first discriminant sub-network for the first pseudo ultrasound image, ψ2(y) represents the true or false judgment result of the second discriminant sub-network for the sample target domain image, and ψ2(φ1(x)) represents the true or false judgment result of the second discriminant sub-network for the first pseudo target domain image.

[0095] Specifically, in order to ensure the consistency of the image content after the image conversion process, L1 loss (also known as the minimum absolute deviation) is applied between the two converted images φ2(φ1(x)) and φ1(φ2(y)) and the original images x and y as the cycle consistency loss L cycle In order to train different segmenters, Dice loss is applied to the segmentation results of images from the target domain (such as CT domain) and the target organ label image s. At the same time, due to the consistent invariance of anatomical structures in image reconstruction and image conversion, L1 loss is used to minimize the distance between related segmentation results, thereby obtaining the segmentation loss L shape Furthermore, by applying the adversarial loss L gan To constrain the cross-domain image transformation. cycle , Lshape and L gan The total loss function L of these three losses is used to train the neural network model, which can effectively ensure that the generation part (the first generation subnetwork φ1 and the second generation subnetwork φ2) and the discriminative part (the first discriminative subnetwork ψ1 and the second discriminative subnetwork ψ2) of the neural network model are increasingly powerful, so that the first generation subnetwork φ1 and the second generation subnetwork φ2 can output realistic images. Furthermore, during the training process, the Adam optimizer can be used to optimize the weight parameters. It should be noted that, as can be understood by those skilled in the art, λ cycle ,λ shape and λ gan The specific value of can be set according to the actual situation, and the present invention does not limit this. It should be noted that, as those skilled in the art can understand, when training the first generating sub-network φ1 and the second generating sub-network φ2, the cycle consistency loss L cycle , segmentation loss L shape and adversarial loss L gan These three losses can be used to update the weight parameters of the first generating sub-network φ1 and the second generating sub-network φ2 (that is, the weight parameters of the first generating sub-network φ1 and the second generating sub-network φ2 can be updated based on the total loss function L). When training the first discriminant sub-network ψ1 and the second discriminant sub-network ψ2, the adversarial loss L can be used to update the weight parameters of the first generating sub-network φ1 and the second generating sub-network φ2. gan To update the weight parameters of the first discriminant sub-network ψ1 and the second discriminant sub-network ψ2. and the second segmentation sub-network When the segmentation loss L shape To update the first segmentation sub-network and the second segmentation sub-network The weight parameter of .

[0096] Furthermore, the calculation formula of Dice loss is as follows:

[0097] Here, ε is a constant to prevent the denominator from being zero.

[0098] It should be noted that, as those skilled in the art will appreciate, the value of ε may be 0.001 or other values, and may be specifically set according to actual conditions, and the present invention is not limited thereto.

[0099] In some exemplary embodiments, acquiring a sparse pseudo target domain 3D image according to the pseudo target domain image sequence includes:

[0100] For each frame of the pseudo target domain image in the pseudo target domain image sequence, based on the position information and depth information of the ultrasound probe corresponding to the frame of the pseudo target domain image, the frame of the pseudo target domain image is converted into a three-dimensional space coordinate system;

[0101] An interpolation process is performed on the target domain image sequence converted into the three-dimensional space coordinate system to obtain a sparse pseudo target domain three-dimensional image.

[0102] Therefore, by tracking the position and posture of the ultrasound probe in real time and accurately fusing it with the ultrasound image, the present invention can achieve more accurate three-dimensional modeling of the target organ (such as the heart), facilitating doctors to observe and analyze the target organ (such as the heart) in real time. By using the ultrasound probe's position information to combine two-dimensional image data into three-dimensional space, the spatial consistency of two-dimensional data at different angles can be exploited to more easily restore image content information.

[0103] Specifically, for each pseudo target domain image frame in the pseudo target domain image sequence, the spatial resolution information of the pseudo target domain image frame can be obtained based on the depth information of the ultrasound probe corresponding to the pseudo target domain image frame when the corresponding ultrasound image is acquired. Based on the spatial resolution information, the pseudo target domain image frame can be transformed from the pixel image coordinate system to the physical image coordinate system. Furthermore, based on the pose information (pose information in the three-dimensional spatial coordinate system) of the ultrasound probe corresponding to the pseudo target domain image frame when the corresponding ultrasound image is acquired, the pseudo target domain image frame can be transformed from the physical image coordinate system to the three-dimensional spatial coordinate system. Since the coordinates of each pixel point in the transformed pseudo target domain image sequence do not necessarily fall exactly on integer coordinates, these points that do not fall on integer coordinates are sub-pixel points. Therefore, it is necessary to interpolate these sub-pixel points in the target domain image sequence obtained after the transformation, and estimate the pixel values ​​(e.g., CT values ​​or HU values) of adjacent pixels based on the pixel values ​​(e.g., CT values ​​or HU values) of these sub-pixel points and spatial information. Specifically, an interpolation method in the prior art may be used for interpolation processing, such as a nearest neighbor interpolation method, a trilinear interpolation method, a PV interpolation method (partial volume interpolation method), and the like.

[0104] In some exemplary embodiments, acquiring a dense pseudo target domain 3D image based on the sparse pseudo target domain 3D image includes:

[0105] A completion model is used to complete the image content of the sparse pseudo target domain three-dimensional image to obtain the dense pseudo target domain three-dimensional image, wherein the completion model is trained using the second training sample.

[0106] Specifically, the network structure of the completion model can be, but is not limited to, a three-dimensional U-Net. Please refer to Figure 3, which is a schematic diagram of an embodiment of the present invention providing a completion model for image content completion. As shown in Figure 3, by using the completion model to complete the image content of the sparse pseudo-target domain three-dimensional image, the sparse information can be completed using the prior knowledge of the target domain (such as CT) that is easier to collect, and then a more refined and clear target organ structure (such as heart structure) image can be provided for the subsequent reconstruction of the target organ three-dimensional model (such as a heart three-dimensional model), which helps doctors to accurately analyze small lesions and abnormal areas. It should be noted that, as can be understood by those skilled in the art, the sparse pseudo-target domain three-dimensional image can be regarded as being composed of multiple layers of two-dimensional images superimposed. Figure 3 is an example of using the completion model to complete the image content of a certain layer of two-dimensional image in the sparse pseudo-target domain three-dimensional image.

[0107] In some exemplary embodiments, the second training sample is obtained by the following steps:

[0108] Acquire multiple real target domain 3D medical images and multiple 3D white noise images;

[0109] For each of the three-dimensional medical images in the real target domain, performing noise processing on the three-dimensional medical image in the real target domain to obtain a corresponding noise sample image;

[0110] For each of the real target domain three-dimensional medical images, the real target domain three-dimensional medical image is used as a label and the corresponding noise sample image is used to form a third training sample;

[0111] Training the pre-created second generation model according to the third training sample to obtain a trained second generation model;

[0112] Using the second generative model to perform noise reduction processing on each of the three-dimensional white noise images to obtain a corresponding expanded target domain three-dimensional medical image;

[0113] The real target domain three-dimensional medical image and the expanded target domain three-dimensional medical image are both used as sample target domain three-dimensional medical images;

[0114] For each of the sample target domain three-dimensional medical images, downsampling the sample target domain three-dimensional medical image to obtain a corresponding sample sparse target domain three-dimensional medical image;

[0115] For each of the sample target domain three-dimensional medical images, the sample target domain three-dimensional medical image is used as a label to form a second training sample with its corresponding sample sparse target domain three-dimensional medical image.

[0116] Therefore, by adopting the second generative model to expand the training data set required for training the completion model, the diversity of the training data set can be expanded, thereby improving the robustness of the algorithm and avoiding the problem of low robustness caused by the small data set, so that the trained completion model can better complete the image content of the sparse pseudo-target domain three-dimensional image. It should be noted that, as those skilled in the art can understand, the relevant content on how to add noise processing to the real target domain three-dimensional medical image can refer to the relevant technology known to those skilled in the art, and will not be repeated here. It should also be noted that, as those skilled in the art can understand, the sample target domain three-dimensional medical image can be downsampled by simulating the ultrasound data acquisition process to obtain a sample sparse target domain three-dimensional medical image.

[0117] Specifically, the second generation model may include but is not limited to a generation subnetwork in a generative adversarial network, a VAE (Variational Autoencoder), a diffusion model, etc.

[0118] Furthermore, let's take the diffusion model as an example. The diffusion model is an unconditional generative method that can transform a Gaussian distribution into a distribution of empirical data. In particular, the forward diffusion process is a Markov chain, which is a process of gradually adding noise to a clean image. It can be expressed as:

[0119] here It is represented as the intermediate image with noise added at step t, represents the image at step 0, which is the real target domain image (such as a real CT image), β t represents the noise level, α t =1-β t , N is the normal distribution, I is the identity matrix. Generally, for simplicity, β t Follows a fixed decay mode. When the total number of iterations T is large enough, Only noise can be observed in . Using the CNN model to gradually reverse this forward diffusion process, that is, Perform noise reduction and estimate the reverse diffusion process by the following formula:

[0120] in, ∈ θThe diffusion noise can be trained by L2 loss function (i.e. L2 norm loss function, also known as least square error) (i.e. the second generation model is trained, ∈ θ is the weight parameter of the trained second generative model. Thus, realistic target domain data (such as CT data) can be obtained from the randomly sampled white noise through the reverse diffusion process, thereby expanding and completing the sample set required for model training.

[0121] Furthermore, the training process of the completion model can be optimized using the L2 loss function in combination with the Adam optimizer. It should be noted that more information about how to use the second training sample to train the completion model can be referred to the relevant technology known to those skilled in the art, and will not be repeated here.

[0122] In some exemplary embodiments, performing three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ includes:

[0123] A surface reconstruction algorithm is used to perform three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ.

[0124] Therefore, by using a surface reconstruction algorithm to perform three-dimensional reconstruction on the target organ structure image, the target organ structure image can be converted into a visual target organ three-dimensional model (such as a three-dimensional heart model) for doctors to view, making it easier for doctors to analyze and process.

[0125] Specifically, the target organ in the target organ structure image is generally a solid object. This is because images acquired by medical imaging devices such as CT or MRI are represented in units of voxels. A voxel is a discrete point in three-dimensional space with a uniform volume. Each voxel in a CT or MRI image contains information about the tissue's absorption and density. Surface reconstruction algorithms, such as common triangulation methods and Poisson reconstruction algorithms, can be used to convert the extracted boundary and surface information into a more geometric model representation. These algorithms can represent the model's surface using forms such as point clouds or triangular meshes, achieving visualization and facilitating analysis and processing by physicians. It should be noted that, as those skilled in the art will appreciate, modeling methods other than surface reconstruction algorithms can also be used to reconstruct a three-dimensional model of the target organ. For details, reference can be made to well-known three-dimensional modeling techniques, which will not be elaborated upon here. It should also be noted that, as those skilled in the art will appreciate, further details regarding surface reconstruction algorithms can also be made to well-known related techniques, which will not be elaborated upon here.

[0126] In some exemplary embodiments, performing target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image includes:

[0127] A pre-trained segmentation model is used to segment the target organ anatomical structure of the dense pseudo target domain three-dimensional image to obtain the target organ structure image.

[0128] Specifically, the network structure of the segmentation model is a three-dimensional network structure. Furthermore, the network structure of the segmentation model can be, but is not limited to, a three-dimensional U-Net. Please refer to Figure 4, which is a schematic diagram of using a segmentation model to segment the anatomical structure of a target organ according to one embodiment of the present invention. As shown in Figure 4, the segmentation model can quickly and accurately segment the anatomical structure of the target organ, thereby shortening the doctor's learning curve and helping the doctor to perform clinical diagnosis and treatment accurately, efficiently and safely. It should be noted that, as can be understood by those skilled in the art, Figure 4 only shows a schematic diagram of a section of input and output three-dimensional data.

[0129] Furthermore, training the segmentation model requires preparing multiple target domain scan data (e.g., CT or MRI scan data) as input, with segmentation data of the target organ's anatomical structure serving as labels. During network training, Dice loss combined with the Adam optimizer is used for optimization. It should be noted that, as those skilled in the art will appreciate, the specific training process for the segmentation model can be referenced to relevant techniques known to those skilled in the art, and will not be further elaborated here.

[0130] Based on the same inventive concept, the present invention further provides a three-dimensional ultrasound modeling system, as shown in Figure 5, which is a block diagram of a three-dimensional ultrasound modeling system provided by one embodiment of the present invention. As shown in Figure 5, the three-dimensional ultrasound modeling system provided by the present invention includes a generation module 110, a repair module 120, a segmentation module 130, and a modeling module 140, wherein the generation module 110 is configured to use a first generation model to perform domain conversion on each frame of an acquired target organ ultrasound image sequence to obtain a corresponding pseudo-target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images acquired by an ultrasound probe at different positions and depths within the cavity of the target organ; the repair module 120 is configured to obtain a sparse pseudo-target domain three-dimensional image based on the pseudo-target domain image sequence, and obtain a dense pseudo-target domain three-dimensional image based on the sparse pseudo-target domain three-dimensional image; the segmentation module 130 is configured to perform target organ anatomical structure segmentation on the dense pseudo-target domain three-dimensional image to obtain a target organ structural image; and the modeling module 140 is configured to perform three-dimensional reconstruction based on the target organ structural image to obtain a target organ three-dimensional model.

[0131] Therefore, the present invention combines the high spatial resolution characteristics of three-dimensional medical imaging in target domains such as CT and MRI to process ultrasound image sequences into dense target domain (e.g., CT domain) three-dimensional images. This can provide more detailed and clear images of the target organ structure (e.g., heart structure) for the subsequent reconstruction of a three-dimensional model of the target organ (e.g., a three-dimensional model of the heart), helping doctors accurately analyze small lesions and abnormal areas, reducing the impact of image quality on diagnostic efficiency, and improving diagnostic and treatment efficiency. In addition, the present invention uses three-dimensional ultrasound technology to obtain real-time intracavitary conditions of the target organ (e.g., the heart), and because ultrasound technology is non-radioactive, it can be used throughout the entire surgical process, effectively improving surgical efficiency and reducing surgical complications, greatly improving surgical safety. In addition, since the target organ ultrasound image sequence in the present invention is obtained by using an ultrasound probe to collect multiple frames of ultrasound images in different postures and depths in the cavity of the target organ, the proximal structure of the target organ (such as the heart) close to the ultrasound probe (i.e., the structure at a shallower depth) can be refined to improve the accuracy of the three-dimensional modeling. At the same time, by obtaining relevant information about the distal structure of the target organ (such as the heart) away from the ultrasound probe (i.e., the structure at a deeper depth), it can play an auxiliary role to improve the overall accuracy of the three-dimensional modeling. In addition, based on ultrasound image processing and three-dimensional reconstruction, the present invention can quickly complete the reconstruction of the anatomical structure of the target organ during surgery, especially the reconstruction of complex cardiac cavity structures. By displaying the reconstruction results in a three-dimensional mapping system, the entire surgical process can be clearly visualized, which helps assist doctors to perform atrial septal puncture more safely, complete ablation, and effectively monitor adverse events such as thrombosis and pericardial tamponade.

[0132] In some exemplary embodiments, the restoration module 120 is configured to obtain a sparse pseudo target domain 3D image by the following steps:

[0133] For each frame of the pseudo target domain image in the pseudo target domain image sequence, based on the position information and depth information of the ultrasound probe corresponding to the frame of the pseudo target domain image, the frame of the pseudo target domain image is converted into a three-dimensional space coordinate system;

[0134] An interpolation process is performed on the target domain image sequence converted into the three-dimensional space coordinate system to obtain a sparse pseudo target domain three-dimensional image.

[0135] In some exemplary embodiments, the restoration module 120 is configured to obtain a dense pseudo-target domain 3D image by the following steps:

[0136] Obtaining a dense pseudo target domain 3D image according to a sparse pseudo target domain 3D image, including:

[0137] A completion model is used to complete the image content of the sparse pseudo target domain three-dimensional image to obtain the dense pseudo target domain three-dimensional image, wherein the completion model is trained using the second training sample.

[0138] In some exemplary embodiments, the segmentation module 130 is configured to use a pre-trained segmentation model to perform target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain the target organ structure image.

[0139] In some exemplary embodiments, the modeling module 140 is configured to perform three-dimensional reconstruction based on the target organ structure image using a surface reconstruction algorithm to obtain a three-dimensional model of the target organ.

[0140] Based on the same inventive concept, the present invention also provides an electronic device. Please refer to Figure 6, which schematically shows a block diagram of the electronic device provided by one embodiment of the present invention. As shown in Figure 6, the electronic device includes a processor 210 and a memory 230. The memory 230 stores a computer program. When the computer program is executed by the processor 210, the three-dimensional ultrasound modeling method described above is implemented. Since the electronic device provided by the present invention and the three-dimensional ultrasound modeling method provided by the present invention belong to the same inventive concept, the electronic device provided by the present invention at least has all the beneficial effects of the three-dimensional ultrasound modeling method provided by the present invention. For details, please refer to the above description of the beneficial effects of the three-dimensional ultrasound modeling method provided by the present invention. Therefore, the beneficial effects of the electronic device provided by the present invention will not be repeated here.

[0141] As shown in Figure 6, the electronic device also includes a communication interface 220 and a communication bus 240, wherein the processor 210, the communication interface 220, and the memory 230 communicate with each other via the communication bus 240. The communication bus 240 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 240 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus. The communication interface 220 is used for communication between the above-mentioned electronic device and other devices.

[0142] The processor 210 described in the present invention may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 210 is the control center of the electronic device, connecting various parts of the entire electronic device using various interfaces and lines.

[0143] The memory 230 may be used to store the computer program. The processor 210 implements various functions of the electronic device by running or executing the computer program stored in the memory 230 and calling the data stored in the memory 230.

[0144] The memory 230 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0145] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, can implement the three-dimensional ultrasound modeling method described above. Because the readable storage medium provided by the present invention and the three-dimensional ultrasound modeling method provided by the present invention are part of the same inventive concept, the readable storage medium provided by the present invention possesses at least all the beneficial effects of the three-dimensional ultrasound modeling method provided by the present invention. For details, please refer to the above description of the beneficial effects of the three-dimensional ultrasound modeling method provided by the present invention. Therefore, the beneficial effects of the readable storage medium provided by the present invention will not be elaborated upon here.

[0146] It should be noted that, as will be appreciated by those skilled in the art, the readable storage medium provided by the present invention can adopt any combination of one or more computer-readable media. The readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer hard disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this article, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0147] Furthermore, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0148] In summary, compared with the prior art, the three-dimensional ultrasound modeling method, system, electronic device, and readable storage medium provided by the present invention have the following advantages:

[0149] (1) The present invention combines the high spatial resolution characteristics of three-dimensional medical images in target domains such as CT and MRI to process ultrasound image sequences into three-dimensional images of a dense target domain (e.g., a CT domain). This can provide more detailed and clear images of the target organ structure (e.g., a heart structure) for the subsequent reconstruction of a three-dimensional model of the target organ (e.g., a three-dimensional model of the heart), helping doctors to accurately analyze subtle lesions and abnormal areas.

[0150] (2) The present invention can obtain the intracavitary conditions of the target organ (such as the heart) in real time by adopting three-dimensional ultrasound technology. Since ultrasound technology is non-radioactive, it can be used throughout the operation, which can effectively improve the efficiency of the operation, reduce surgical complications, and greatly improve the safety of the operation.

[0151] (3) Since the target organ ultrasound image sequence in the present invention is obtained by using an ultrasound probe to collect multiple frames of ultrasound images in different postures and depths in the cavity of the target organ, the proximal structure of the target organ (such as the heart) close to the ultrasound probe (i.e., the structure at a shallower depth) can be refined to improve the accuracy of three-dimensional modeling. At the same time, by obtaining relevant information of the distal structure of the target organ (such as the heart) away from the ultrasound probe (i.e., the structure at a deeper depth), it can play an auxiliary role to improve the overall accuracy of three-dimensional modeling.

[0152] (4) The present invention is based on ultrasonic image processing and three-dimensional reconstruction, which can quickly complete the reconstruction of the anatomical structure of the target organ during surgery, especially the reconstruction of the complex cardiac cavity structure. By displaying the reconstruction results in a three-dimensional mapping system, the entire surgical process can be clearly visualized, which helps assist doctors to perform atrial septal puncture more safely, complete ablation, and effectively monitor adverse events such as thrombosis and pericardial tamponade.

[0153] (5) The present invention can achieve more accurate three-dimensional modeling of the target organ (such as the heart) by tracking the position and posture of the ultrasound probe in real time and accurately fusing it with the ultrasound image, making it easier for doctors to observe and analyze the target organ (such as the heart) in real time.

[0154] (6) By using the position information of the ultrasound probe to combine the two-dimensional image data into three-dimensional space, the spatial consistency of the two-dimensional data at different angles can be used to more easily restore the image content information. At the same time, the completion model trained with the data in the target domain (such as CT data) can be used to complete the sparse information. The data in the target domain (such as CT data) that is easier to collect can be used to provide more prior knowledge, thus avoiding the problem of low robustness caused by the lack of ultrasound data.

[0155] (7) The present invention, by leveraging the powerful analytical capabilities of artificial intelligence algorithms, can process and analyze a large amount of intracardiac ultrasound data in real time and generate an accurate heart model in real time. By processing and modeling real-time data, it helps doctors more accurately assess the structure, function, and hemodynamics of the heart structure, thereby better guiding doctors in making diagnostic and treatment decisions. The present invention can provide more accurate and comprehensive guidance for the diagnosis and treatment of heart diseases, and help promote the development of imaging for cardiologists and the advancement of clinical practice.

[0156] It should be noted that, as will be appreciated by those skilled in the art, the computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] It should be noted that the devices and methods disclosed in the embodiments of this document may also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the devices, methods, and computer program products according to the various embodiments of this document. In this regard, each box in the flowchart or block diagram may represent a module, program, or portion of code, wherein the module, program segment, or portion of code contains one or more executable instructions for implementing a specified logical function, and the module, program segment, or portion of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this document may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0158] It should be noted that the above description is merely a description of preferred embodiments of the present invention and does not limit the scope of the present invention. Any changes or modifications made by a person skilled in the art based on the above disclosure are within the scope of protection of the present invention. Obviously, various changes and modifications may be made by a person skilled in the art without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the present invention and its equivalents, the present invention is intended to include such modifications and variations.

Claims

1. A three-dimensional ultrasound modeling method, characterized in that: include: A first generation model is used to perform domain conversion on each frame of an ultrasound image in an acquired target organ ultrasound image sequence to acquire a corresponding pseudo target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images acquired by an ultrasound probe in different postures and at different depths in a cavity of the target organ; Acquire a sparse pseudo target domain three-dimensional image according to the pseudo target domain image sequence, and acquire a dense pseudo target domain three-dimensional image according to the sparse pseudo target domain three-dimensional image; Performing target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image; A three-dimensional reconstruction is performed based on the target organ structure image to obtain a three-dimensional model of the target organ.

2. The three-dimensional ultrasound modeling method according to claim 1, characterized in that: The first generation model is obtained by the following steps: Acquire multiple first training samples, wherein each of the first training samples includes a sample ultrasound image, a sample target domain image, and a corresponding target organ label image, wherein the target organ label image is obtained by segmenting and labeling the sample target domain image; The first training sample is used to train a pre-built neural network model, wherein the neural network model includes a first generating subnetwork, a first discriminating subnetwork, a first segmenting subnetwork, a second generating subnetwork, a second discriminating subnetwork, and a second segmenting subnetwork, wherein the first generating subnetwork is used to convert the image received by it from the ultrasound domain to the target domain, the second generating subnetwork is used to convert the image received by it from the target domain to the ultrasound domain, the first discriminating subnetwork is used to judge the true or false situation of the image received by it in the ultrasound domain, the second judging subnetwork is used to judge the true or false situation of the image received by it in the target domain, the first segmenting subnetwork is used to perform anatomical structure segmentation of the target organ in the image received by it in the ultrasound domain, and the second segmenting subnetwork is used to perform anatomical structure segmentation of the target organ in the image received by it in the target domain; The first generation subnetwork in the trained neural network model is used as the first generation model.

3. The three-dimensional ultrasound modeling method according to claim 2, characterized in that: The using the first training sample to train the pre-built neural network model includes: In each iteration of training, the first training sample is first used to train the first generating sub-network and the first dividing sub-network. The cutting sub-network, the second generating sub-network and the second segmenting sub-network are trained, and then the first discriminant sub-network and the second discriminant sub-network are trained using the first training sample.

4. The three-dimensional ultrasound modeling method according to claim 2, characterized in that: The neural network model is trained according to the following loss function: L = λ cycle *L cycle +λ shape *L shape +λ gan *L gan L cycle =||φ2(φ1(x)-x)||1+||φ1(φ2(y)-y)||1 L gan =logψ1(x)+log(1-ψ1(φ2(y)))+logψ2(y)+log(1-ψ2(φ1(x))) Among them, L cycle is the cycle consistency loss, L shape is the segmentation loss, L gan To counter the loss, λ cycle For L cycle The weight, λ shape For L shape The weight, λ gan For L gan , x is a sample ultrasound image, y is a sample target domain image, s is a target organ label image, φ1(x) represents a first pseudo target domain image generated by the first generation subnetwork performing domain conversion on the sample ultrasound image, φ2(y) represents a first pseudo ultrasound image generated by the second generation subnetwork performing domain conversion on the sample target domain image, φ2(φ1(x)) represents a second pseudo ultrasound image generated by the second generation subnetwork performing domain conversion on the first pseudo target domain image, φ1(φ2(y)) represents a second pseudo target domain image generated by the first generation subnetwork performing domain conversion on the first pseudo ultrasound image, represents the segmentation result of the second segmentation sub-network performing segmentation of the target organ anatomical structure on the sample target domain image, represents a segmentation result of the first segmentation sub-network performing segmentation of the anatomical structure of the target organ on the first pseudo ultrasound image, represents the segmentation result of the first segmentation sub-network performing segmentation of the target organ anatomical structure on the sample ultrasound image, represents the segmentation result of the target organ anatomical structure segmentation of the first pseudo target domain image by the second segmentation sub-network, ψ1(x) represents the true or false judgment result of the sample ultrasound image by the first discriminant sub-network, ψ1(φ2(y)) represents the true or false judgment result of the first pseudo ultrasound image by the first discriminant sub-network, ψ2(y) represents the true or false judgment result of the sample target domain image by the second discriminant sub-network, ψ2(φ1(x)) represents the true or false judgment result of the first pseudo target domain image by the second discriminant sub-network.

5. The three-dimensional ultrasound modeling method according to claim 1, characterized in that: The step of acquiring a sparse pseudo target domain three-dimensional image according to the pseudo target domain image sequence comprises: For each frame of the pseudo target domain image in the pseudo target domain image sequence, based on the position information and depth information of the ultrasound probe corresponding to the frame of the pseudo target domain image, the frame of the pseudo target domain image is converted into a three-dimensional space coordinate system; Interpolation processing is performed on the target domain image sequence converted to the three-dimensional space coordinate system to obtain sparse pseudo targets Domain three-dimensional image.

6. The three-dimensional ultrasound modeling method according to claim 1, characterized in that: The step of acquiring a dense pseudo target domain three-dimensional image according to the sparse pseudo target domain three-dimensional image comprises: A completion model is used to complete the image content of the sparse pseudo target domain three-dimensional image to obtain the dense pseudo target domain three-dimensional image, wherein the completion model is trained using the second training sample.

7. The three-dimensional ultrasound modeling method according to claim 6, characterized in that: The second training sample is obtained by the following steps: Acquire multiple real target domain three-dimensional medical images and multiple three-dimensional white noise images; For each of the real target domain three-dimensional medical images, adding noise processing to the real target domain three-dimensional medical image to obtain a corresponding noise sample image; For each of the three-dimensional medical images in the real target domain, the three-dimensional medical image in the real target domain is used as a label to form a third training sample with the corresponding noise sample image; Training the pre-created second generation model according to the third training sample to obtain a trained second generation model; Using the second generation model to perform noise reduction processing on each of the three-dimensional white noise images to obtain a corresponding expanded target domain three-dimensional medical image; The real target domain three-dimensional medical image and the expanded target domain three-dimensional medical image are both used as sample target domain three-dimensional medical images; For each of the sample target domain three-dimensional medical images, downsampling the sample target domain three-dimensional medical image is performed to obtain a corresponding sample sparse target domain three-dimensional medical image; For each of the sample target domain three-dimensional medical images, the sample target domain three-dimensional medical image is used as a label to form a second training sample with its corresponding sample sparse target domain three-dimensional medical image.

8. The three-dimensional ultrasound modeling method according to claim 1, characterized in that: The three-dimensional reconstruction is performed according to the target organ structure image to obtain a three-dimensional model of the target organ, including: A surface reconstruction algorithm is used to perform three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ.

9. The three-dimensional ultrasound modeling method according to claim 1, characterized in that: The step of performing target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image includes: The pre-trained segmentation model is used to segment the target organ anatomical structure of the dense pseudo target domain three-dimensional image to obtain the target organ structure image.

10. A three-dimensional ultrasound modeling system, characterized in that: include: A generation module is configured to perform domain conversion on each frame of an ultrasound image in an acquired target organ ultrasound image sequence using a first generation model to acquire a corresponding pseudo target domain image sequence, wherein the target organ ultrasound image sequence includes multiple frames of ultrasound images acquired by an ultrasound probe in different postures and at different depths in a cavity of the target organ; a restoration module, configured to acquire a sparse pseudo target domain 3D image according to the pseudo target domain image sequence, and acquire a dense pseudo target domain 3D image according to the sparse pseudo target domain 3D image; a segmentation module configured to perform target organ anatomical structure segmentation on the dense pseudo target domain three-dimensional image to obtain a target organ structure image; and A modeling module is configured to perform three-dimensional reconstruction based on the target organ structure image to obtain a three-dimensional model of the target organ.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the three-dimensional ultrasound modeling method according to any one of claims 1 to 9 is implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional ultrasound modeling method according to any one of claims 1 to 9 is implemented.

Images