Heart structure segmentation model training and segmentation method for fetal echocardiography

Abstract

The invention relates to the technical field of medical image processing, in particular to a fetal echocardiogram heart structure segmentation model training and segmentation method, and the method comprises the steps: obtaining four-cavity view images of a plurality of fetal echocardiograms, and marking the four-cavity view images; generating a training data set according to the labeled four-cavity view image and the correspondingly segmented mask image; the training data set is utilized to train a heart structure segmentation model, the heart structure segmentation model outputs a heart structure segmentation result, the heart structure segmentation model comprises an image encoder, a prompt encoder and a mask decoder, a global attention module of the image encoder comprises an adapter, and a global attention module of the image encoder comprises a mask decoder; the image encoder and the mask decoder are connected through a cross attention module. Therefore, the problems of poor heart structure segmentation accuracy and the like in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a cardiac structure segmentation model training and segmentation method for fetal echocardiography. Background Art

[0002] Fetal echocardiography is crucial in prenatal diagnosis, especially in the assessment of fetal heart structure and function, with the advantages of being non-invasive, real-time, and repeatable. The apical four-chamber view is a commonly used perspective, which can clearly show the four chambers of the heart and their connections, and is of great significance for the diagnosis of congenital heart disease and the assessment of cardiac function. However, accurate segmentation of cardiac structures in the four-chamber view is extremely challenging: traditional image processing methods (such as threshold segmentation, edge detection, etc.) are effective in simple high-contrast images, but they are difficult to cope with the low contrast, blurred edges, and overlapping structures of echocardiography. In addition, the influence of physiological activities such as fetal movement increases the difficulty of segmentation.

[0003] Deep learning has been widely used in the field of medical imaging and has achieved remarkable results. It achieves high-precision segmentation of medical images by automatically learning features. However, in echocardiographic segmentation, it faces two major challenges: first, data scarcity and high annotation costs limit model training; second, existing models have difficulty integrating global and local information, resulting in insufficient segmentation accuracy and robustness. To this end, related technologies have attempted to introduce large visual models. These models excel in natural image segmentation and possess strong representation and generalization capabilities. However, due to the lack of prior knowledge of medical images, they are not very effective in ultrasound image segmentation, especially fetal echocardiography. Summary of the Invention

[0004] The present application provides a cardiac structure segmentation model training and segmentation method for fetal echocardiography to solve the problems of poor accuracy of cardiac structure segmentation in related technologies.

[0005] The first aspect of the present application provides a method for training a cardiac structure segmentation model for fetal echocardiography, comprising the following steps: acquiring a plurality of four-chamber view images of fetal echocardiography, and annotating the four-chamber view images; generating a training data set based on the annotated four-chamber view images and the corresponding segmented mask images; using the training data set to train a cardiac structure segmentation model, the cardiac structure segmentation model outputting a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, the global attention module of the image encoder includes an adapter, and the image encoder and the mask decoder are connected by a cross-attention module.

[0006] Optionally, during the training process, the parameters of the adapter and the cross-attention module are optimized based on the objective function until the target stopping condition is reached, and the training of the cardiac structure segmentation model is stopped.

[0007] Optionally, the global attention module includes a multi-head self-attention mechanism, a residual connection module and a feedforward neural network, and the adapter is located between the multi-head self-attention mechanism and the residual connection module, and the adapter is used to fine-tune the image encoder through downsampling linear projection, activation function and upsampling linear projection operations.

[0008] Optionally, the cross-attention module includes a spatial cross-attention module and a channel cross-attention module, wherein the spatial cross-attention module is used to locate high-level semantic areas to pixel-level positions of low-level features; and the channel cross-attention module is used to add channels related to cardiac structures.

[0009] Optionally, the cross-attention module is further used to calculate the association information between the output features of the image encoder and the hint features of the hint encoder, and generate a dense hint supplement for the cardiac structure segmentation model based on the association information.

[0010] Optionally, after stopping training the cardiac structure segmentation model, the method further includes: obtaining a test data set for the cardiac structure segmentation model; evaluating the trained cardiac structure segmentation model using the test data set, and optimizing the model structure and / or hyperparameters of the cardiac segmentation model based on the evaluation results.

[0011] The second aspect of the present application provides a method for cardiac structure segmentation of fetal echocardiography, which is implemented using a cardiac structure segmentation model trained by the cardiac structure segmentation model training method for fetal echocardiography of the above-mentioned embodiment, and includes the following steps: obtaining a four-chamber view image of the fetal echocardiogram to be segmented; inputting the four-chamber view image of the fetal echocardiogram to be segmented into the trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs a corresponding cardiac structure segmentation result.

[0012] In a third aspect, an embodiment of the present application provides a cardiac structure segmentation model training device for fetal echocardiography, comprising: a first acquisition module for acquiring a plurality of four-chamber view images of fetal echocardiography and labeling the four-chamber view images; a generation module for generating a training data set based on the labeled four-chamber view images and the corresponding segmented mask images; a training module for training a cardiac structure segmentation model using the training data set, the cardiac structure segmentation model outputting a cardiac structure segmentation result, wherein the cardiac structure segmentation model comprises an image encoder, a prompt encoder and a mask decoder, the global attention module of the image encoder comprises an adapter, and the image encoder and the mask decoder are connected via a cross-attention module.

[0013] Optionally, during the training process, the parameters of the adapter and the cross-attention module are optimized based on the objective function until the target stopping condition is reached, and the training of the cardiac structure segmentation model is stopped.

[0014] Optionally, the global attention module includes a multi-head self-attention mechanism, a residual connection module and a feedforward neural network, and the adapter is located between the multi-head self-attention mechanism and the residual connection module, and the adapter is used to fine-tune the image encoder through downsampling linear projection, activation function and upsampling linear projection operations.

[0015] Optionally, the cross-attention module includes a spatial cross-attention module and a channel cross-attention module, wherein the spatial cross-attention module is used to locate high-level semantic areas to pixel-level positions of low-level features; and the channel cross-attention module is used to add channels related to cardiac structures.

[0016] Optionally, the cross-attention module is further used to calculate the association information between the output features of the image encoder and the hint features of the hint encoder, and generate a dense hint supplement for the cardiac structure segmentation model based on the association information.

[0017] Optionally, it also includes: an optimization module, which is used to obtain a test data set of the cardiac structure segmentation model after stopping training the cardiac structure segmentation model; use the test data set to evaluate the trained cardiac structure segmentation model, and optimize the model structure and / or hyperparameters of the cardiac segmentation model based on the evaluation results.

[0018] In a fourth aspect, an embodiment of the present application provides a cardiac structure segmentation device for fetal echocardiography, which is implemented using a cardiac structure segmentation model trained by a cardiac structure segmentation model training device for fetal echocardiography as in the above-mentioned embodiment, and includes: a second acquisition module for acquiring a four-chamber view image of a fetal echocardiogram to be segmented; an input module for inputting the four-chamber view image of the fetal echocardiogram to be segmented into the trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs a corresponding cardiac structure segmentation result.

[0019] A fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to execute the cardiac structure segmentation model training method for fetal echocardiography as in the above-mentioned embodiment, or the cardiac structure segmentation method for fetal echocardiography as in the above-mentioned embodiment.

[0020] Therefore, this application has at least the following beneficial effects:

[0021] The embodiment of the present application can annotate multiple four-chamber view images of fetal echocardiograms, and generate a training data set based on the annotated four-chamber view images and the corresponding segmented mask images. The training data set is used to train a cardiac structure segmentation model, and the cardiac structure segmentation model outputs a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, and the global attention module of the image encoder includes an adapter. The image encoder and the mask decoder are connected by a cross-attention module. By introducing the adapter and the cross-attention module, while maintaining the powerful representation ability of the cardiac structure segmentation model, the targeted optimization of the echocardiogram is achieved, thereby improving the fetal echocardiogram. Figure 4 The accuracy of cardiac structure segmentation in the cavity view is improved. This solves the technical problem of poor accuracy of cardiac structure segmentation in related technologies.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 This is a flowchart of a method for training a cardiac structure segmentation model for fetal echocardiography according to an embodiment of the present application;

[0025] Figure 2 This is a flowchart of a method for training a cardiac structure segmentation model for fetal echocardiography according to one embodiment of the present application;

[0026] Figure 3 This is a flowchart of a method for segmenting cardiac structures in fetal echocardiography according to an embodiment of the present application;

[0027] Figure 4 This is an example diagram of a cardiac structure segmentation model training device for fetal echocardiography provided according to an embodiment of the present application;

[0028] Figure 5 This is an example diagram of a cardiac structure segmentation device for fetal echocardiography provided according to an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] Fetal echocardiography plays a crucial role in prenatal diagnosis, particularly in assessing fetal cardiac structure and function, offering advantages such as non-invasiveness, real-time performance, and high reproducibility. The apical four-chamber view is one of the most commonly used views in fetal echocardiography, clearly demonstrating the four chambers of the heart (left ventricle, left atrium, right ventricle, and right atrium) and their connections. This is crucial for diagnosing congenital heart disease and assessing cardiac function. However, accurate segmentation of cardiac structures in the four-chamber view is a challenging task. Traditional cardiac structure segmentation methods primarily rely on image processing techniques, such as threshold segmentation, edge detection, and morphological processing. These methods achieve good results with simple, high-contrast images, but perform poorly with complex and variable echocardiographic images. Echocardiographic images typically exhibit low contrast, blurred edges, and overlapping structures. Furthermore, the influence of fetal physiological activities such as movement and respiration within the mother's body can make it difficult to clearly identify cardiac structures in the images, further complicating segmentation.

[0032] With the rapid development of deep learning technology, deep neural networks have been widely applied to the field of medical imaging and have made significant progress. These methods achieve high-precision segmentation of medical images by automatically learning image features. However, the application of deep learning technology to echocardiographic image segmentation still faces many challenges. On the one hand, the scarcity of echocardiographic image data and the high cost of annotation limit the training scale and performance improvement of deep learning models. On the other hand, existing deep learning models often struggle to simultaneously integrate global and local information when processing echocardiographic images, resulting in insufficient accuracy and robustness of segmentation results.

[0033] In order to overcome these challenges, related technologies have attempted to apply visual large models to the field of medical image segmentation. Visual large models perform well in natural image segmentation tasks and have strong representation and generalization capabilities. However, due to the lack of prior knowledge about medical images, existing visual large models do not perform well in ultrasound image segmentation tasks. Especially when processing fetal echocardiogram images, due to the complexity of the fetal heart structure and the particularity of ultrasound images, visual large models often find it difficult to accurately capture the detailed features of the heart structure, resulting in deviations in the segmentation results. In addition, the labeling habits of ultrasound doctors are different, resulting in certain differences in the labeling of the data. Therefore, how to effectively apply visual large models to fetal echocardiography is a key issue. Figure 4 The segmentation of cardiac structures in the cavity view has become a problem that needs to be solved urgently.

[0034] To this end, the present application provides a method for training a cardiac structure segmentation model for fetal echocardiography, in which a plurality of four-chamber view images of fetal echocardiography can be annotated, and a training data set can be generated based on the annotated four-chamber view images and the corresponding segmented mask images. The training data set is used to train the cardiac structure segmentation model, and the cardiac structure segmentation model outputs a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, and the global attention module of the image encoder includes an adapter, and the image encoder and the mask decoder are connected by a cross-attention module. By introducing the adapter and the cross-attention module, while maintaining the powerful representation ability of the cardiac structure segmentation model, the targeted optimization of the echocardiogram is achieved, thereby improving the fetal echocardiography. Figure 4 The accuracy of cardiac structure segmentation in the cavity view is improved. This solves the problem of poor accuracy of cardiac structure segmentation in related technologies.

[0035] Specifically, Figure 1 A flowchart of a method for training a cardiac structure segmentation model for fetal echocardiography provided in an embodiment of the present application.

[0036] like Figure 1 As shown, the fetal echocardiogram cardiac structure segmentation model training method includes the following steps:

[0037] In step S101 , a plurality of four-chamber view images of fetal echocardiograms are acquired and the four-chamber view images are labeled.

[0038] Among them, the four-chamber view image of fetal echocardiography can select the end-systolic and end-diastolic echocardiographic images. Figure 4 The cavity view is used as a key section for cardiac structure segmentation, and the four-chamber view images are annotated, which can be done manually.

[0039] In step S102 , a training data set is generated according to the annotated four-chamber view images and the corresponding segmented mask images.

[0040] It is understandable that the embodiment of the present application can generate a training data set based on the annotated four-chamber view images and the corresponding segmented mask images to facilitate the subsequent training of the cardiac structure segmentation model.

[0041] In step S103, a cardiac structure segmentation model is trained using the training data set, and the cardiac structure segmentation model outputs a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, the global attention module of the image encoder includes an adapter, and the image encoder and the mask decoder are connected through a cross attention module.

[0042] Among them, the structure of the cardiac structure segmentation model can be based on the visual large model SAM2, and the image encoder can be a Transformer encoder.

[0043] It can be understood that the embodiment of the present application can use the training data set to train the cardiac structure segmentation model, and the cardiac structure segmentation model outputs the cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, and the global attention module of the image encoder includes an adapter. The image encoder and the mask decoder are connected by a cross attention module. By introducing the adapter and the cross attention module, while maintaining the powerful representation ability of SAM2, the targeted optimization of the echocardiogram is achieved, thereby improving the fetal echocardiogram. Figure 4 Accuracy of cardiac structure segmentation in cavity view.

[0044] In an embodiment of the present application, the global attention module includes a multi-head self-attention mechanism, a residual connection module and a feedforward neural network. The adapter is located between the multi-head self-attention mechanism and the residual connection module. The adapter is used to fine-tune the image encoder through downsampling linear projection, activation function and upsampling linear projection operations.

[0045] It can be understood that the global attention module of the embodiment of the present application includes a multi-head attention mechanism, a residual connection module and a feedforward neural network, and the adapter is located between the multi-head self-attention mechanism and the residual connection module. The adapter is used to fine-tune the image encoder through downsampling linear projection, activation function and upsampling linear projection operations.

[0046] In an embodiment of the present application, the cross-attention module includes a spatial cross-attention module and a channel cross-attention module, wherein the spatial cross-attention module is used to locate the high-level semantic area to the pixel-level position of the low-level feature; the channel cross-attention module is used to add channels related to the cardiac structure.

[0047] It can be understood that the cross-attention module of the embodiment of the present application includes a spatial cross-attention module and a channel cross-attention module. The spatial cross-attention module is used to locate the high-level semantic area to the pixel-level position of the low-level feature, and the channel cross-attention module is used to add channels related to the heart structure.

[0048] Specifically, a multi-level feature cross-attention module is integrated in the jump connection between the image encoder and the mask decoder to perform spatial cross-attention and channel cross-attention calculations to integrate and interact multi-level features and realize the fusion and interaction of multi-level features.

[0049] In an embodiment of the present application, the cross-attention module is further used to calculate the association information between the output features of the image encoder and the prompt features of the prompt encoder, and generate a dense prompt supplement for the cardiac structure segmentation model based on the association information.

[0050] It can be understood that the cross-attention module of the embodiment of the present application is further used to calculate the association between the output features of the image encoder and the prompt features of the prompt encoder, and generate a dense prompt supplement for the cardiac structure segmentation model to improve the accuracy and robustness of cardiac structure segmentation.

[0051] In an embodiment of the present application, during the training process, the parameters of the adapter and the cross-attention module are optimized based on the objective function until the target stopping condition is reached, and the training of the cardiac structure segmentation model is stopped.

[0052] Among them, the objective function can adopt a combination of Dice loss (dice loss) and BCE loss (binary cross entropy loss); the target stopping condition can be that the number of iterations is reached, or the segmentation accuracy reaches a certain standard.

[0053] It can be understood that, during the training process, the embodiment of the present application optimizes the parameters of the adapter and the cross-attention module based on the objective function, freezes the pre-training parameters of SAM2, and stops training the cardiac structure segmentation model until the target stopping condition is reached.

[0054] In addition, it should be noted that the embodiment of the present application can divide the training data set into two parts of data, one part of the data is used for pre-training the cardiac structure segmentation model, and the other part of the data is used for formal training of the cardiac structure segmentation model, wherein the pre-training can provide prior knowledge of echocardiography for formal training.

[0055] Specifically, the cardiac structure segmentation model in the embodiments of this application is based on the large-scale visual model SAM2, comprising an image encoder, a cue encoder, and a mask decoder. The core structure of the SAM2 model is based on powerful pre-trained visual representations. It achieves deep extraction and understanding of image features through a multi-stage stack of Transformer encoder layers. Each Transformer encoder layer incorporates a multi-head self-attention mechanism and a feedforward neural network, enabling it to capture the complex dependencies between different regions in the image. Furthermore, the SAM2 model, through its carefully designed mask prediction head, achieves precise segmentation of different objects or regions in the image, demonstrating its superior performance in complex visual tasks. The difference lies in the introduction of an adapter to the global attention block in the image encoder of the original SAM2 model for efficient fine-tuning of the image encoder, and the introduction of a cross-attention module between the encoder and decoder for feature fusion and interaction. During subsequent pre-training and training, only the parameters of the adapter and cross-attention module are updated, leaving the pre-trained parameters of SAM2 frozen. The objective function adopts a combination of Dice loss and BCE loss.

[0056] In an embodiment of the present application, after stopping training the cardiac structure segmentation model, it also includes: obtaining a test data set for the cardiac structure segmentation model; using the test data set to evaluate the trained cardiac structure segmentation model, and optimizing the model structure and / or hyperparameters of the cardiac segmentation model based on the evaluation results.

[0057] Among them, the evaluation indicators include IoU and Dice coefficient.

[0058] It is understandable that the embodiment of the present application can use the test data set to evaluate the trained cardiac structure segmentation model, and optimize the model structure and hyperparameters of the cardiac segmentation model based on the evaluation results to further improve the segmentation performance of the cardiac structure segmentation model.

[0059] In summary, the cardiac structure segmentation model training method of fetal echocardiography of this application is as follows: Figure 2 As shown, the following steps are included:

[0060] Step 1: Obtain fetal echocardiography Figure 4 The cavity view images are labeled under the guidance of professional doctors, a semantic segmentation training dataset is constructed, and data preprocessing is performed.

[0061] First, according to professional books and doctor's advice, select the end-systolic and end-diastolic echocardiogram. Figure 4The cavity view is used as the key section for cardiac structure segmentation, and the four cardiac structures required by the four-chamber view are the left ventricle, left atrium, right ventricle, and right atrium. Then, according to the guidance of professional doctors, the target area is drawn, and the segmentation mask image of the corresponding part is generated. The original image is matched with the target mask image one by one to construct a training data set.

[0062] Step 2: Determine the fine-tuning method of the large visual model, build a fine-tuning network (i.e., a cardiac structure segmentation model) based on the pre-trained large visual model SAM2, and construct the network's objective function.

[0063] The network structure is based on the large visual model SAM2, including image encoder, prompt encoder and mask decoder. The core structure of the SAM2 model is based on powerful pre-trained visual representation. It realizes deep extraction and understanding of image features through multi-stage stacking of Transformer encoder layers. Each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward neural network, which can capture the complex dependencies between different regions in the image. At the same time, the SAM2 model achieves accurate segmentation of different objects or regions in the image through a carefully designed mask prediction head, demonstrating its excellent performance in complex visual tasks. The difference is that in the image encoder of the original SAM2 model, the global attention block introduces an adapter to efficiently fine-tune the image encoder, and introduces a cross-attention module between the encoder and decoder for feature fusion and interaction. In the subsequent pre-training and training process, only the parameters of the introduced adapter and cross-attention module are updated, and the pre-training parameters of SAM2 are frozen. The objective function adopts a combination of Dice loss and BCE loss.

[0064] Step 3: Take out part of the data set as a pre-training set, and determine the pre-training process based on the network and the objective function to complete the pre-training.

[0065] The pre-training process aims to minimize the value of the objective function. The pre-trained network model is obtained according to the final weight parameters of the adapter and cross-attention module in the image encoder, providing prior knowledge of echocardiography for subsequent formal training.

[0066] Step 4: Based on the data set, network, and objective function, determine the training process and formally train the network.

[0067] The training process aims to minimize the value of the objective function and obtain the trained network model based on the final weight parameters of each layer of the network.

[0068] In step 5, the trained model is evaluated using an independent test set, and the model is optimized and adjusted based on the evaluation results to further improve the segmentation performance.

[0069] The evaluation process evaluates the model's generalization ability on different datasets by calculating metrics such as IoU and Dice coefficient. The model is then optimized and adjusted based on the evaluation results, further improving the model's segmentation performance by adjusting the model structure and hyperparameters.

[0070] The following describes a method for training a cardiac structure segmentation model for fetal echocardiography using a specific embodiment, including the following steps:

[0071] Step 1: Select the four-chamber views of the heart at end-systole and end-diastole from the fetal ultrasound images as image samples. First, according to professional books and doctor's advice, the target area is drawn in the corresponding section sample according to the target area, and a mask image containing the target area is generated. The original image and the mask image are matched one-to-one to construct a training set. The labeled images are then preprocessed by normalization and cropping. The size of the ultrasound image and mask image is uniformly scaled to 256×256 to obtain the preprocessed training set.

[0072] Step 2: Construct a cardiac structure segmentation model. The overall network structure is based on the SAM2 model, including an image encoder, a cue encoder, and a mask decoder. The core structure of the SAM2 model is based on powerful pre-trained visual representations. It achieves deep extraction and understanding of image features through a multi-stage stack of Transformer encoder layers. Each Transformer encoder layer contains a multi-head self-attention mechanism and a feedforward neural network, which can capture the complex dependencies between different regions in the image. Furthermore, the SAM2 model achieves accurate segmentation of different objects or regions in the image through a carefully designed mask prediction head, demonstrating its excellent performance in complex visual tasks.

[0073] The difference lies in the introduction of adapters to the residual connections of the multi-head attention and feedforward networks in each global attention block of the image encoder. This allows for efficient fine-tuning of the image encoder through operations such as downsampling linear projection, activation function, and upsampling linear projection. A multi-level feature cross-attention module is integrated into the skip connection between the image encoder and the mask decoder to perform spatial cross-attention and channel cross-attention calculations, integrating and interacting multi-level features and achieving multi-level feature fusion and interaction. A cue information supplementation strategy is proposed, in which cross-attention is calculated on features at different levels of the image encoder output to supplement dense cue information, improving segmentation accuracy and robustness. During subsequent pre-training and training, only the parameters of the introduced adapter and cross-attention module are updated, while the pre-trained parameters of SAM2 are frozen. The objective function adopts a combination of Dice loss and BCE loss.

[0074] Step 3: Select part of the data set from step 1 to form a pre-training set. Based on the network structure and objective function of step 2, pre-train the semantic segmentation network. The training process aims to minimize the value of the objective function. The pre-trained fine-tuning model is obtained according to the final weight parameters of the adapter in the image encoder, the multi-level feature cross-attention module, and the prompt information supplement module, providing prior knowledge of echocardiography for subsequent formal training.

[0075] Step 4: Based on the dataset of step 1 and the network structure and objective function of step 2, formally train the semantic segmentation network. The training process aims to minimize the value of the objective function and obtain the trained network model based on the final weight parameters of the introduced modules and strategies.

[0076] Step 5: Evaluate the segmentation model trained in Step 4 using an independent test set. By calculating metrics such as Intersection over Union (IoU) and the Dice coefficient, we assess the model's generalization ability across different datasets. Based on the evaluation results, we optimize and adjust the model. By adjusting the model structure, hyperparameters, and other settings, we can further improve the model's segmentation performance.

[0077] In general, the cardiac structure segmentation model training method of fetal echocardiography in the embodiment of the present application first collects a large number of fetal echocardiograms. Figure 4 The team collected images of the ventricular cavity view and labeled the data under the guidance of professional doctors, providing a sufficient and effective dataset for fine-tuning the visual large model. Secondly, they innovatively improved a more advanced visual large model fine-tuning method. While maintaining the powerful representational capabilities of the visual large model, it can achieve targeted optimization of echocardiographic images, integrate and interact with the multi-level features in the image encoder, enhance the model's understanding ability, and improve the accuracy and robustness of segmentation.

[0078] This application introduces specific modules to achieve targeted optimization of echocardiographic images while maintaining the powerful representation capabilities of SAM2. The visual large model fine-tuning model used in this application has the characteristics of high precision, high efficiency and high generalization performance. It solves the difficulty of accurately capturing the detailed features of the heart structure due to the complexity of the fetal heart structure and the particularity of ultrasound images. It also balances the differences in ultrasound examination techniques and examination standards of professional doctors, improves the accuracy of segmentation, and is easy to expand to other medical ultrasound image segmentation tasks.

[0079] According to the cardiac structure segmentation model training method for fetal echocardiography proposed in the embodiment of the present application, multiple four-chamber view images of fetal echocardiography can be annotated, and a training data set can be generated based on the annotated four-chamber view images and the corresponding segmented mask images. The training data set is used to train the cardiac structure segmentation model, and the cardiac structure segmentation model outputs the cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, and the global attention module of the image encoder includes an adapter. The image encoder and the mask decoder are connected by a cross-attention module. By introducing the adapter and the cross-attention module, while maintaining the powerful representation ability of the cardiac structure segmentation model, the targeted optimization of the echocardiogram is achieved, thereby improving the fetal echocardiography. Figure 4 Accuracy of cardiac structure segmentation in cavity view.

[0080] The embodiment of the present application also provides a method for segmenting cardiac structures in fetal echocardiography.

[0081] like Figure 3 As shown, the cardiac structure segmentation method for fetal echocardiography is implemented using the cardiac structure segmentation model trained by the above-mentioned cardiac structure segmentation model training method for fetal echocardiography, and includes the following steps:

[0082] In step S201 , a four-chamber view image of a fetal echocardiogram to be segmented is acquired.

[0083] In step S202, the four-chamber view image of the fetal echocardiogram to be segmented is input into the trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs a corresponding cardiac structure segmentation result.

[0084] According to the cardiac structure segmentation method of fetal echocardiography proposed in the embodiment of the present application, the four-chamber view image of the fetal echocardiography to be segmented can be input into a trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs the corresponding cardiac structure segmentation result to achieve accurate segmentation of the cardiac structure.

[0085] Next, a fetal echocardiographic cardiac structure segmentation model training device and a fetal echocardiographic cardiac structure segmentation device proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0086] Figure 4 4 is a block diagram of a cardiac structure segmentation model training device for fetal echocardiography according to an embodiment of the present application.

[0087] like Figure 4 As shown, the fetal echocardiogram cardiac structure segmentation model training device 10 includes: a first acquisition module 101 , a generation module 102 and a training module 103 .

[0088] Among them, the first acquisition module 101 is used to acquire multiple four-chamber view images of fetal echocardiograms and annotate the four-chamber view images; the generation module 102 is used to generate a training data set based on the annotated four-chamber view images and the corresponding segmented mask images; the training module 103 is used to train a cardiac structure segmentation model using the training data set, and the cardiac structure segmentation model outputs a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, the global attention module of the image encoder includes an adapter, and the image encoder and the mask decoder are connected through a cross-attention module.

[0089] In an embodiment of the present application, during the training process, the parameters of the adapter and the cross-attention module are optimized based on the objective function until the target stopping condition is reached, and the training of the cardiac structure segmentation model is stopped.

[0090] In an embodiment of the present application, the global attention module includes a multi-head self-attention mechanism, a residual connection module and a feedforward neural network. The adapter is located between the multi-head self-attention mechanism and the residual connection module. The adapter is used to fine-tune the image encoder through downsampling linear projection, activation function and upsampling linear projection operations.

[0091] In an embodiment of the present application, the cross-attention module includes a spatial cross-attention module and a channel cross-attention module, wherein the spatial cross-attention module is used to locate the high-level semantic area to the pixel-level position of the low-level feature; the channel cross-attention module is used to add channels related to the cardiac structure.

[0092] In an embodiment of the present application, the cross-attention module is further used to calculate the association information between the output features of the image encoder and the prompt features of the prompt encoder, and generate a dense prompt supplement for the cardiac structure segmentation model based on the association information.

[0093] In the embodiment of the present application, the device 10 of the embodiment of the present application further includes: an optimization module.

[0094] Among them, the optimization module is used to obtain a test data set of the cardiac structure segmentation model after stopping training the cardiac structure segmentation model; use the test data set to evaluate the trained cardiac structure segmentation model, and optimize the model structure and / or hyperparameters of the cardiac segmentation model based on the evaluation results.

[0095] It should be noted that the above explanation of the embodiment of the training method for the cardiac structure segmentation model of fetal echocardiography is also applicable to the training device for the cardiac structure segmentation model of fetal echocardiography of this embodiment, and will not be repeated here.

[0096] According to the cardiac structure segmentation model training device for fetal echocardiography proposed in the embodiment of the present application, multiple four-chamber view images of fetal echocardiography can be annotated, and a training data set can be generated based on the annotated four-chamber view images and the corresponding segmented mask images. The training data set is used to train the cardiac structure segmentation model, and the cardiac structure segmentation model outputs the cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, and the global attention module of the image encoder includes an adapter. The image encoder and the mask decoder are connected by a cross-attention module. By introducing the adapter and the cross-attention module, while maintaining the powerful representation ability of the cardiac structure segmentation model, the targeted optimization of the echocardiogram is achieved, thereby improving the fetal echocardiography. Figure 4 Accuracy of cardiac structure segmentation in cavity view.

[0097] Figure 5 3 is a block diagram of a heart segmentation method for fetal echocardiography according to an embodiment of the present application.

[0098] like Figure 5 As shown, the cardiac structure segmentation device 20 for fetal echocardiography includes: a second acquisition module 201 and an input module 202 .

[0099] Among them, the second acquisition module 201 is used to obtain the four-chamber view image of the fetal echocardiogram to be segmented; the input module 202 is used to input the four-chamber view image of the fetal echocardiogram to be segmented into the trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs the corresponding cardiac structure segmentation result.

[0100] It should be noted that the above explanations of the embodiment of the method for segmenting cardiac structure of fetal echocardiography are also applicable to the apparatus for segmenting cardiac structure of fetal echocardiography of this embodiment, and will not be repeated here.

[0101] According to the cardiac structure segmentation device for fetal echocardiography proposed in an embodiment of the present application, the four-chamber view image of the fetal echocardiogram to be segmented can be input into a trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs the corresponding cardiac structure segmentation result to achieve accurate segmentation of the cardiac structure.

[0102] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0103] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0104] When the processor 602 executes the program, the cardiac structure segmentation model training method for fetal echocardiography or the cardiac structure segmentation method for fetal echocardiography provided in the above embodiments is implemented.

[0105] Furthermore, the electronic device further includes:

[0106] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0107] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0108] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0109] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0110] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0111] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0114] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0115] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0116] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

Claims

1. A method for training a cardiac structure segmentation model for fetal echocardiography, characterized in that: The following steps are involved: acquiring a plurality of four-chamber view images of fetal echocardiography, and annotating the four-chamber view images; Generate a training dataset based on the annotated four-chamber view images and the corresponding segmented mask images; A cardiac structure segmentation model is trained using the training data set, and the cardiac structure segmentation model outputs a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, the global attention module of the image encoder includes an adapter, and the image encoder and the mask decoder are connected through a cross attention module.

2. The method for training a cardiac structure segmentation model for fetal echocardiography according to claim 1, wherein: During the training process, the parameters of the adapter and the cross-attention module are optimized based on the objective function until the target stopping condition is reached, and the training of the cardiac structure segmentation model is stopped.

3. The method for training a cardiac structure segmentation model for fetal echocardiography according to claim 1, wherein: The global attention module includes a multi-head self-attention mechanism, a residual connection module and a feedforward neural network. The adapter is located between the multi-head self-attention mechanism and the residual connection module. The adapter is used to fine-tune the image encoder through downsampling linear projection, activation function and upsampling linear projection operations.

4. The method for training a cardiac structure segmentation model for fetal echocardiography according to claim 1, wherein: The cross attention module includes a spatial cross attention module and a channel cross attention module, wherein, The spatial crisscross attention module is used to locate high-level semantic regions to the pixel-level positions of low-level features; The channel cross attention module is used to increase channels related to cardiac structures.

5. The method for training a cardiac structure segmentation model for fetal echocardiography according to claim 3, wherein: The cross-attention module is further used to calculate the association information between the output features of the image encoder and the prompt features of the prompt encoder, and generate a dense prompt supplement for the cardiac structure segmentation model based on the association information.

6. The method for training a cardiac structure segmentation model for fetal echocardiography according to claim 2, wherein: After stopping the training of the cardiac structure segmentation model, the method further includes: Acquiring a test data set for the cardiac structure segmentation model; The trained cardiac structure segmentation model is evaluated using the test data set, and the model structure and / or hyperparameters of the cardiac segmentation model are optimized based on the evaluation results.

7. A method for segmenting cardiac structures in fetal echocardiography, characterized in that: The cardiac structure segmentation model trained by the cardiac structure segmentation model training method for fetal echocardiography according to any one of claims 1 to 6 is implemented, comprising the following steps: acquiring a four-chamber view image of a fetal echocardiogram to be segmented; The four-chamber view image of the fetal echocardiogram to be segmented is input into a trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs a corresponding cardiac structure segmentation result.

8. A cardiac structure segmentation model training device for fetal echocardiography, characterized in that: include: a first acquisition module, configured to acquire a plurality of four-chamber view images of fetal echocardiograms and annotate the four-chamber view images; A generation module, used to generate a training data set based on the annotated four-cavity view images and the corresponding segmented mask images; A training module is used to train a cardiac structure segmentation model using the training data set, and the cardiac structure segmentation model outputs a cardiac structure segmentation result, wherein the cardiac structure segmentation model includes an image encoder, a prompt encoder and a mask decoder, the global attention module of the image encoder includes an adapter, and the image encoder and the mask decoder are connected through a cross attention module.

9. A cardiac structure segmentation device for fetal echocardiography, characterized in that: The cardiac structure segmentation model trained by the cardiac structure segmentation model training device for fetal echocardiography according to claim 8 is implemented, comprising: a second acquisition module, configured to acquire a four-chamber view image of a fetal echocardiogram to be segmented; An input module is used to input the four-chamber view image of the fetal echocardiogram to be segmented into a trained cardiac structure segmentation model, and the cardiac structure segmentation model outputs a corresponding cardiac structure segmentation result.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cardiac structure segmentation model training method for fetal echocardiography according to any one of claims 1 to 6, or the cardiac structure segmentation method for fetal echocardiography according to claim 7.

Images