An uncertainty-aware multi-task ultrasound image segmentation method and system

Abstract

The application discloses an uncertainty-aware multi-task ultrasound image segmentation method and system. The method obtains ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, extracts shared image features and multiple task semantic features; determines a task probability distribution based on the shared image features and fuses to obtain current task semantic features; generates semantic modulation parameters according to the current task semantic features, and determines semantic modulation reliability parameters in combination with image-text compatibility, task routing uncertainty and segmentation prediction uncertainty to adaptively modulate the shared image features; task memory constraints are constructed in the training stage, and a target ultrasound image segmentation model is trained to output a target region segmentation result in the inference stage. Thus, the accuracy, stability and robustness of multi-task ultrasound image segmentation can be improved by reducing false semantic enhancement, task interference and feature drift.

Description

Technical Field

[0001] This application relates to the fields of medical image analysis and artificial intelligence technology, and in particular to a multi-task ultrasound image segmentation method and system with uncertainty perception. Background Technology

[0002] Ultrasound imaging, as one of the most widely used medical imaging technologies in clinical medicine, boasts advantages such as being non-invasive, real-time, safe, low-cost, and portable. It is widely applied in medical diagnostic scenarios such as breast lesion detection, thyroid nodule screening, cardiac structure assessment, and left ventricular function analysis. With the development of artificial intelligence and deep learning technologies, assisted diagnostic methods based on automatic medical image segmentation have gradually become an important research direction in the field of medical image analysis. By automatically identifying and accurately segmenting lesion regions or anatomical structures in ultrasound images, not only can the diagnostic efficiency of doctors be improved, but also important data support can be provided for disease screening, quantitative analysis, and treatment evaluation.

[0003] Currently, researchers have proposed various automatic segmentation methods based on convolutional neural networks, Transformer networks, and large-scale pre-trained models for ultrasound image segmentation. In particular, the general medical image segmentation models and text-guided segmentation models that have emerged in recent years have improved the generalization performance of models across different segmentation tasks by incorporating the collaborative modeling capabilities of visual features and textual semantic information. However, in multi-task ultrasound image segmentation scenarios, existing methods still have many shortcomings due to the large differences in target structure, significant fluctuations in image quality, blurred target boundaries, and imbalanced sample distribution among different ultrasound datasets. For example, textual semantic information often only participates in target localization during the decoding stage, making it difficult to effectively guide shared visual features; when image quality is poor or task category judgment is unclear, semantic information is prone to erroneous enhancement, affecting the accuracy of segmentation results; furthermore, multi-task joint training is prone to problems such as inter-task feature interference, negative transfer, and task forgetting, thereby reducing the model's segmentation stability and generalization ability for different ultrasound targets.

[0004] Therefore, in the process of multi-task ultrasound image segmentation, the insufficient fusion depth of text semantics and visual features, the low reliability of semantic modulation, and the tendency of multi-task joint training to generate task interference and feature drift have become problems that urgently need to be solved. Summary of the Invention

[0005] This application provides an uncertainty-aware multi-task ultrasound image segmentation method and system, aiming to solve the problems of insufficient fusion depth of text semantics and visual features, low reliability of semantic modulation, and easy task interference and feature drift in the multi-task joint training process of existing technologies in the multi-task ultrasound image segmentation process.

[0006] A first aspect is an uncertainty-aware multi-task ultrasound image segmentation method, the method comprising: Obtain ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, and perform feature encoding on the ultrasound images and task semantic information respectively to obtain shared image features and multiple task semantic features; Based on the shared image features, the task probability distribution corresponding to the current ultrasound image is determined, and the multiple task semantic features are fused according to the task probability distribution to obtain the current task semantic features; Based on the semantic features of the current task, semantic modulation parameters for adjusting the shared image features are generated, and semantic modulation reliability parameters are determined according to the image-text compatibility between the shared image features and the semantic features of the current task, the task routing uncertainty corresponding to the task probability distribution, and the segmentation prediction uncertainty corresponding to the current ultrasound image. The semantic modulation parameters are adjusted according to the semantic modulation reliability parameters, and the shared image features are adaptively modulated based on the adjusted semantic modulation parameters to obtain semantically enhanced image features; During the model training phase, a task memory constraint is constructed based on the semantically enhanced image features, the historical task feature prototype of the corresponding task, and the corresponding task semantic features obtained from the multiple task semantic features. The ultrasound image segmentation model is then trained based on the joint training objective including the task memory constraint to obtain the target ultrasound image segmentation model. During the model inference stage, the semantically enhanced image features corresponding to the ultrasound image to be segmented and the semantic features of the current task are input into the target ultrasound image segmentation model to obtain the target region segmentation result in the ultrasound image to be segmented.

[0007] Optionally, in the above scheme, obtaining the ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks includes: Multiple sets of ultrasound images corresponding to different anatomical regions or different segmentation targets are acquired, and the multiple sets of ultrasound images are resized and standardized to obtain standard ultrasound images. Based on the segmentation targets corresponding to each set of standard ultrasound images, task description text is constructed to obtain task text prompts corresponding to different ultrasound image segmentation tasks. The standard ultrasound image, the task text prompt, the corresponding task identifier, and the segmentation label mask are associated to obtain multi-task ultrasound image training samples.

[0008] Optionally, in the above scheme, the step of feature encoding the ultrasound image and the task semantic information respectively includes: The standard ultrasound image is input into an image coding network shared by multiple ultrasound image segmentation tasks for feature extraction, resulting in shared image features used to characterize the target region and image context. The task text prompts corresponding to each ultrasound image segmentation task are input into a text encoding network for semantic encoding to obtain task semantic features that correspond one-to-one with each ultrasound image segmentation task. The task semantic features are stored according to the task identifier to obtain a task semantic feature library for subsequent task semantic selection.

[0009] Optionally, in the above scheme, determining the task probability distribution corresponding to the current ultrasound image based on the shared image features includes: Global feature aggregation is performed on the shared image features to obtain the global image features corresponding to the current ultrasound image; The global features of the image are input into the task routing network to predict the task category, thereby obtaining the task prediction values ​​of the current ultrasound image for each ultrasound image segmentation task. The predicted task values ​​are subjected to probability normalization to obtain the task probability distribution, and each task probability in the task probability distribution is used as the semantic selection weight of the corresponding task semantic features.

[0010] Optionally, in the above scheme, fusing the semantic features of the multiple tasks according to the task probability distribution and generating the semantic modulation parameters based on the current task semantic features includes: The semantic selection weights are used to perform weighted aggregation on multiple task semantic features in the task semantic feature library to obtain the current task semantic features. Based on the task probability distribution, the task-level modulation parameters corresponding to different ultrasound image segmentation tasks are weighted and fused to obtain the task modulation parameters. The semantic features of the current task are input into the semantic parameter generation network for parameter mapping to obtain semantic residual modulation parameters; The task modulation parameters and the semantic residual modulation parameters are fused to obtain the semantic modulation parameters, which include feature scaling parameters and feature offset parameters.

[0011] Optionally, in the above scheme, determining the semantic modulation reliability parameters includes: Global feature mapping is performed on the shared image features, and the similarity between the mapped image features and the semantic features of the current task is calculated to obtain the image-text compatibility. A coarse segmentation probability map is generated based on the shared image features. The pixel prediction entropy is calculated according to the target category probability corresponding to each pixel in the coarse segmentation probability map. The pixel prediction entropy is then aggregated to obtain the segmentation prediction uncertainty. The task routing entropy is calculated based on the probability of each task in the task probability distribution, and the task routing uncertainty is obtained. The image-text compatibility, the segmentation prediction uncertainty, and the task routing uncertainty are combined to obtain reliability features. These reliability features are then input into a reliability evaluation network. Based on the output of the reliability evaluation network and a preset maximum modulation intensity, the semantic modulation reliability parameters are obtained.

[0012] Optionally, in the above scheme, adjusting the semantic modulation parameters according to the semantic modulation reliability parameters and adaptively modulating the shared image features based on the adjusted semantic modulation parameters includes: Based on the feature scaling parameter and feature offset parameter in the semantic modulation parameters, the shared image features are linearly modulated to obtain semantically modulated features; The semantic modulation features are weighted using the semantic modulation reliability parameters to obtain reliable semantic modulation features; The reliable semantic adjustment features are residually fused with the shared image features to obtain the semantically enhanced image features.

[0013] Optionally, in the above scheme, the step of constructing task memory constraints based on the semantically enhanced image features and the historical task feature prototypes of the corresponding task includes: During the model training phase, the segmentation label mask corresponding to the current ultrasound image is adjusted to a feature size that matches the semantically enhanced image features to obtain the target region feature mask; Based on the target region feature mask, foreground features of the target region in the semantically enhanced image features are aggregated to obtain the target region foreground features; Based on the task identifier corresponding to the current ultrasound image, retrieve the historical task feature prototype of the corresponding ultrasound image segmentation task from the task memory; Calculate the feature difference between the foreground features of the target region and the historical task feature prototype to obtain the feature offset degree; The feature offset degree is compared with a preset offset threshold. When the feature offset degree is greater than the preset offset threshold, a prototype consistency constraint is activated to constrain the foreground features of the target region to remain consistent with the prototype features of the historical task.

[0014] Optionally, in the above scheme, the step of constructing task memory constraints based on the corresponding task semantic features and training the ultrasound image segmentation model based on the task memory constraints includes: Based on the task identifier corresponding to the current ultrasound image, the corresponding task semantic features are obtained from the plurality of task semantic features; The foreground features of the target region are input into a semantic projection network for feature space mapping to obtain foreground semantic features; Calculate the semantic similarity between the foreground semantic features and the corresponding task semantic features, and construct text alignment constraints based on the semantic similarity. The prototype consistency constraint and the text alignment constraint are combined to obtain the task memory constraint; When the feature offset is not greater than a preset offset threshold, the foreground features of the target region and the historical task feature prototype are fused using an exponential moving average method to obtain an updated historical task feature prototype. When the feature offset is greater than the preset offset threshold, the historical task feature prototype remains unchanged, and the task memory constraint is used for parameter optimization of the ultrasound image segmentation model.

[0015] Secondly, an uncertainty-aware multi-task ultrasound image segmentation system, the system comprising: The feature encoding module is used to acquire ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, and to perform feature encoding on the ultrasound images and task semantic information respectively to obtain shared image features and multiple task semantic features. The task routing module is used to determine the task probability distribution corresponding to the current ultrasound image based on the shared image features, and to fuse the multiple task semantic features according to the task probability distribution to obtain the current task semantic features. An uncertainty-aware semantic modulation module is used to generate semantic modulation parameters based on the semantic features of the current task, determine semantic modulation reliability parameters according to image-text compatibility, task routing uncertainty and segmentation prediction uncertainty, and adjust the semantic modulation parameters according to the semantic modulation reliability parameters to adaptively modulate the shared image features to obtain semantically enhanced image features. The text alignment task memory module is used to construct task memory constraints during the model training phase based on the semantically enhanced image features, the historical task feature prototypes of the corresponding task, and the corresponding task semantic features obtained from the multiple task semantic features. The model training module is used to train the ultrasound image segmentation model based on the joint training objective including the task memory constraint, so as to obtain the target ultrasound image segmentation model. The segmentation output module is used to input the semantically enhanced image features corresponding to the ultrasound image to be segmented and the semantic features of the current task into the target ultrasound image segmentation model during the model inference stage, so as to obtain the target region segmentation result in the ultrasound image to be segmented.

[0016] Compared with the prior art, this application has at least the following beneficial effects: Based on further analysis and research of existing technical problems, this application recognizes that in multi-task ultrasound image segmentation, there are issues such as insufficient fusion depth of text semantics and visual features, low reliability of semantic modulation, and the susceptibility of task interference and feature drift during multi-task joint training. This application addresses these problems by performing feature encoding on the ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, obtaining shared image features that characterize general visual information and task semantic features that distinguish different segmentation targets. Based on the shared image features, a task probability distribution corresponding to the current ultrasound image is determined, and this distribution is used to adaptively fuse multiple task semantic features. This ensures that the obtained current task semantic features match the actual segmentation target of the input ultrasound image, avoiding semantic confusion between different ultrasound image segmentation tasks. Furthermore, semantic modulation parameters are generated based on the current task semantic features, and semantic modulation reliability parameters are determined by considering image-text compatibility, task routing uncertainty, and segmentation prediction uncertainty. This allows the strength of the task semantics' influence on the shared image features to vary with the degree of image-text matching and the clarity of task judgment. The accuracy and stability of segmentation prediction are dynamically adjusted to enhance visual features related to the current segmentation target when reliability is high, and suppress excessive interference of erroneous task semantics on shared image features when reliability is low, resulting in semantically enhanced image features that balance task specificity and original visual information. Simultaneously, during model training, task memory constraints are constructed based on semantically enhanced image features, historical task feature prototypes for the corresponding task, and corresponding task semantic features. The ultrasound image segmentation model is trained using a joint training objective that includes these task memory constraints, ensuring that target features for different ultrasound image segmentation tasks maintain stable visual representation and semantic orientation, reducing task interference and feature drift in multi-task joint training. Therefore, during model inference, task semantic selection, reliable semantic modulation, and target region segmentation can be automatically completed based on the ultrasound image to be segmented. This addresses the problems of insufficient fusion depth of text semantics and visual features, low reliability of semantic modulation, and task interference and feature drift easily generated in multi-task joint training in existing technologies, improving the accuracy, stability, and robustness of multi-task ultrasound image segmentation. Attached Figure Description

[0017] Figure 1 A flowchart illustrating an uncertainty-aware multi-task ultrasound image segmentation method provided in one embodiment of this application; Figure 2 A schematic diagram of the overall process of an uncertainty-aware multi-task ultrasound image segmentation method provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of an uncertainty-aware semantic modulation module provided in one embodiment of this application; Figure 4This is a schematic diagram of the structure of a text alignment task memory module provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] The following explanations are provided for the English abbreviations appearing in the following embodiments: Feature-wise Linear Modulation (FiLM) is a term used to describe the modulation of linear features. The full English name for Text-Aligned Task Memory is Text-Aligned Task Memory, abbreviated as TATM. The full name of the Segment Anything Model is Segment Anything Model, abbreviated as SAM; The full name of the Text-guided Segment Anything Model is TextSAM; The full name of the Dice similarity coefficient is Dice Similarity Coefficient, abbreviated as DSC. The full English name for Intersection over Union (IoU) is Intersection over Union. The full name of normalized surface distance is Normalized Surface Distance, abbreviated as NSD; The full name of the Gaussian Error Linear Unit is Gaussian Error Linear Unit, abbreviated as GELU; The logical sigmoid function is abbreviated as Sigmoid. The full name of the normalization layer is Layer Normalization, abbreviated as LN.

[0020] like Figure 1 and Figure 2 As shown, this embodiment provides an uncertainty-aware multi-task ultrasound image segmentation method for segmenting ultrasound images of different anatomical regions or different segmentation targets within the same ultrasound image segmentation model. Multiple ultrasound image segmentation tasks may include breast lesion segmentation, thyroid nodule segmentation, cardiac structure segmentation, and left ventricle segmentation, or other ultrasound image segmentation tasks with corresponding task text prompts and segmentation annotation masks.

[0021] First, acquire the ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks. Ultrasound images can originate from ultrasound imaging equipment, publicly available ultrasound image datasets, or authorized clinical ultrasound image databases. Task semantic information describes the anatomical structure or lesion target corresponding to the current ultrasound image segmentation task and can be represented using task text prompts. For example, the task text prompt for the breast lesion segmentation task could be "segment breast lesions in ultrasound images," the task text prompt for the thyroid nodule segmentation task could be "segment thyroid nodules in ultrasound images," the task text prompt for the heart structure segmentation task could be "segment heart structures in ultrasound images," and the task text prompt for the left ventricle segmentation task could be "segment the left ventricle in ultrasound images."

[0022] Feature encoding is performed separately for ultrasound images and task semantic information. Ultrasound images are input into an image encoding network shared by all ultrasound image segmentation tasks, resulting in shared image features. Task semantic information is input into a text encoding network, yielding multiple task semantic features. The shared image features characterize lesion regions, anatomical structures, target boundaries, and image context in the ultrasound image, while the multiple task semantic features characterize the semantic differences between different segmentation targets. Using a shared image encoding network allows multiple tasks to share basic visual feature extraction capabilities, avoiding the need to build a complete image segmentation model for each segmentation target separately.

[0023] Subsequently, the task probability distribution corresponding to the current ultrasound image is determined based on shared image features. Specifically, global feature aggregation is performed on the shared image features to obtain global image features, which are then input into the task routing network. The task routing network outputs the probability that the current ultrasound image belongs to each ultrasound image segmentation task. During the model training phase, the task routing network is trained under supervision using the task identifiers corresponding to the training samples. During the model inference phase, the task routing network automatically outputs the task probability distribution based on the global image features of the ultrasound image to be segmented.

[0024] The semantic features of multiple tasks are fused based on the task probability distribution. Each task probability serves as a semantic selection weight for its corresponding task semantic feature. Weighted aggregation of multiple task semantic features yields the current task semantic feature. Compared to selecting only the single task semantic feature with the highest probability, weighted aggregation can preserve the semantic information of other candidate tasks when there is ambiguity in task judgment, reducing the impact of hard task category switching on subsequent segmentation processes.

[0025] Semantic modulation parameters are generated based on the semantic features of the current task to adjust shared image features. These parameters include at least feature scaling parameters to change the response intensity of different feature dimensions, and feature offset parameters to change the bias states of different feature dimensions. The task probability distribution can also be used to fuse task-level modulation parameters corresponding to different ultrasound image segmentation tasks, so that the final semantic modulation parameters simultaneously carry the task judgment result and task text semantics of the current image.

[0026] Furthermore, semantic modulation reliability parameters are determined based on the image-text compatibility between shared image features and current task semantic features, the task routing uncertainty corresponding to the task probability distribution, and the segmentation prediction uncertainty corresponding to the current ultrasound image. Image-text compatibility is used to represent the degree of matching between the visual content of the current ultrasound image and the semantic features of the current task; task routing uncertainty is used to represent the clarity of the task routing network's judgment of the current task category; and segmentation prediction uncertainty is used to represent the stability of the model's prediction of the spatial location and boundary of the target region in the current ultrasound image.

[0027] The semantic modulation parameters are adjusted based on the semantic modulation reliability parameter, and the adjusted semantic modulation parameters are used to adaptively modulate the shared image features to obtain semantically enhanced image features. When the image-text compatibility is high, the task routing uncertainty is low, and the segmentation prediction uncertainty is low, the modulation intensity of the task semantics on the shared image features is increased; when the image-text matching is insufficient, the task judgment is unclear, or the segmentation prediction is unstable, the modulation intensity of the task semantics on the shared image features is reduced, so that the modulated image features remain close to the state of the original shared image features.

[0028] During the model training phase, foreground features of the target region are extracted based on semantically enhanced image features and corresponding segmentation annotation masks. Task memory constraints are then constructed based on these foreground features, historical task feature prototypes, and current task semantic features. Historical task feature prototypes represent stable foreground feature centers formed during training for the corresponding ultrasound image segmentation task. Task memory constraints include prototype consistency constraints that ensure the foreground features of the target region remain consistent with historical task feature prototypes, and text alignment constraints that ensure the foreground features of the target region remain consistent with the current task semantic features.

[0029] The ultrasound image segmentation model is trained based on a joint training objective that includes task memory constraints, resulting in a target ultrasound image segmentation model. The joint training objective also includes a main segmentation loss, a task routing loss, and a semantic modulation gating regularization term, enabling the model to simultaneously learn ultrasound target segmentation capabilities, task recognition capabilities, reliable semantic modulation capabilities, and task feature preservation capabilities.

[0030] During the model inference phase, shared feature encoding, task routing, task semantic fusion, and uncertainty-aware semantic modulation are performed on the ultrasound image to be segmented. The resulting semantically enhanced image features and the current task semantic features are input into the mask decoding network of the target ultrasound image segmentation model, outputting a segmentation mask for the target region in the ultrasound image to be segmented. The mask decoding network may include a cue encoder and a mask decoder. The cue encoder processes the current task semantic features, and the mask decoder fuses the current task semantic features with the semantically enhanced image features to generate the target region segmentation result. The overall process consists of six sequentially linked stages: multi-part image input, shared encoding, task routing, reliable semantic modulation, task memory, and mask decoding.

[0031] Through the above settings, the task probability distribution can adaptively select task semantics for the current ultrasound image, avoiding semantic confusion between different ultrasound segmentation targets. Image-text compatibility, task routing uncertainty, and segmentation prediction uncertainty jointly control the semantic modulation intensity, which can suppress erroneous enhancement caused by unreliable task semantics. Task memory constraints can maintain a stable foreground representation for each ultrasound image segmentation task. Therefore, this implementation can alleviate the problems of insufficient fusion of text semantics and visual features, low reliability of semantic modulation, and task interference and feature drift in multi-task joint training.

[0032] In one embodiment, multiple sets of ultrasound images corresponding to different anatomical regions or different segmentation targets are acquired. These multiple sets of ultrasound images can cover different anatomical regions such as the breast, thyroid, and heart, and can include both binary and multi-class segmentation tasks. For data formed from ultrasound videos, video frames with pixel-level segmentation annotations can be extracted as ultrasound images; for data stored as single images, the image and its corresponding segmentation annotation mask can be directly read.

[0033] Multiple sets of ultrasound images are resized and normalized to obtain standard ultrasound images. Resizing ensures that the same batch of input images meets the size requirements of the image coding network; normalization adjusts the grayscale range of the image based on pixel distribution while preserving lesion texture, anatomical structure, and target boundary information in the ultrasound images. For datasets with different original sizes, adaptive resizing can be performed separately, rather than requiring all datasets to be resized to the same size.

[0034] In a specific application, four publicly available ultrasound image segmentation datasets—BUSI, DDTI, EchoNet-Dynamic, and CardiacUDA—can be used. These four datasets cover different anatomical regions such as the breast, thyroid, and heart, and can be used to construct multi-task ultrasound image training samples.

[0035] The BUSI dataset comprises 780 breast ultrasound images from 600 female patients aged 25 to 75 years. The average image size is approximately 500×500 pixels. It includes normal, benign, and malignant breast cases, and provides corresponding lesion segmentation annotations. In a specific experiment, normal, benign, and malignant samples were retained; for samples with multiple lesion regions, the segmentation masks for multiple lesions were merged into a single overall mask; all BUSI images were uniformly resized to 256×256 pixels. The segmentation annotation mask for normal samples does not include lesion regions and is used to evaluate the model's ability to suppress false positives in images with missing targets.

[0036] The DDTI dataset includes 99 cases and 134 thyroid ultrasound images, covering types such as thyroiditis, cystic nodules, adenomas, and cancers. A pre-cleaned, cropped, and irrelevant region-removed version of the DDTI dataset can be used, with all images uniformly resized to 256×256 pixels.

[0037] The EchoNet-Dynamic dataset includes 10,030 apical four-chamber echocardiogram videos, providing left ventricular endocardial boundary annotations, primarily corresponding to end-systole and end-diastole. The publicly available annotated frames can be used as image segmentation samples, generating 20,046 ultrasound images and corresponding segmentation annotation masks, while maintaining the original 112×112 pixel image size. Data segmentation is performed based on patient unique identifiers to avoid adjacent frames from the same ultrasound sequence appearing simultaneously in both the training and test sets.

[0038] The CardiacUDA dataset includes echocardiography video data, with each patient corresponding to a parasternal left ventricular long-axis view, a pulmonary artery long-axis view, a left ventricular short-axis view, and an apical four-chamber view. Figure 4 Multiple ultrasound perspectives were used. Video resolution ranged from 800×600 pixels to 1024×768 pixels, with each video segment comprising over 100 frames and covering at least one complete cardiac cycle. Five frames were selected for pixel-level annotation, categorized as left ventricle, right ventricle, left atrium, and right atrium. The manually annotated frames were extracted and uniformly resized to 256×256 pixels, resulting in a cardiac structure segmentation dataset of 2250 images. The five annotated frames in each video segment were distributed across end-systole, end-diastole, and three intermediate phases, rather than using only consecutive frames.

[0039] Five-fold cross-validation can be used on the above dataset, and a patient-level data partitioning strategy can be adopted in each fold to ensure that the images, video frames or labeled samples corresponding to the same patient appear in only one data partition, thereby preventing data leakage caused by similar images of the same patient entering the training set and the test set at the same time.

[0040] Based on the segmentation targets corresponding to each set of standard ultrasound images, task description text is constructed to obtain task text prompts. For example, task text prompts such as "segmenting breast lesions in ultrasound images," "segmenting thyroid nodules in ultrasound images," "segmenting cardiac structures in ultrasound images," and "segmenting the left ventricle in ultrasound images" can be constructed. The task text prompts are not only used for masking decoding, but also serve as semantic priors for subsequent semantic modulation and text alignment task memory.

[0041] Standard ultrasound images, task text prompts, corresponding task identifiers, and segmentation annotation masks are associated to obtain multi-task ultrasound image training samples. Task identifiers are used to supervise the task routing network during training and to retrieve information about the corresponding task from the task semantic feature library and task memory. Segmentation annotation masks are used to calculate the main segmentation loss and extract foreground features of the target region.

[0042] By setting up the above-mentioned data, ultrasound data from different anatomical regions, different original sizes, and different data formats can be organized into tasks. This enables the establishment of a clear correspondence between images, task semantics, task identifiers, and segmentation annotations. This provides a unified data input basis for subsequent shared image encoding, task routing, task semantic fusion, and task memory constraints. At the same time, patient-level data segmentation reduces the risk of data leakage affecting the evaluation results.

[0043] In one embodiment, a standard ultrasound image is input into an image coding network shared by multiple ultrasound image segmentation tasks for feature extraction, resulting in shared image features. The image coding network can be a medical image-adapted SAM image encoder or an image encoder from TextSAM, with multiple ultrasound image segmentation tasks sharing the basic parameters of the image coding network.

[0044] Shared image features can be represented as X. Shared image features X preserve the lesion region, anatomical structure, target boundary, and contextual semantic information in the ultrasound image. Its spatial dimension can be downsampled compared to the original ultrasound image, and its feature dimension is used to carry the shared visual representation for different ultrasound image segmentation tasks.

[0045] Shared image encoding does not mean that different tasks use the exact same final features. Instead, it first extracts common visual information, and then the shared image features are adaptively calibrated according to the current task by the subsequent task routing and semantic modulation processes. This can reduce the cost of deploying multiple models while preserving feature adjustment space for different tasks.

[0046] The task text prompts corresponding to each ultrasound image segmentation task are input into a text encoding network for semantic encoding, resulting in task semantic features that correspond one-to-one with each ultrasound image segmentation task. The text encoding network can be jointly optimized with the image encoding network during training, or a pre-trained text encoder can be used to extract text semantics, and a mapping layer can be used to match the task semantic features with the input dimensions of the subsequent modulation network and prompt encoder.

[0047] Task semantic features are used to represent the semantic differences between different segmentation targets such as "breast lesion", "thyroid nodule", "heart structure" and "left ventricle". Unlike tasks that only use textual prompts in the masking and decoding stage, task semantic features also participate in semantic modulation parameter generation, image-text compatibility calculation and text alignment task memory constraints.

[0048] The semantic features of different tasks are stored according to task identifiers to obtain a task semantic feature library. The task semantic feature library can be represented as T, where each storage location corresponds to an ultrasound image segmentation task and stores the corresponding task semantic features. During model training, the corresponding task semantic features can be read based on the task identifiers of the training samples; during model inference, multiple task semantic features in the task semantic feature library T can be weighted and fused according to the task probability distribution.

[0049] exist Figure 2 In the overall structure shown, each group of ultrasound images is preprocessed and then input into a shared image encoding network to obtain shared image features; multiple task text prompts are input into a text encoding network to obtain task semantic features and form a task semantic feature library; the shared image features are further fed into a task routing network and an uncertainty-aware semantic modulation module, while the task semantic features are simultaneously fed into a semantic modulation module, a text alignment task memory module, and a mask decoding part.

[0050] With the above settings, the shared image coding network can extract general visual information applicable to multiple ultrasound image segmentation tasks, and the task semantic feature library can provide explicit semantic priors corresponding to different targets, so that image visual information and task text information can form stable representations before entering the task routing and semantic modulation process, thus laying the foundation for subsequent cross-modal feature fusion and unified multi-task segmentation.

[0051] In one embodiment, global feature aggregation is performed on the shared image features to obtain the global image features corresponding to the current ultrasound image. Global feature aggregation can employ global average pooling to converge the shared image features in the spatial dimension, obtaining a global visual query vector q that can represent the overall visual content of the current ultrasound image.

[0052] The global features of the image are input into the task routing network for task category prediction. The task routing network can employ a multilayer perceptron, mapping the global image features to task prediction values ​​corresponding to the number of ultrasound image segmentation tasks. Then, the task prediction values ​​are subjected to Softmax probability normalization to obtain the task probability distribution.

[0053] Where R(·) represents the task routing network, q represents the global image features, and p represents the task probability distribution. Each probability value in the task probability distribution corresponds to an ultrasound image segmentation task, and the sum of all probability values ​​is 1.

[0054] During model training, the real task category is determined based on the task identifiers of the training samples, and a task routing loss is used to supervise the task routing network, enabling the network to establish a mapping relationship between global image features and corresponding segmentation tasks. Cross-entropy loss can be used for the task routing loss to improve the consistency between the task probability distribution and the real task category.

[0055] During the model inference phase, it is not necessary to pre-provide the model with a defined task category for the ultrasound image to be segmented. The task routing network automatically generates a task probability distribution based on the global image features of the ultrasound image to be segmented. This task probability distribution is used to determine the segmentation target most likely to correspond to the current ultrasound image, and also serves as a fusion weight for task semantic features and task-level modulation parameters.

[0056] The semantic selection weights of the corresponding task semantic features are used as the probabilities of each task in the task probability distribution. When the probability of a certain task is significantly higher than that of other tasks, the current task semantic features are mainly composed of the semantic features of the task corresponding to that task. When the probabilities of multiple tasks are similar, the corresponding task semantic features are fused according to the probabilities of multiple tasks to avoid incorrect semantic injection caused by directly using a single task semantic when the task judgment is unclear.

[0057] The task probability distribution is also used to calculate task routing uncertainty. When the task probability is concentrated on a single task, it indicates that the task routing decision is relatively clear; when the task probability is dispersed across multiple tasks, it indicates that the task routing network has high task decision uncertainty. This task routing uncertainty is further involved in determining the semantic modulation reliability parameters.

[0058] With the above settings, the task routing network can automatically determine the probability of each candidate segmentation task based on the overall visual content of the ultrasound image itself. The task probability distribution simultaneously supports task semantic selection, task-level modulation parameter fusion, and task routing uncertainty calculation, so that the task recognition result and subsequent feature modulation form a continuous data transmission relationship, and reduce the usage limitations caused by relying on manual specification of task categories.

[0059] like Figure 3 As shown, the uncertainty-aware semantic modulation module includes a semantic branch, a text-image compatibility branch, a segmentation prediction uncertainty branch, a task routing uncertainty branch, a reliability gating section, and a feature calibration section. This embodiment focuses on illustrating the implementation process of the semantic branch. To maintain consistent symbol meaning, Figure 3 The image embedding and calibration features indicated by lowercase x and x′ correspond to the shared image feature X and semantically enhanced image feature X′ mentioned below, respectively.

[0060] Using the probabilities of each task in the task probability distribution as semantic selection weights, the task semantic feature library is processed. The semantic features of the current task are obtained by weighted aggregation of multiple task semantic features:

[0061] Where p represents the task probability distribution, T represents the task semantic feature library, and z_π represents the current task semantic features fused based on the current task probability distribution. This calculation allows task semantic selection to be driven by the actual visual content of the input ultrasound image, rather than always using a fixed task semantic.

[0062] Based on the task probability distribution, the task-level modulation parameters corresponding to different ultrasound image segmentation tasks are weighted and fused to obtain the task modulation parameters. Specifically, learnable task-level feature scaling parameters and task-level feature offset parameters can be set for each ultrasound image segmentation task, and the task-level parameters are weighted and summed using the probability of each task as the fusion weight to obtain the task-level feature scaling parameters and task-level feature offset parameters corresponding to the current ultrasound image.

[0063] The semantic features of the current task are input into the semantic parameter generation network for parameter mapping to obtain semantic residual modulation parameters. For example... Figure 3 As shown, the semantic parameter generation network can sequentially include a normalization layer, a first linear mapping layer, a Gaussian error linear unit (GELU), a second linear mapping layer, and a hyperbolic tangent mapping layer. The normalization layer is used to stabilize the numerical distribution of semantic features for different tasks; the first and second linear mapping layers are used to map the semantic features of the current task to a parameter space corresponding to the number of shared image feature channels; the GELU is used for nonlinear feature mapping; and the hyperbolic tangent mapping layer is used to limit the numerical range of the semantic residual modulation parameters.

[0064] The semantic modulation parameters are obtained by fusing the task modulation parameters and the semantic residual modulation parameters. These semantic modulation parameters include a feature scaling parameter γ and a feature offset parameter β. Some of these parameters are derived from task-level learnable parameters obtained by fusing based on the task probability distribution, while others are derived from the semantic residual modulation parameters generated from the current task's semantic features.

[0065] Task-level learnable parameters are used to preserve the specificity of different ultrasound image segmentation tasks, while semantic residual modulation parameters are used to introduce task text semantics into the shared image feature layer. After the two types of parameters are fused, the text semantics no longer serve only as target cues in the masking and decoding stage, but can directly participate in the task adaptive calibration of shared image features.

[0066] By using the above settings, the task semantic features and task-level modulation parameters are synchronously fused according to the task probability distribution, which can keep the current task semantic features, task modulation parameters and input ultrasound image consistent. At the same time, the combination of task-level parameters and text semantic parameters not only preserves the feature adjustment differences of different tasks, but also allows text semantics to participate deeply in image feature calibration, thereby improving the problem that text semantics only acts on the decoding stage and cannot effectively guide the sharing of visual features.

[0067] In one embodiment, image-text compatibility, segmentation prediction uncertainty, and task routing uncertainty are determined separately, and these three types of information are used together to determine semantic modulation reliability parameters.

[0068] Regarding image and text compatibility, such as Figure 3 As shown, global average pooling is performed on shared image features to obtain visual query features. These visual query features are then mapped to the feature space through a linear mapping layer and a normalization layer. Simultaneously, the semantic features of the current task are normalized to ensure that the mapped visual query features and the semantic features of the current task are in a comparable feature space.

[0069] The cosine similarity between the mapped visual query features and the current task semantic features is calculated to obtain the image-text compatibility s. The higher the image-text compatibility, the more consistent the overall visual content in the current ultrasound image is with the target described by the current task semantic features; the lower the image-text compatibility, the more likely the task semantics corresponding to the task routing result does not match the visual content of the current image.

[0070] To address segmentation prediction uncertainty, coarse segmentation prediction values ​​are generated based on shared image features. These coarse segmentation prediction values ​​can be unnormalized coarse segmentation logits. The coarse segmentation logits are then converted into a coarse segmentation probability map using the Sigmoid function. Finally, pixel prediction entropy is calculated based on the target class probability corresponding to each pixel in the coarse segmentation probability map.

[0071] Pixel prediction entropy represents the uncertainty of the model in determining whether a corresponding pixel belongs to the target region or the background region. When the target class probability of a pixel is close to certainty, the prediction entropy of that pixel is low; when the target class probability of a pixel is in an intermediate state where it is difficult to determine, the prediction entropy of that pixel is high. The segmentation prediction uncertainty *u* is obtained by averaging or other aggregation processing of the pixel prediction entropies in the coarse segmentation probability map.

[0072] For task routing uncertainty, the task routing entropy is calculated based on the probability of each task in the task probability distribution, resulting in the task routing uncertainty h. When the task probability is mainly concentrated on a single task, the task routing entropy is low; when the task probability is relatively evenly distributed among multiple tasks, the task routing entropy is high.

[0073] The reliability features are obtained by concatenating or combining image-text compatibility (s), segmentation prediction uncertainty (u), and task routing uncertainty (h) with other features. These reliability features are then input into a reliability evaluation network, which may include a linear mapping layer and a sigmoid function, to obtain semantic modulation reliability parameters.

[0074] Where η represents the semantic modulation reliability parameter, η_max represents the preset maximum modulation strength, W_g represents the learnable parameter of the reliability assessment network, s represents image-text compatibility, u represents segmentation prediction uncertainty, h represents task routing uncertainty, and [s,u,h] represents the combined result of the three types of reliability information.

[0075] Task probability distribution, image-text compatibility, segmentation prediction uncertainty, and task routing uncertainty reflect, respectively, "which segmentation target the current sample corresponds to," "whether the image matches the task semantics," "whether the current spatial segmentation result is stable," and "whether the task category judgment is clear." Among them, the task probability distribution is used to generate the current task semantic features and task modulation parameters. The other three types of reliability information jointly control the same semantic modulation channel, rather than acting independently on unrelated processing modules.

[0076] When the task probability distribution is clear, the image-text compatibility is high, the segmentation prediction uncertainty is low, and the task routing uncertainty is low, the reliability assessment network outputs a large semantic modulation reliability parameter; as long as any aspect of image-text matching, segmentation prediction, or task routing shows that the current sample is in an unreliable state, the semantic modulation reliability parameter will decrease accordingly.

[0077] Through the above settings, image-text compatibility, segmentation prediction uncertainty, and task routing uncertainty can be jointly evaluated from three different levels: semantic matching, spatial prediction, and task recognition. This avoids the problem that using only a single confidence index cannot identify other sources of failure, thus forming a reliable semantic injection mechanism that enhances multi-source consistency and suppresses multi-source conflicts.

[0078] In one embodiment, the shared image feature X is linearly adjusted at the feature level based on the feature scaling parameter γ and the feature offset parameter β to obtain the semantically adjusted feature γ⊙X+β. Here, the feature scaling parameter γ is used to adjust the response intensity of different feature dimensions in the shared image feature X, and the feature offset parameter β is used to change the bias state of the corresponding feature dimension.

[0079] The semantic modulation features are weighted using the semantic modulation reliability parameter η to obtain the reliable semantic modulation feature η·(γ⊙X+β). Since η is already determined by the preset maximum modulation strength η_max and the output of the reliability evaluation network, the preset maximum modulation strength is not introduced again.

[0080] The reliable semantic adjustment features are residually fused with the original shared image features X to obtain the semantically enhanced image features X′.

[0081] The residual feature modulation process is represented as follows:

[0082] Where X represents shared image features, X′ represents semantically enhanced image features, η represents semantic modulation reliability parameters, γ represents feature scaling parameters, β represents feature offset parameters, and ⊙ represents element-wise multiplication. This residual structure preserves the original SAM image representation while allowing the model to make appropriate adjustments based on task semantics and reliability information. Therefore, the model can achieve more stable and adaptive feature representation in multi-task ultrasound segmentation.

[0083] exist Figure 3 In the structure shown, shared image features, feature scaling parameters, feature offset parameters, and semantic modulation reliability parameters are jointly input into the FiLM calibration part, and semantically enhanced image features are calculated according to the residual formula described above. These semantically enhanced image features are input into the mask decoding network to generate target region segmentation masks; and also into the text alignment task memory module during the model training phase to extract foreground features of the target region.

[0084] When the semantic modulation reliability parameter is large, the influence of semantic adjustment features on the final semantically enhanced image features increases, enabling the task semantics to more fully enhance the visual response related to the current task. When the semantic modulation reliability parameter is small, the final semantically enhanced image features mainly retain the original shared image features, enabling the model to automatically regress from the unreliable task semantic modulation state to a state close to the original visual representation.

[0085] During the model training phase, a semantic modulation gating regularization term can be set to constrain excessively large semantic modulation reliability parameters or excessively strong semantic modulation results, so as to avoid semantic modulation from destroying the basic visual representations already obtained by the image coding network.

[0086] With the above settings, residual modulation introduces task-specific semantics while preserving the original shared image features, so that the model will not completely cover the basic visual information due to the injection of task text semantics; the reliability parameter further controls the actual effect strength of task semantics according to the current sample state, thereby reducing the risk of incorrect task bias, false positive enhancement and boundary missegmentation, and improving the stability of feature expression under different ultrasound image segmentation tasks.

[0087] like Figure 4 As shown, the text alignment task memory module includes a foreground feature extraction part, a drift evaluation part, a text semantic branch, a text alignment part, a task prototype update part, and a memory loss generation part. This embodiment focuses on the foreground feature extraction of the target region, feature drift evaluation, and prototype consistency constraints.

[0088] During model training, the text alignment task memory module receives the semantically enhanced image features X′ and the corresponding segmentation label mask M. The segmentation label mask M is adjusted to match the spatial size of the semantically enhanced image features X′, resulting in the target region feature mask M′. The resized target region feature mask M′ maintains a spatial correspondence with the semantically enhanced image features X′.

[0089] Foreground features of the target region in the semantically enhanced image are aggregated using a target region feature mask to obtain the foreground features of the target region. Foreground feature aggregation can be performed using mask-weighted average pooling, and its calculation process is as follows:

[0090] Where X′ represents semantically enhanced image features, M′ represents the target region feature mask after resizing, f_t represents the target region foreground features of the current sample, ε represents the smoothing term used to prevent the denominator from being zero, and ⊙ represents element-wise multiplication.

[0091] Foreground features are aggregated by using a feature mask of the target region, so that the foreground features of the target region mainly reflect the visual information of the real lesion or target anatomical structure, rather than the background features in the entire ultrasound image.

[0092] The task memory maintains a corresponding historical task feature prototype c_t for each ultrasound image segmentation task. The historical task feature prototype c_t represents the stable foreground feature center gradually accumulated during the model training process for the corresponding task. Based on the task identifier corresponding to the current ultrasound image, the historical task feature prototype c_t for the corresponding ultrasound image segmentation task is read from the task memory.

[0093] Calculate the cosine difference between the foreground features of the target region and the historical task feature prototypes to obtain the feature shift degree:

[0094] Where d_t represents the feature offset of the current target region foreground features relative to the historical task feature prototype, f_t represents the target region foreground features, c_t represents the historical task feature prototype of the corresponding task, and Cosine(·) represents the cosine similarity calculation.

[0095] The feature offset degree is compared with a preset offset threshold. When the feature offset degree is greater than the preset offset threshold, it indicates that there is a significant offset between the foreground features of the current target region and the stable visual representation already formed by the corresponding task, and the prototype consistency constraint is activated; when the feature offset degree is not greater than the preset offset threshold, it indicates that the foreground features of the current target region are within the stable feature range of the corresponding task.

[0096] The prototype consistency constraint is used to reduce the difference between the foreground features of the target region and the prototype features of the historical task, forming the prototype consistency loss L_proto. During model training, the prototype consistency loss L_proto optimizes the parameters to make the foreground features of the target region for the same ultrasound image segmentation task move closer to the corresponding prototype features of the historical task.

[0097] It should be noted that prototype consistency constraints influence subsequent feature extraction results through participation in model parameter optimization, rather than directly modifying the already extracted foreground features of the target region. When a large feature shift is detected, abnormal shift features are suppressed first through prototype consistency constraints, rather than immediately using abnormal shift features to update the historical task feature prototype.

[0098] By using the above settings, the target region foreground features can be extracted from the semantically enhanced image features using the real segmentation annotation mask, which enables the task memory to focus on representing the real lesions or anatomical structures. By using the historical task feature prototype and feature offset degree to identify abnormal feature drift during the training process, the interference of other tasks on the stable visual representation of the current task can be reduced, and the performance decline that occurs in the later stage of multi-task joint training for small sample tasks or small lesion tasks can be alleviated.

[0099] In one embodiment, based on the task identifier corresponding to the current ultrasound image, the corresponding task text feature e_t is read from the task semantic feature library T. The task text feature e_t is a single task semantic feature selected based on the real task identifier, used for text alignment task memory constraints during the training phase; the current task semantic feature z_π is obtained by weighted fusion of multiple task semantic features in the task semantic feature library T using the task probability distribution, and is used for semantic modulation and model inference. The task text feature e_t and the current task semantic feature z_π originate from the same task semantic feature library T, but their acquisition methods and functions differ.

[0100] Foreground features of the target region are input into a semantic projection network for feature space mapping to obtain foreground semantic features. The semantic projection network can employ a lightweight multilayer perceptron to map the foreground features of the target region in visual space to a semantic space that matches the task's semantic features.

[0101] The cosine similarity between the foreground semantic features and the task text features e_t is calculated to obtain the semantic similarity level. Based on the semantic similarity level, a text alignment constraint is constructed, forming the text alignment loss L_align. The text alignment loss L_align is used to constrain the visual target region features to maintain consistency with the corresponding task text semantics, ensuring that the visual prototypes in the task memory not only possess visual stability but also have a clear task semantic orientation.

[0102] Combining the prototype consistency loss and the text alignment loss yields the memory loss corresponding to the task memory constraint:

[0103] Where L_mem represents memory loss, L_proto represents prototype consistency loss, used to constrain the foreground features of the current target region to maintain consistency with the prototype features of the historical task; L_align represents text alignment loss, used to constrain the visual features to align semantically with the task text; and λ_align represents the weight of the text alignment loss. Prototype consistency loss, as a fundamental term of memory loss, has a fixed weight of 1, while λ_align controls the relative contribution of text alignment loss to memory loss.

[0104] The update method for the historical task feature prototype is determined based on the feature offset degree and a preset offset threshold. When the feature offset degree is no greater than the preset offset threshold, it indicates that the foreground features of the current target region are in a stable state. An exponential moving average method is used to fuse the foreground features of the target region and the historical task feature prototype to obtain the updated historical task feature prototype.

[0105] Where c_t represents the historical task feature prototypes before and after the update, f_t represents the foreground features of the current target region, and μ represents the momentum coefficient. The momentum coefficient μ controls the proportion of historical task feature prototypes that retain historical information and absorb current feature information.

[0106] When the feature offset exceeds a preset offset threshold, the historical task feature prototype remains unchanged. The task memory is not immediately updated using the foreground features of the current target region. Instead, prototype consistency constraints and text alignment constraints are used for parameter optimization of the ultrasound image segmentation model. This avoids abnormal offset features from contaminating the already formed stable task prototype.

[0107] The model training process simultaneously optimizes main segmentation performance, task routing capability, semantic modulation reliability, and task memory stability. The overall loss function is:

[0108] Where L represents the overall loss, L_seg represents the main segmentation loss, which is composed of Dice loss and BCE loss; L_task represents the task routing loss, used to supervise the task routing network to correctly identify the task to which the current sample belongs; L_η represents the FiLM gating regularization term constrained by the semantic modulation reliability parameter η, used to suppress excessively strong FiLM modulation; L_mem represents the memory loss generated by the text alignment task memory module; λ_task, λ_η, and λ_mem represent the weight coefficients of the task routing loss, the FiLM gating regularization term, and the memory loss, respectively. L_seg is the main optimization objective, and the other three terms serve as auxiliary supervision or regularization constraints, used to stabilize task routing, constrain FiLM modulation strength, and mitigate task feature drift and forgetting, respectively. The weight of the prototype consistency loss is fixed at 1, and the relative contribution of the text alignment loss to the memory loss is controlled by λ_align.

[0109] The main segmentation loss serves as the primary optimization objective of the model, while the task routing loss, semantic modulation gating regularization term, and memory loss act as auxiliary supervision or regularization constraints. The trainable parameters of the image coding network, task routing network, semantic modulation network, semantic projection network, and mask decoding network are jointly optimized using the overall loss to obtain the target ultrasound image segmentation model.

[0110] During the inference phase, segmentation labels are no longer needed, nor is target region foreground feature extraction and task prototype update performed based on real segmentation labels. Instead, the target region segmentation result is directly output using the trained image encoding, task routing, reliable semantic modulation, and mask decoding capabilities.

[0111] Through the above settings, the prototype consistency constraint can maintain the stability of visual features of the same task, the text alignment constraint can make the visual prototype maintain clear task semantics, and the selective update based on the degree of feature offset can avoid abnormal features from polluting the task memory. The joint training objective enables the four types of objectives—segmentation, task routing, reliable semantic modulation, and task memory—to cooperate with each other, thereby reducing negative transfer and catastrophic forgetting between tasks and improving the continuous training stability and generalization ability of the multi-task ultrasound image segmentation model.

[0112] like Figure 2As shown, this embodiment provides an uncertainty-aware multi-task ultrasound image segmentation system, including a feature encoding module, a task routing module, an uncertainty-aware semantic modulation module, a text alignment task memory module, a model training module, and a segmentation output module. Each module can be implemented by a processor executing program instructions stored in memory, or it can be implemented using hardware circuits with corresponding data processing functions.

[0113] The feature encoding module is used to acquire ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, and to perform feature encoding on the ultrasound images and task semantic information respectively to obtain shared image features and multiple task semantic features. The feature encoding module may include an image preprocessing unit, a shared image encoding network, a text encoding network, and a task semantic feature library.

[0114] The task routing module performs global feature aggregation on shared image features to obtain global image features, and outputs the task probability distribution corresponding to the current ultrasound image based on the global image features. The task routing module also uses the probabilities of each task in the task probability distribution as semantic selection weights to fuse the semantic features of multiple tasks to obtain the semantic features of the current task.

[0115] The uncertainty-aware semantic modulation module generates semantic modulation parameters based on the task probability distribution and the semantic features of the current task, and determines semantic modulation reliability parameters based on image-text compatibility, task routing uncertainty, and segmentation prediction uncertainty. The module performs linear feature adjustment on shared image features based on the semantic modulation parameters, weights the obtained semantically adjusted features using the semantic modulation reliability parameters, and performs residual fusion with the weighted reliable semantically adjusted features and shared image features to obtain semantically enhanced image features.

[0116] The text alignment task memory module is used during model training to extract foreground features of the target region based on semantically enhanced image features and segmentation annotation masks. It reads historical task feature prototypes from the task memory, calculates the degree of feature shift, and constructs prototype consistency constraints and text alignment constraints. The text alignment task memory module also updates historical task feature prototypes when the foreground features of the target region are stable, and keeps them unchanged when there is a significant shift in the foreground features of the target region.

[0117] The model training module is used to construct a joint training objective based on the main segmentation loss, task routing loss, semantic modulation gating regularization term and memory loss, and to optimize the parameters of the trainable network in the system using the joint training objective to obtain the target ultrasound image segmentation model.

[0118] The segmentation output module is used during the model inference phase to input the semantically enhanced image features corresponding to the ultrasound image to be segmented and the semantic features of the current task into the target ultrasound image segmentation model. The segmentation output module may include a cue encoder, a mask decoder, and a segmentation output unit. The cue encoder is used to process the semantic features of the current task, the mask decoder is used to fuse the semantic features of the current task with the semantically enhanced image features, and the segmentation output unit is used to generate a segmentation mask for the target region in the ultrasound image to be segmented.

[0119] The system output can be quantitatively evaluated using metrics such as DSC, IoU, and NSD. Experimental results on four public datasets are shown in Table 1. Table 1

[0120] As shown in Table 1, after adding uncertainty-aware semantic modulation alone, both DSC and NSD on the four datasets are improved compared to direct multi-task joint training of TextSAM. Further adding text alignment task memory further significantly improves tasks with small sample sizes and prone to feature drift, such as BUSI and DDTI, while CardiacUDA and EchoNet also maintain an improving trend. Multidimensional reliability gating mainly improves the reliable semantic modulation of the current samples, while text alignment task memory further alleviates task-level feature drift in multi-task joint training; the two complement each other.

[0121] With the above settings, each functional module can complete the unified segmentation of multiple ultrasound targets such as breast lesions, thyroid nodules, cardiac structures and left ventricle in a single system, reducing the resource cost of deploying multiple segmentation models separately, and improving the segmentation accuracy, stability, robustness and clinical auxiliary analysis value through the complementary effects of reliability gating and task memory.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multi-task ultrasound image segmentation method with uncertainty perception, characterized in that, include: Obtain ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, and perform feature encoding on the ultrasound images and task semantic information respectively to obtain shared image features and multiple task semantic features; Based on the shared image features, the task probability distribution corresponding to the current ultrasound image is determined, and the multiple task semantic features are fused according to the task probability distribution to obtain the current task semantic features; Based on the semantic features of the current task, semantic modulation parameters for adjusting the shared image features are generated, and semantic modulation reliability parameters are determined according to the image-text compatibility between the shared image features and the semantic features of the current task, the task routing uncertainty corresponding to the task probability distribution, and the segmentation prediction uncertainty corresponding to the current ultrasound image. The semantic modulation parameters are adjusted according to the semantic modulation reliability parameters, and the shared image features are adaptively modulated based on the adjusted semantic modulation parameters to obtain semantically enhanced image features; During the model training phase, a task memory constraint is constructed based on the semantically enhanced image features, the historical task feature prototype of the corresponding task, and the corresponding task semantic features obtained from the multiple task semantic features. The ultrasound image segmentation model is then trained based on the joint training objective including the task memory constraint to obtain the target ultrasound image segmentation model. During the model inference stage, the semantically enhanced image features corresponding to the ultrasound image to be segmented and the semantic features of the current task are input into the target ultrasound image segmentation model to obtain the target region segmentation result in the ultrasound image to be segmented.

2. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 1, characterized in that, The acquisition of ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks includes: Multiple sets of ultrasound images corresponding to different anatomical regions or different segmentation targets are acquired, and the multiple sets of ultrasound images are resized and standardized to obtain standard ultrasound images. Based on the segmentation targets corresponding to each set of standard ultrasound images, task description text is constructed to obtain task text prompts corresponding to different ultrasound image segmentation tasks. The standard ultrasound image, the task text prompt, the corresponding task identifier, and the segmentation label mask are associated to obtain multi-task ultrasound image training samples.

3. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 2, characterized in that, The feature encoding of the ultrasound image and the task semantic information respectively includes: The standard ultrasound image is input into an image coding network shared by multiple ultrasound image segmentation tasks for feature extraction, resulting in shared image features used to characterize the target region and image context. The task text prompts corresponding to each ultrasound image segmentation task are input into a text encoding network for semantic encoding to obtain task semantic features that correspond one-to-one with each ultrasound image segmentation task. The task semantic features are stored according to the task identifier to obtain a task semantic feature library for subsequent task semantic selection.

4. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 3, characterized in that, Determining the task probability distribution corresponding to the current ultrasound image based on the shared image features includes: Global feature aggregation is performed on the shared image features to obtain the global image features corresponding to the current ultrasound image; The global features of the image are input into the task routing network to predict the task category, thereby obtaining the task prediction values ​​of the current ultrasound image for each ultrasound image segmentation task. The predicted task values ​​are subjected to probability normalization to obtain the task probability distribution, and each task probability in the task probability distribution is used as the semantic selection weight of the corresponding task semantic features.

5. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 4, characterized in that, The step of fusing the semantic features of the multiple tasks according to the task probability distribution and generating the semantic modulation parameters based on the current task semantic features includes: The semantic selection weights are used to perform weighted aggregation on multiple task semantic features in the task semantic feature library to obtain the current task semantic features. Based on the task probability distribution, the task-level modulation parameters corresponding to different ultrasound image segmentation tasks are weighted and fused to obtain the task modulation parameters. The semantic features of the current task are input into the semantic parameter generation network for parameter mapping to obtain semantic residual modulation parameters; The task modulation parameters and the semantic residual modulation parameters are fused to obtain the semantic modulation parameters, which include feature scaling parameters and feature offset parameters.

6. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 1, characterized in that, The determination of semantic modulation reliability parameters includes: Global feature mapping is performed on the shared image features, and the similarity between the mapped image features and the semantic features of the current task is calculated to obtain the image-text compatibility. A coarse segmentation probability map is generated based on the shared image features. The pixel prediction entropy is calculated according to the target category probability corresponding to each pixel in the coarse segmentation probability map. The pixel prediction entropy is then aggregated to obtain the segmentation prediction uncertainty. The task routing entropy is calculated based on the probability of each task in the task probability distribution, and the task routing uncertainty is obtained. The image-text compatibility, the segmentation prediction uncertainty, and the task routing uncertainty are combined to obtain reliability features. These reliability features are then input into a reliability evaluation network. Based on the output of the reliability evaluation network and a preset maximum modulation intensity, the semantic modulation reliability parameters are obtained.

7. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 6, characterized in that, The step of adjusting the semantic modulation parameters according to the semantic modulation reliability parameters, and adaptively modulating the shared image features based on the adjusted semantic modulation parameters, includes: Based on the feature scaling parameter and feature offset parameter in the semantic modulation parameters, the shared image features are linearly modulated to obtain semantically modulated features; The semantic modulation features are weighted using the semantic modulation reliability parameters to obtain reliable semantic modulation features; The reliable semantic adjustment features are residually fused with the shared image features to obtain the semantically enhanced image features.

8. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 1, characterized in that, The construction of task memory constraints based on the semantically enhanced image features and the corresponding historical task feature prototypes includes: During the model training phase, the segmentation label mask corresponding to the current ultrasound image is adjusted to a feature size that matches the semantically enhanced image features to obtain the target region feature mask; Based on the target region feature mask, foreground features of the target region in the semantically enhanced image features are aggregated to obtain the target region foreground features; Based on the task identifier corresponding to the current ultrasound image, retrieve the historical task feature prototype of the corresponding ultrasound image segmentation task from the task memory; Calculate the feature difference between the foreground features of the target region and the historical task feature prototype to obtain the feature offset degree; The feature offset degree is compared with a preset offset threshold. When the feature offset degree is greater than the preset offset threshold, a prototype consistency constraint is activated to constrain the foreground features of the target region to remain consistent with the prototype features of the historical task.

9. The uncertainty-aware multi-task ultrasound image segmentation method according to claim 8, characterized in that, The step of constructing task memory constraints based on corresponding task semantic features and training the ultrasound image segmentation model based on the task memory constraints includes: Based on the task identifier corresponding to the current ultrasound image, the corresponding task semantic features are obtained from the plurality of task semantic features; The foreground features of the target region are input into a semantic projection network for feature space mapping to obtain foreground semantic features; Calculate the semantic similarity between the foreground semantic features and the corresponding task semantic features, and construct text alignment constraints based on the semantic similarity. The prototype consistency constraint and the text alignment constraint are combined to obtain the task memory constraint; When the feature offset is not greater than a preset offset threshold, the foreground features of the target region and the historical task feature prototype are fused using an exponential moving average method to obtain an updated historical task feature prototype. When the feature offset is greater than the preset offset threshold, the historical task feature prototype remains unchanged, and the task memory constraint is used for parameter optimization of the ultrasound image segmentation model.

10. A multi-task ultrasound image segmentation system with uncertainty perception, characterized in that, include: The feature encoding module is used to acquire ultrasound images and task semantic information corresponding to multiple ultrasound image segmentation tasks, and to perform feature encoding on the ultrasound images and task semantic information respectively to obtain shared image features and multiple task semantic features. The task routing module is used to determine the task probability distribution corresponding to the current ultrasound image based on the shared image features, and to fuse the multiple task semantic features according to the task probability distribution to obtain the current task semantic features. An uncertainty-aware semantic modulation module is used to generate semantic modulation parameters based on the semantic features of the current task, determine semantic modulation reliability parameters according to image-text compatibility, task routing uncertainty and segmentation prediction uncertainty, and adjust the semantic modulation parameters according to the semantic modulation reliability parameters to adaptively modulate the shared image features to obtain semantically enhanced image features. The text alignment task memory module is used to construct task memory constraints during the model training phase based on the semantically enhanced image features, the historical task feature prototypes of the corresponding task, and the corresponding task semantic features obtained from the multiple task semantic features. The model training module is used to train the ultrasound image segmentation model based on the joint training objective including the task memory constraint, so as to obtain the target ultrasound image segmentation model. The segmentation output module is used to input the semantically enhanced image features corresponding to the ultrasound image to be segmented and the semantic features of the current task into the target ultrasound image segmentation model during the model inference stage, so as to obtain the target region segmentation result in the ultrasound image to be segmented.

Images