Fetal ultrasound image segmentation method, system and product based on image segmentation model

Abstract

The invention discloses a fetal ultrasound image segmentation method, system and product based on an image segmentation model, and the method comprises the steps: firstly, carrying out the fine adjustment of a feature self-adaptive disturbance module, initializing standard normal Gaussian noise, calculating the mean value and variance of features through normalization, aligning the noise and the style of the features, calculating the weight of the features through an activation function, and carrying out the recognition of the features; and generating feature-adaptive interference noise. Then, a set of learnable Tokens is introduced through an instance perception prompt generation module, interaction with image features is carried out through a cross attention mechanism, and generated interaction features and the Tokens serve as dense and sparse prompts to be input into a subsequent decoding network. And finally, based on a Token noise perturbation algorithm, modeling a Token internal relationship to generate a perturbation weight, and interfering the Token in cooperation with Gaussian noise to prevent over-fitting between the Token and a specific instance. According to the method, the large model is finely adjusted from noise, the general segmentation capability of the model in an unknown domain is improved, and the problem of domain generalization fetal ultrasound image segmentation is solved from a new angle.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence applied to medical ultrasound image segmentation, and relates to a fetal ultrasound image segmentation method, system and product, in particular to a fetal ultrasound image generalization segmentation method, system and product based on robust fine-tuning segmentation large model. BACKGROUND

[0002] Ultrasound imaging plays a key role in medical diagnosis and treatment intervention during pregnancy due to its non-invasive, safe and easy-to-obtain characteristics. In recent years, a large number of deep learning-based ultrasound image segmentation models have been proposed and have shown significant application potential. However, when these models are applied to data from other hospitals or centers, their performance often decreases, which brings inconvenience to clinical actual use. The field generalization segmentation of fetal ultrasound images aims to train a model on a single source domain and achieve robust segmentation prediction in an unseen fetal ultrasound image domain. Traditional generalization methods mainly rely on style transformation or regularization strategies, but ultrasound images themselves have limitations in quality, usually manifested as low resolution, poor contrast and low signal-to-noise ratio, which leads to unsatisfactory application results of traditional methods. In recent years, with the rise of visual large models, the strong generalization ability inherited by large models through large-scale pre-training images provides a new paradigm for the domain generalization segmentation of fetal ultrasound images. Currently, various methods have attempted to apply the segmentation large model SAM to downstream tasks such as medical image segmentation and have achieved significant performance. However, SAM still has difficulties in processing low-quality fetal ultrasound images, and there is currently a lack of general segmentation solutions designed specifically for fetal ultrasound images. SUMMARY

[0003] To solve the technical problem of the lack of general segmentation solutions designed specifically for fetal ultrasound images in the prior art, the application provides a fetal ultrasound image generalization segmentation method, system and product based on robust fine-tuning segmentation large model.

[0004] The technical scheme of the method of the application is as follows: a fetal ultrasound image segmentation method based on an image segmentation model, the fetal ultrasound image obtained is subjected to image segmentation using a trained image segmentation model to obtain a segmented fetal ultrasound image; The image segmentation model is composed of an image encoder, a prompt encoder and a decoder. The image encoder is provided with a fine-tuning feature adaptive perturbation module before the adapter structure; the fine-tuning feature adaptive perturbation module includes a feature noise alignment submodule, a noise adaptability submodule, and a feature interference submodule; the feature noise alignment submodule is used to align noise and features through statistical information of characteristics; the noise adaptability submodule is used to perform feature adaptive weight control on the aligned noise; the feature interference submodule is used to interfere with the fine-tuning features during training using feature adaptive noise; The prompt encoder automatically generates a prompt module based on tokens, and the tokens interact with the image to perceive and automatically generate feature prompts that are most suitable for the current image. It includes a prompt generation module and a prompt interaction submodule. The prompt generation module is used to introduce a set of learnable tokens as subsequent decoder prompts. The prompt interaction submodule is used to perceive and refine the tokens and instance-level image features.

[0005] As a preference, the feature noise alignment submodule first randomly initializes a standard distributed Gaussian noise , and then calculate the current fine-tuning features by normalization The mean and variance , get its statistical information, and then through back normalization, multiply the standard Gaussian noise by the variance and add the mean to get the noise aligned with the feature style.

[0006] As an advantage, the noise adaptation submodule first calculates the fine-tuning feature by the sigmoid function , get the weight based on the relative size of the features , and then by weight Multiply the style-aligned noise to control the noise interference.

[0007] Preferably, the feature interference submodule includes a dimensionality reduction linear layer to compress a large vector into a small vector; a nonlinear activation function; and a dimensionality increase linear layer to expand the compressed small vector back to the original dimension.

[0008] Preferably, the prompt generation submodule uses a set of learnable Tokens. For each image embedding ,in, Indicates Token, represents the field of real numbers, is the number of tokens, C is the feature dimension size, H and W represent the height and width of the feature; Token is used as a query through the cross attention mechanism to calculate the refined features, which are used as dense prompts , also serves as a sparse hint , provide supplementary information.

[0009] As a preferred, the prompt interaction module calculates the similarity between each Token and other Tokens to determine the perturbation strength , finally apply the perturbation strength to the standard Gaussian noise, and then apply the noise to the Token, output To prevent overfitting; ;

[0010] where, Token, is the number of Tokens, C is the feature dimension size, is the identity matrix, denotes the formula sum by row.

[0011] As a preferred, the image encoder outputs the image features Imageembedding and the prompt encoder outputs the prompt features prompt tokens, which are added pixel by pixel and then input into the self-attention mechanism layer, and then the image features Imageembedding are fused and input into the first Token to image cross-attention layer and the first multi-layer perceptron mlp, and finally the image features Imageembedding are fused and input into the image to Token cross-attention layer; The output of the image to Token cross-attention layer is fused with the output of the first multi-layer perceptron mlp, and then input into the second Token to image cross-attention layer, and then output through the second multi-layer perceptron mlp. The IoU prediction dedicated Token; The output of the second Token to image cross-attention layer is fused with the output of the image to Token cross-attention layer after being processed by a 2-layer convolutional layer, and then the mask is output.

[0012] As a preferred, the trained image segmentation model, in the training process, the loss function used is the joint loss of Crossentropy loss segmentation loss and Diceloss segmentation loss ;

[0013] where, is a balance hyperparameter; the initial learning rate is set to 5e-4, and the AdamW optimizer is used, and the weight decay coefficient is set to 0.1. The hyperparameter N of the Token number is set to 50.

[0014] The technical scheme adopted by the system of the present application is: an image segmentation model-based fetal ultrasound image segmentation system, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the image segmentation model-based fetal ultrasound image segmentation method.

[0015] The technical scheme adopted by the system of the present application is: an image segmentation model-based fetal ultrasound image segmentation product, comprising computer program instructions, characterized in that: when the computer program instructions are run on a computer, the computer executes any one of the image segmentation model-based fetal ultrasound image segmentation methods.

[0016] Compared with the prior art, the beneficial effects of the present application include: the present application is different from the conventional image domain generalization method based on style transformation or regularization. The method based on noise robust fine-tuning of large models not only can make full use of the segmentation large model with strong generalization ability to downstream tasks, but also provides a new fine-tuning paradigm. In the model training, fine-designed noise is added to promote the model to improve its anti-interference ability and stimulate the model to perform well in the case of serious noise interference, thereby improving the robustness of the model and its generalization ability in the unseen domain, and achieving significant improvement in fetal ultrasound image generalization segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0017] The technical scheme of the present application is further described below using examples and specific embodiments. In addition, some drawings are also used in the process of describing the technical scheme. For those skilled in the art, other drawings and the intent of the present application can also be obtained from these drawings without creative labor.

[0018] Figure 1 The image segmentation model network structure diagram of the present application embodiment is shown in the figure. Figure 2 The decoder network structure diagram of the present application embodiment is shown in the figure. Figure 3 The multi-center fetal ultrasound image schematic diagram obtained in the experiment of the present application embodiment is shown in the figure. DETAILED DESCRIPTION

[0019] In order to facilitate those skilled in the art to understand and implement the present application, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0020] See Figure 1The embodiment provides a fetal ultrasound image segmentation method based on an image segmentation model, fetal ultrasound images are acquired, image segmentation is performed on the fetal ultrasound images by using a trained image segmentation model, and segmented fetal ultrasound images are obtained; The image segmentation model comprises an image encoder, a prompt encoder and a decoder; The image encoder is provided with a fine-tuning feature adaptive disturbance module in front of an adapter structure; the fine-tuning feature adaptive disturbance module comprises a feature noise alignment sub-module, a noise adaptability sub-module and a feature interference sub-module; the feature noise alignment sub-module is used for aligning noise with features by using statistical information of the features; the noise adaptability sub-module is used for performing feature-adaptive weight control on the aligned noise; and the feature interference sub-module is used for interfering fine-tuning features with the feature-adaptive noise during training. The prompt encoder is provided with a Token-based prompt automatic generation module, Tokens and images are interactively perceived, and the most suitable feature prompts for current images are automatically generated; the prompt automatic generation module comprises a prompt generation sub-module and a prompt interaction sub-module; the prompt generation sub-module is used for introducing a group of learnable Tokens as subsequent decoder prompts; and the prompt interaction sub-module is used for performing attention perception and refinement on Token and image features at an instance level.

[0021] In an implementation, the fine-tuning feature adaptive disturbance module is based on the image encoder; because the parameter quantity of a segmentation large model is huge and the training cost is high, the image encoder of the large model is fine-tuned by using an adapter structure in the embodiment, a small amount of parameters can be used for training, and the segmentation large model can be migrated to a downstream medical task; the feature adaptive disturbance proposed on the basis of the adapter fine-tuning structure in the embodiment is to interfere with the intermediate fine-tuning features by using adaptive noise, and the purpose is to enable the model to maintain good robustness during fine-tuning, thereby improving the generalization performance.

[0022] The feature noise alignment module is used for aligning noise with features by using statistical information of the features; firstly, a standard distribution Gaussian noise is randomly initialized Secondly, the mean and the variance of the current fine-tuning feature are calculated by normalization , the statistical information is obtained, then the standard Gaussian noise is multiplied by the variance and added to the mean by denormalization, and the noise aligned with the feature style is obtained.

[0023] The noise adaptability module is used for performing feature-adaptive weight control on the style-aligned noise; in the embodiment, the interference degree of the noise disturbed by the feature self-adaptive control is first calculated by using a sigmoid function , to obtain a weight based on the relative size of the features Then, the weight is multiplied by the noise after style alignment to control the interference degree of the noise.

[0024] The feature interference module is configured to interfere with the fine-tuning features in the training by the feature-adaptive noise; in this embodiment, the designed feature-adaptive noise is added to the current fine-tuning features to perform noise interference, and then input into the adapter fine-tuning structure; the adapter structure is a small neural network module, mainly including three parts: 1. a dimension reduction linear layer, which compresses a large vector in the original model into a small vector; 2. a nonlinear activation function; and 3. a dimension expansion linear layer, which expands the compressed small vector back to the original dimension In an embodiment, the instance-aware prompt generation module is improved based on a prompt encoder; the original SAM prompt needs to manually submit appropriate points, boxes, and masks, which is too costly for medical segmentation images; therefore, a Token-based automatic prompt generation module is proposed, which interacts with images based on Tokens to automatically generate the most suitable feature prompts for the current image.

[0025] The prompt generation submodule introduces a set of learnable Tokens as subsequent decoder prompts; the Tokens interact with instances, optimize image embeddings, and generate sufficient prompts to guide the subsequent decoder to perform segmentation. Through a set of learnable Tokens, for each image embedding wherein, represents a Token, represents a real number field, is the number of Tokens, C is the feature dimension size, H and W represent the height and width of the feature; the Token is taken as a query through cross-attention mechanism to calculate refined features, which are used as dense prompts which are also used as sparse prompts to provide supplementary information.

[0026] To further improve the performance of the module, considering that the learnable parameters may cause overfitting, this embodiment adopts a more effective Token and instance decoupling strategy; through the prompt interaction module, the Token and the instance-level image features are subjected to attention perception and refinement; first, the similarity between each Token and other Tokens is calculated to determine the perturbation strength Then, the perturbation strength is applied to the standard Gaussian noise, and the noise is applied to the Token to output to prevent overfitting; ;

[0027] where, is Token, is the number of Token, C is the feature dimension size, is the identity matrix, denotes the formula sum by row.

[0028] See Figure 2 In an embodiment, the decoder, the image feature Imageembedding output by the image encoder and the prompt feature prompt tokens output by the prompt encoder, are added pixel by pixel and then input into a self-attention mechanism layer, and then the output of the self-attention mechanism layer is fused with the image feature Imageembedding and input into a first Token-to-image cross-attention layer and a first multi-layer perceptron (MLP), and finally the output of the first multi-layer perceptron (MLP) is fused with the image feature Imageembedding and input into an image-to-Token cross-attention layer; the output of the image-to-Token cross-attention layer is fused with the output of the second multi-layer perceptron (MLP) and then input into a second Token-to-image cross-attention layer, and then the output of the second Token-to-image cross-attention layer is input into a third multi-layer perceptron (MLP) and then the output of the third multi-layer perceptron (MLP) is fused with the output of the image-to-Token cross-attention layer after being processed by a 2-layer convolutional layer to output a mask.

[0029] Figure 2 In the formula, Image embedding is the image feature, prompt tokens is the prompt feature, outputtokenper mask is the mask corresponding to each feature, loU outputtoken is the IoU prediction dedicated Token, dotproductper mask is the mask dot product, masks is the output mask, loU scores is the mask confidence score; image totoken attn. is the image-to-Token cross-attention layer, mlp is the multi-layer perceptron, token to image attn. is the Token-to-image cross-attention layer, self attn. is the self-attention layer, and tokento image attn. is the Token-to-image cross-attention layer.

[0030] In an embodiment, the trained image segmentation model, in the training process, adopts a joint loss of Crossentropy loss segmentation loss and Diceloss segmentation loss .

[0031] wherein, is a balance hyperparameter; the specific experimental setting is 0.8. In the fine-tuning of the adapter structure in this embodiment, the proportion rank of compressed features is set to 4. The initial learning rate for training is set to 5e-4, and the AdamW optimizer is used, and the weight decay coefficient is set to 0.1. The hyperparameter N of the token number is set to 50.

[0032] The embodiment also provides a fetal ultrasound image segmentation system based on an image segmentation model, including: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the fetal ultrasound image segmentation method based on the image segmentation model.

[0033] The embodiment also provides a fetal ultrasound image segmentation product based on an image segmentation model, including computer program instructions, which, when running on a computer, cause the computer to execute the fetal ultrasound image segmentation method based on the image segmentation model.

[0034] The following further illustrates the present application through experiments.

[0035] The experimental data includes fetal ultrasound images of three medical centers, denoted as A, B and C, respectively, with 4123, 5392 and 5297 fetal ultrasound images and corresponding GT masks.

[0036] The experiment respectively trains one domain as the source domain, and tests on the remaining two domains. In the experiment, Dice Coefficient (Dice), mean Intersection over Union (mIoU), Average Surface Distance (ASD) and Hausdorff Distance (HD) are used as evaluation criteria.

[0037]

[0038]

[0039]

[0040] wherein, A is a predicted mask, and B is a real mask.

[0041] The experimental results are shown in the following Figure 3 and Table 1: Table 1

[0042] The experimental results show that the method of the application can significantly improve the generalization segmentation performance of the large segmentation model for ultrasound images.

[0043] The application provides a noise robust fine-tuning large segmentation model method, system and product in a fetal ultrasound image generalization segmentation task. In the training process, fine design noise is introduced to interfere with the intermediate features and instance-level generated prompts. By fine-tuning a large segmentation model with strong generalization ability, the robustness and anti-interference ability of the model are improved, thereby improving the generalization ability.

[0044] It should be understood that the above-described embodiments are part of the embodiments of the application, rather than all the embodiments. In addition, the technical features of each embodiment or a single embodiment provided by the application can be combined with each other to form a feasible technical solution, and such combination is not restricted by the order of steps and / or structure composition mode, but must be based on the realization by a person of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be realized, it is considered that such combination of technical solutions does not exist and is not within the protection scope of the application.

[0045] It should be understood that the above description of the preferred embodiments is more detailed, and therefore should not be considered as limiting the scope of patent protection of the application. A person of ordinary skill in the art can make substitutions or modifications under the inspiration of the application without departing from the scope of protection claimed by the application. All substitutions or modifications fall within the scope of protection of the application, and the scope of protection claimed by the application should be subject to the appended claims.

Claims

1. A fetal ultrasound image segmentation method based on an image segmentation model, characterized by: The acquired fetal ultrasound image is segmented using the trained image segmentation model to obtain a segmented fetal ultrasound image; The image segmentation model consists of an image encoder, a hint encoder and a decoder; The image encoder is provided with a fine-tuning feature adaptive perturbation module before the adapter structure; The fine-tuning feature adaptive perturbation module, It includes a feature noise alignment submodule, a noise adaptation submodule, and a feature interference submodule; It includes a feature noise alignment submodule for aligning noise with features through feature statistics; The noise adaptability submodule is used to perform feature adaptive weight control on the aligned noise; The feature interference submodule is used for interfering the fine-tuning features with feature adaptive noise during training; The prompt encoder automatically generates a prompt module based on the token, and the token interacts with the image to automatically generate the most suitable feature prompt for the current image; It includes a prompt generation module and a prompt interaction submodule; the prompt generation module is used to introduce a set of learnable tokens as subsequent decoder prompts; The prompt interaction submodule is used to perceive and refine the image features at the token and instance levels.

2. The fetal ultrasound image segmentation method based on the image segmentation model according to claim 1, characterized in that: The feature noise alignment submodule first randomly initializes a standard distributed Gaussian noise , and then calculate the current fine-tuning features by normalization The mean and variance , get its statistical information, and then through back normalization, multiply the standard Gaussian noise by the variance and add the mean to get the noise aligned with the feature style.

3. The fetal ultrasound image segmentation method based on the image segmentation model according to claim 1, characterized in that: The noise adaptation submodule first calculates the fine-tuning feature through the sigmoid function , get the weight based on the relative size of the features , and then by weight Multiply the style-aligned noise to control the noise interference.

4. The fetal ultrasound image segmentation method based on an image segmentation model according to claim 1, wherein: The feature interference submodule includes a dimensionality reduction linear layer to compress a large vector into a small vector; a nonlinear activation function; and a dimensionality increase linear layer to expand the compressed small vector back to the original dimension.

5. The fetal ultrasound image segmentation method based on the image segmentation model according to claim 1, characterized in that: The prompt generation submodule uses a set of learnable tokens. , for each image embedding ,in, Indicates Token, represents the field of real numbers, is the number of tokens, C is the feature dimension size, H and W represent the height and width of the feature; Token is used as a query through the cross attention mechanism to calculate the refined features, which are used as dense prompts , also serves as a sparse hint , providing supplementary information.

6. The fetal ultrasound image segmentation method based on an image segmentation model according to claim 1, wherein: The prompt interaction module calculates the similarity between each token and other tokens to determine the disturbance intensity. Finally, the perturbation intensity is applied to the standard Gaussian noise, and the noise is applied to the Token, output To prevent overfitting; ； in, For Token, is the number of tokens, C is the feature dimension size, is the identity matrix, Indicates row summation.

7. The fetal ultrasound image segmentation method based on an image segmentation model according to claim 1, wherein: The decoder, the image feature Imageembedding output by the image encoder and the prompt feature prompt tokens output by the prompt encoder are added pixel by pixel and input into the self-attention mechanism layer, and then fused with the image feature Imageembedding and input into the first Token to image cross attention layer and the first multi-layer perceptron mlp, and finally fused with the image feature Imageembedding and input into the image to Token cross attention layer; the image to Token cross attention layer output is fused with the first multi-layer perceptron mlp output and input into the second Token to image cross attention layer, and then outputs the IoU prediction special Token after passing through the second multi-layer perceptron mlp; the second Token to image cross attention layer output passes through the third multi-layer perceptron mlp, and is fused with the image to Token cross attention layer output after being processed by 2 layers of convolution layers to output the mask.

8. The fetal ultrasound image segmentation method based on an image segmentation model according to any one of claims 1 to 7, characterized in that: The trained image segmentation model uses the Crossentropy loss loss function during training. and Diceloss segmentation loss joint losses; in, To balance hyperparameters, the initial learning rate for training was set to 5e-4, and the AdamW optimizer was used with a weight decay coefficient of 0.

1. The hyperparameter N for the number of tokens was set to 50.

9. A fetal ultrasound image segmentation system based on an image segmentation model, comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the fetal ultrasound image segmentation method based on the image segmentation model as described in any one of claims 1 to 8.

10. A fetal ultrasound image segmentation product based on an image segmentation model, comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is enabled to execute the fetal ultrasound image segmentation method based on an image segmentation model according to any one of claims 1 to 8.

Images