Breast ultrasonic image segmentation method and device and computer equipment

Abstract

The invention relates to the technical field of image processing, in particular to a breast ultrasound image segmentation method and device and computer equipment, and the method comprises the steps: obtaining a breast ultrasound image; inputting the breast ultrasound image into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training a YOLO module by a first training sample set, and the image segmentation model is obtained by training an initial image segmentation model by a second training sample set. According to the method, the YOLO-based automatic prompt generation model is introduced, automatic prompt without human interaction is realized by utilizing the advantages of the YOLO-based automatic prompt generation model in prompt box generation, the automation degree of breast ultrasound image segmentation is greatly improved, a rapid and accurate segmentation means is provided for breast cancer screening and diagnosis, and the method has remarkable clinical application value.

Description

Technical Field

[0001] The present specification relates to the field of image processing technology, and in particular to methods, devices and computer equipment for segmenting breast ultrasound images. Background Art

[0002] In medical image analysis, accurate segmentation of breast cancer is of great significance for early diagnosis and effective treatment. Breast ultrasound images have become an important tool for breast cancer screening and diagnosis due to their non-invasive and safe advantages. However, due to the low contrast, complex morphology and fuzzy boundaries of breast ultrasound images, it is extremely challenging to achieve automatic and accurate segmentation of lesion areas.

[0003] In recent years, deep learning-based segmentation methods have made significant progress in the field of medical image segmentation. For example, U-Net and its variants train data in an end-to-end framework through convolutional neural networks and are widely used in medical image segmentation. Such methods usually rely on a large amount of labeled data and refined network structure adjustments to improve segmentation accuracy. However, for the accurate segmentation of complex lesion areas and the improvement of generalization ability, traditional methods such as U-Net still face limitations in terms of boundary detail preservation and robustness to different target morphologies. In addition, training large medical image segmentation models requires the support of a large amount of labeled data, which may make it difficult for the model to achieve accurate segmentation in small sample cases, especially for low-contrast breast lesion areas.

[0004] To overcome these problems, segmentation methods based on large models have gradually emerged. Such methods usually use pre-trained weights and transfer learning to integrate a large amount of image knowledge from different fields to achieve fine-tuning for specific tasks. For example, models such as Segment Anything Model (SAM) can show high accuracy in a wide range of segmentation tasks through user-provided prompts (such as annotation points or boxes), further expanding the application potential of medical image segmentation. However, the application of large models based on SAM in medical image segmentation is still insufficient, especially in achieving full automation and reducing manual interaction prompts, which still needs further exploration and optimization. Summary of the invention

[0005] In order to solve the problems in the prior art of automated application and manual interactive prompting of large SAM-based models in medical image segmentation, this specification provides a breast ultrasound image segmentation method, apparatus and computer equipment.

[0006] An embodiment of the present specification provides a method for segmenting a breast ultrasound image, the method comprising: acquiring a breast ultrasound image; inputting the breast ultrasound image into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training a YOLO module with a first training sample set, and the image segmentation model is obtained by training an initial image segmentation model with a second training sample set.

[0007] According to one aspect of an embodiment of the present specification, the image segmentation model includes: an image encoder, a mask decoder and a prompt encoder, the prompt generation model is connected to the prompt encoder, the prompt encoder is used to convert the prompt features output by the prompt generation model into feature vectors, the image encoder is used to generate an image potential representation of a breast ultrasound image, and input the image potential representation to the mask decoder.

[0008] According to one aspect of the embodiments of this specification, the mask decoder is used to combine the feature vector with the potential representation of the image and then perform mask processing.

[0009] According to one aspect of an embodiment of the present specification, a prompt generation model is trained in the following manner: a first training sample set is obtained, the first training sample set including: a breast ultrasound sample image and a label bounding box corresponding to a lesion area in the breast ultrasound sample image; the breast ultrasound sample image is input into a YOLO module to obtain a predicted bounding box of the breast ultrasound sample image by the YOLO module; a loss function of the YOLO module is calculated based on the predicted bounding box and the label bounding box; the YOLO module is iteratively updated based on the loss function to construct a prompt generation model.

[0010] According to one aspect of an embodiment of the present specification, calculating the loss function of the YOLO module according to the predicted bounding box and the label bounding box includes: determining a center coordinate loss function according to the center coordinates of the label bounding box and the center coordinates of the predicted bounding box; determining a dimension loss function according to the size of the label bounding box and the size of the predicted bounding box; determining a lesion category loss function according to the lesion category indicated by the label bounding box and the lesion category indicated by the predicted bounding box; determining a confidence loss function according to the true confidence corresponding to the label bounding box and the predicted confidence corresponding to the predicted bounding box; determining the loss function of the YOLO module according to the center coordinate loss function, the dimension loss function, the lesion category loss function and the confidence loss function.

[0011] According to one aspect of an embodiment of the present specification, an image segmentation model is trained in the following manner: a second training sample set is obtained, the second training sample set including: a breast ultrasound sample image and a mask label corresponding to the breast ultrasound sample image; the breast ultrasound sample image is input into an initial image segmentation model to obtain a predicted mask of the breast ultrasound sample image by the initial image segmentation model; a loss function is constructed based on the mask label and the predicted mask; and the initial image segmentation model is iteratively updated based on the loss function to construct an image segmentation model.

[0012] According to one aspect of an embodiment of the present specification, the loss function is constructed in the following manner: constructing a Dice loss function according to the following formula; constructing a binary cross entropy loss function according to the following formula; constructing a joint loss function based on the Dice loss function and the binary cross entropy loss function, and the joint loss function is the loss function for training the initial image segmentation model.

[0013] An embodiment of the present specification also provides a breast ultrasound image segmentation device, the device comprising: an acquisition unit, used to acquire a breast ultrasound image; a mask prediction unit, used to input the breast ultrasound image into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training a YOLO module with a first training sample set, and the image segmentation model is obtained by training an initial image segmentation model with a second training sample set.

[0014] The embodiment of the present specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the breast ultrasound image segmentation method.

[0015] The embodiment of this specification also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the breast ultrasound image segmentation method is implemented. The present invention introduces an automatic prompt generation model based on YOLO, and uses its advantages in prompt box generation to achieve automatic prompts without manual interaction, greatly improving the automation level of breast ultrasound image segmentation, providing a fast and accurate segmentation method for breast cancer screening and diagnosis, and has significant clinical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 The figure is a flow chart of a method for segmenting breast ultrasound images according to an embodiment of the present specification;

[0018] Figure 2 The figure is a flow chart of a method for training a prompt generation model according to an embodiment of the present specification;

[0019] Figure 3 The figure is a flow chart of a method for calculating the loss function of a YOLO module according to an embodiment of the present specification;

[0020] Figure 4 The figure is a flow chart of a method for training an image segmentation model according to an embodiment of the present specification;

[0021] Figure 5 A flow chart of a method for constructing a loss function according to an embodiment of this specification is shown;

[0022] Figure 6 The figure shows a schematic diagram of the structure of an image segmentation model according to an embodiment of the present specification;

[0023] Figure 7 The figure shows a schematic diagram of the structure of a breast ultrasound image segmentation device according to an embodiment of the present specification;

[0024] Figure 8 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.

[0025] Description of the accompanying symbols:

[0026] 701, acquisition unit;

[0027] 702, mask prediction unit;

[0028] 802. Computer equipment;

[0029] 804, processor;

[0030] 806. Memory;

[0031] 808, driving mechanism;

[0032] 810, input / output module;

[0033] 812. Input device;

[0034] 814. Output device;

[0035] 816. Presentation equipment;

[0036] 818. Graphical user interface;

[0037] 820, network interface;

[0038] 822, communication link;

[0039] 824. Communication bus. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0041] It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0042] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0043] It should be noted that the breast ultrasound image segmentation method, device and computer equipment of this specification can be used in the field of image processing technology, and can also be used in the field of medical technology. This specification does not limit the application field of the breast ultrasound image segmentation method, device and computer equipment.

[0044] Figure 1The flowchart of a breast ultrasound image segmentation method according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0045] Step 101: Acquire a breast ultrasound image.

[0046] In this step, the breast ultrasound image is the image to be predicted, and the breast ultrasound image is usually blurred and low-contrast. The breast ultrasound image is generally divided into a normal tissue area and a tumor area. The normal tissue area includes breast tissue, adipose tissue, and muscle tissue. The tumor area presents different ultrasound characteristics depending on the benign or malignant nature of the tumor.

[0047] The lesions in breast ultrasound images are complex in shape and vary in size. The shapes of lesions in breast ultrasound images can be described as round, oval or irregular. Lesions with regular shapes (round or oval) often indicate good tissue status and are more common in benign tumors, while lesions with irregular shapes (such as lobed, crab-foot-shaped, etc.) may be dangerous and are more common in malignant tumors.

[0048] Step 102: input the breast ultrasound image into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training the YOLO module with the first training sample set, and the image segmentation model is obtained by training the initial image segmentation model with the second training sample set.

[0049] In an embodiment of the present specification, an image encoder, a mask decoder and a prompt encoder, the prompt generation model is connected to the prompt encoder, the prompt encoder is used to convert the prompt features output by the prompt generation model into a feature vector, the image encoder is used to generate an image potential representation of a breast ultrasound image, and input the image potential representation to the mask decoder, and the mask decoder is used to combine the feature vector with the image potential representation and then perform mask processing.

[0050] In order to further improve the adaptability of the model, the present invention optimizes the CBS, SPPF, C2f and Bottleneck modules in YOLO to achieve accurate extraction of subtle lesion features in breast ultrasound images.

[0051] like Figure 6 As shown in FIG. 1 , the breast ultrasound image is input into the semantic segmentation model, and the hint generation model in the semantic segmentation model automatically generates the bounding box in the breast ultrasound image and the target category information of the lesion in the bounding box. This information is converted into the form of hint coding in the subsequent semantic segmentation task. Specifically, the breast ultrasound image is input into the hint generator to obtain the output hint, which is denoted as P. The hint P is then input into the hint encoder E in the image segmentation model. p, encode the prompt P into a latent vector F p , the formula is as follows:

[0052] F p =E p (P); where F p ∈R k×c represents the latent vector of the prompt P, where P represents the prompt; F p Represents the prompt encoder. In this specification, the YOLO module in the prompt generation model uses a multi-scale learning strategy to improve the adaptability to lesions of different shapes in breast ultrasound images. Multi-scale learning enables the prompt generation model to maintain consistent detection accuracy and robustness when detecting lesion areas of different sizes and shapes.

[0053] The breast ultrasound image is input into the image segmentation model, and processed by the visual encoder and mask decoder in the image segmentation model to generate the potential representation of the breast ultrasound image, denoted as F i The latent representation of the breast ultrasound image and the latent vector output by the prompt generation model are input into the mask decoder D m , and processed by an attention-based multimodal fusion mechanism to generate a segmentation mask output.

[0054] The formula for generating the segmentation mask is as follows:

[0055] Mask=D m (F i ,F p ), where Mask represents the final segmentation mask predicted by the image segmentation model. The process in this step combines the contextual features of the image and the prompt information.

[0056] Figure 2 The flowchart of a method for training a prompt generation model according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0057] Step 201: Acquire a first training sample set, wherein the first training sample set includes: a breast ultrasound sample image and a label boundary box corresponding to a lesion area in the breast ultrasound sample image.

[0058] In this step, breast ultrasound sample images are used to train the YOLO module. The breast ultrasound sample images include ultrasound images from different patients and different lesion types, which are rich in diversity to ensure that the trained prompt generation model has good generalization ability in a wide range of practical application scenarios.

[0059] Specifically, breast ultrasound sample images and their corresponding label bounding boxes come from public datasets, such as the BUSI dataset and Dataset B, which cover various types of breast lesions, including benign and malignant tumors, ensuring the comprehensiveness of the data.

[0060] In this step, each breast ultrasound sample image is accompanied by a bounding box of the lesion area in the sample image, which is called a label bounding box. The label bounding box specifically includes the following information: the upper left coordinate of the label bounding box, the lower right coordinate of the label bounding box, the length and width of the label bounding box, and the shape of the label bounding box. The label bounding box is used as a training label for the YOLO module to enable the model to learn the ability to automatically generate a bounding box.

[0061] Step 202: input the breast ultrasound sample image into a YOLO module to obtain a predicted bounding box of the breast ultrasound sample image by the YOLO module.

[0062] Before inputting the breast ultrasound sample image into the YOLO module, this step also includes: performing multiple data enhancement operations on the breast ultrasound sample image, such as rotation, flipping, contrast adjustment, etc., to expand the sample space of the first training sample set, thereby improving the adaptability of the YOLO module to different image conditions and enhancing the robustness and generalization ability of the prompt generation model.

[0063] In the embodiment of this specification, the YOLO module mainly includes four submodules: CBS (Convolution-BatchNorm-SiLU) module, SPPF (Spatial Pyramid Pooling-Fast) module, C2f Cross Stage Fusion) module, Bottleneck module. The CBS module is the basic building block of the YOLO model. The module first extracts feature information in the image through a convolution operation, then performs batch normalization (Batch Normalization) to ensure the stability of the feature, and finally enhances the nonlinear representation ability of the model through the SiLU activation function. The CBS module can effectively enhance the model's ability to capture the lesion area and ensure the accuracy of the generated prompt box. The SPPF module is responsible for the extraction of multi-scale features, and samples and aggregates image features at multiple scales through pooling windows of different sizes. This module enables the YOLO model to maintain efficient detection capabilities in different sizes and shapes of the lesion area, and is suitable for the detection of complex lesion areas in breast ultrasound images. The design of the SPPF module further enhances the context perception ability of the model, making the generated prompt box more adaptable to the actual characteristics of the lesion.

[0064] The C2f module effectively improves the feature expression capability of the YOLO model by fusing features across stages. The C2f module contains multiple layers of convolution and residual connections, which can deeply mine subtle features in breast ultrasound images. When the model generates bounding box prompts, the presence of the C2f module can help the model more accurately capture lesion areas with blurred boundaries or unclear contrast.

[0065] As the feature compression and extraction unit of the YOLO model, the Bottleneck module uses a combination of 1×1 convolution and 3×3 convolution to efficiently compress and enhance feature dimensions. This module can significantly reduce redundant information while maintaining the main features, which helps the YOLO model maintain higher efficiency when generating prompt boxes, reduces computational complexity, and thus improves the speed of prompt box generation.

[0066] In this step, the breast ultrasound sample image is input into the YOLO module, and the YOLO module automatically outputs the predicted bounding box of the breast ultrasound sample image and the corresponding target category information. For example, for the breast ultrasound sample image of patient A, the predicted bounding box predicted by the YOLO module for the sample image is located in the glandular part, the coordinates of the upper left point of the predicted bounding box are (1.2, 6.3), the coordinates of the lower right point are (3.9, 8.5), and the corresponding target category information is: breast tumor.

[0067] In this step, the bounding box prompts are automatically generated for the breast ultrasound sample images using the YOLO module, and then the bounding box prompts are further input into the SAM model in the subsequent processing process to form a complete automated segmentation process. This modular construction method ensures the accuracy and efficiency of the system and can achieve efficient and low-manual automatic segmentation in practical applications.

[0068] Step 203: Calculate the loss function of the YOLO module according to the predicted bounding box and the label bounding box.

[0069] In this specification, the loss function is used to balance the requirements between the prompt generation and the segmentation task. The optimization of the loss function can help the prompt generator achieve higher accuracy and robustness in the breast ultrasound image segmentation task.

[0070] In some embodiments of the present specification, based on the target detection characteristics of the YOLO module, the regression accuracy of the bounding box, the target classification accuracy, and the positional relationship of the detection box are comprehensively considered to generate a loss function of the YOLO module.

[0071] In some embodiments of the present specification, multiple types of loss functions can be determined based on the label bounding box and the predicted bounding box, and the loss function of the YOLO module can be formed by combining the multiple types of loss functions.

[0072] In this specification, the loss function of the prompt generation model is based on the target detection characteristics of the YOLO module, which comprehensively considers the regression accuracy of the bounding box, the target classification accuracy, and the positional relationship of the detection box to construct the loss function.

[0073] Step 204, iteratively updating the YOLO module according to the loss function to construct a prompt generation model. The YOLO module is iteratively updated according to the loss function in step 203. Specifically, the weight parameters in the YOLO module are iteratively updated until the loss function values ​​of the predicted bounding box and the label bounding box of the YOLO module are calculated to converge to a preset numerical range, or the number of iterations of the YOLO module reaches a preset threshold, then it is determined that the YOLO module training is completed, and the current YOLO module is used as the constructed prompt generation model.

[0074] Figure 3 The flowchart of a method for calculating the loss function of the YOLO module in an embodiment of this specification is shown, which specifically includes the following steps:

[0075] Step 301, determine the center coordinate loss function according to the center coordinates of the label bounding box and the center coordinates of the predicted bounding box. In this step, the center coordinate loss function optimizes the matching degree between the bounding box generated by the prompt generator and the real bounding box by calculating the square error between the center coordinates (x, y) of the predicted bounding box and the center coordinates of the real bounding box. By comparing the IoU (Intersection over Union) between the generated box and the real bounding box, the model can better fit the boundary of the breast lesion area and improve the accuracy of the prompt box.

[0076] The formula of the center coordinate loss function is as follows:

[0077] Among them, (x i ,y i ) represents the i-th predicted bounding box, Represents the center coordinates of the label bounding box. S represents the size of the grid, and B represents the number of bounding boxes predicted for each grid.

[0078] λ coord The weight used to balance the bounding box regression loss in the total loss is usually set to a larger value (e.g., 5) to ensure that the model prioritizes optimizing the position and size of the bounding box.

[0079] Step 302: Determine a dimensionality loss function according to the size of the label bounding box and the size of the predicted bounding box. The formula of the dimensionality loss function is as follows:

[0080]

[0081] Among them, Loss dim Represents the Dimension Loss function, which is used to calculate the difference in width and height between the predicted box and the real box, λ dim Represents the weight coefficient of the dimensional loss function, which is used to balance the importance of different loss terms. represents the indicator function. If there is a target object in the jth bounding box of the i-th grid, then The value of the exponential function is 1, otherwise it is 0; S represents the size of the grid, B represents the number of bounding boxes predicted for each grid; w i represents the width of the i-th predicted bounding box, h i Represents the height of the i-th predicted bounding box. represents the width of the ground truth bounding box, Represents the height of the true bounding box. In this step, in order to give the same penalty to objects of different scales, the width and height are usually taken as square roots to calculate the error.

[0082] Step 303: Determine a lesion category loss function according to the lesion category indicated by the label bounding box and the lesion category indicated by the predicted bounding box.

[0083] The YOLO module is responsible for both target detection and classification tasks. The lesion category loss function is used to ensure that the generated prompt box correctly identifies the category of breast lesions (for example, whether there is a lump).

[0084] The lesion category loss function is as follows:

[0085] Among them, Loss class represents the lesion category loss function, which is used to measure the difference between the predicted category probability and the true category probability. i (c) represents the lesion category indicated by the predicted bounding box in the i-th grid, Represents the lesion category indicated by the label bounding box in the cth grid, and the category includes: at least one of a tumor or no tumor.

[0086] Step 304: determine a confidence loss function according to the true confidence corresponding to the label bounding box and the predicted confidence corresponding to the predicted bounding box.

[0087] This step determines different confidence loss functions according to the lesion category indicated by the prediction bounding box in step 303. Including: when the prediction bounding box indicates that there is no lesion or no tumor, the corresponding configuration does not contain the confidence loss of the target; when the prediction bounding box indicates that there is a lesion or a tumor, the corresponding configuration contains the confidence loss of the target. The confidence loss function adjusts the weight according to whether the target is contained in the prediction bounding box, thereby reducing the probability of generating a prompt box in a non-lesion area. Such a loss function can ensure that the prompt box generated by the prompt generation model has a high confidence and reduce the occurrence of false detection.

[0088] Specifically, the formula of the confidence loss function including the target is as follows:

[0089] Among them, conf_obj represents the confidence of the predicted bounding box indicating the presence of a lesion, C i represents the confidence of the prediction, represents the ground truth confidence, S represents the size of the grid, and B represents the number of bounding boxes predicted for each grid;

[0090] The formula for the confidence loss function without the target is as follows:

[0091] Among them, Conf_noobj represents the confidence that there is no lesion in the predicted bounding box, C i represents the confidence of the prediction, represents the ground truth confidence, S represents the size of the grid, B represents the number of bounding boxes predicted for each grid, and λ noobj Represents the weight factor, which is usually set to a smaller value (such as 0.5) to reduce the false positives of the model in the background area.

[0092] Step 305, determining the loss function of the YOLO module according to the center coordinate loss function, the dimension loss function, the lesion category loss function and the confidence loss function.

[0093] The loss function of the YOLO module is as follows:

[0094] Total Loss = Loss coord +Loss dim +Loss conf_obj +Loss conf_noobj +Loss class Among them, Loss coord Represents the center coordinate loss function, Loss dimRepresents the dimension loss function, Loss conf_obj Represents the confidence loss function of the target, Loss conf_noobj Represents the confidence loss function that does not contain the target, Loss class Represents the lesion category loss function. Through the above loss function, the YOLO module can generate a prompt box that highly matches the lesion area while maintaining a high detection accuracy.

[0095] Figure 4 The flowchart of a method for training an image segmentation model according to an embodiment of the present specification is shown, which specifically includes the following steps:

[0096] Step 401: Acquire a second training sample set, where the second training sample set includes: breast ultrasound sample images and mask labels corresponding to the breast ultrasound sample images.

[0097] In this step, the second training sample set is used to train the image segmentation model. The second training sample set includes: breast ultrasound sample images and corresponding mask labels. The breast ultrasound sample images in the second training sample set can be the same as the breast ultrasound sample images in the first training sample set, or they can be different images.

[0098] Furthermore, the mask label corresponding to the breast ultrasound sample image is a binary mask image, which can be an image predetermined by medical staff. In the embodiment of the present specification, the mask label image is a binary or Boolean image of the same size as the breast ultrasound sample image, in which the selected area is marked as 1 (or True) and the remaining areas are marked as 0 (or False), which is used to identify different semantic areas to be segmented in the breast ultrasound sample image. The role of the mask is to limit the image processing operation to only a specific area without affecting other parts of the image.

[0099] Step 402: input the breast ultrasound sample image into an initial image segmentation model to obtain a prediction mask of the breast ultrasound sample image by the initial image segmentation model.

[0100] In the embodiments of the specification, the image segmentation model (SAM Segment Anything Model) is responsible for the actual segmentation task in the present invention. The initial image segmentation model consists of the following three core modules: visual encoder, mask decoder and prompt encoder.

[0101] In this specification, the visual encoder (ViT, Vision Transformer) is the basic image feature extractor of the initial image segmentation model. The visual encoder divides the input breast ultrasound sample image into multiple small patches, converts it into linear features and adds position encoding, and then inputs it into the Transformer architecture. With its local feature capture capability, the visual encoder provides accurate and comprehensive feature representation for the initial image segmentation model, effectively extracts complex features in breast ultrasound images, and generates high-dimensional image embeddings for subsequent prompting and segmentation tasks.

[0102] In this specification, Mask Decoder is the segmentation module of the initial image segmentation model, responsible for generating the final segmentation mask. Mask Decoder fuses the image features extracted by the visual encoder with the prompt features of the PromptEncoder, and generates the segmentation result through a multi-layer Transformer-based structure. Mask Decoder uses the box prompts generated by YOLO to perform multimodal information fusion to ensure that the segmentation mask can accurately match the lesion area. The Mask Decoder enables the initial image segmentation model to segment the input image without human interaction, achieving automation and high precision of segmentation.

[0103] In this specification, the prompt encoder is used to encode prompt information (such as the bounding box generated by YOLO) into a potential feature vector. The encoder can process different types of prompts and convert the box prompts generated by YOLO into a potential representation suitable for processing by the segmentation model. This encoding process ensures that the automatically generated prompts can be accurately passed to the segmentation module and improve the model's attention to the lesion area in the segmentation task, thereby generating more accurate segmentation results.

[0104] Step 403: construct a loss function according to the mask label and the predicted mask.

[0105] Specifically, according to the predicted mask (segmentation result) and mask label generated in step 402, the loss function is calculated, and the weight parameters of the fully automatic semantic segmentation network are updated using the gradient descent algorithm to improve the segmentation accuracy. The loss function in this step is constructed by the Dice loss function and the binary cross entropy loss function.

[0106] Figure 5 The flowchart of a method for constructing a loss function in an embodiment of the present specification is shown, which specifically includes the following steps:

[0107] Step 501, construct a Dice loss function.

[0108] In this step, the Dice loss function is used to measure the overlap between the predicted mask and the true mask label to optimize the regional consistency of the trained image segmentation model. For the lesion area in the breast ultrasound sample image, the Dice loss function helps the image segmentation model to cover the target area more accurately and improve the segmentation consistency.

[0109] In this step, the calculation formula of the Dice loss function is as follows:

[0110] Among them, G i represents the mask label of the i-th breast ultrasound sample image, P i Represents the predicted mask of the initial image segmentation model for the i-th breast ultrasound sample image.

[0111] Step 502, construct a binary cross entropy loss function.

[0112] The binary cross entropy loss function is constructed according to the following formula:

[0113] L BCE =-∑ i [G i log(P i )+(1-G i )log(1-P i )]; where G i represents the mask label of the i-th breast ultrasound sample image, P i Represents the predicted mask of the i-th breast ultrasound sample image by the initial image segmentation model. Binary cross entropy loss (BCE) is used to optimize the classification accuracy of each pixel and effectively solve the problem of blurred lesion boundaries. By calculating the error between the predicted mask and the true mask at each pixel value, the binary cross entropy loss function segments the edge, so that the initial image segmentation model can still maintain a high segmentation accuracy in complex morphology and low contrast areas.

[0114] Step 503: construct a joint loss function according to the Dice loss function and the binary cross entropy loss function, where the joint loss function is a loss function for training the initial image segmentation model.

[0115] Step 404, iteratively updating the initial image segmentation model according to the loss function to construct an image segmentation model.

[0116] In this step, the Dice loss function and the binary cross entropy loss function are combined to construct a joint loss function, and the initial image segmentation model is iteratively trained using the joint loss function until the value of the loss function of the initial image segmentation model converges to a preset range, and the initial image segmentation model training is determined to be completed, and the image segmentation model is constructed. In this specification, the Dice loss function and the binary cross entropy loss function can be added as the joint loss function, and the formula is as follows: L = (1-α)L Dice +αL BCE ; Among them, α represents the weight coefficient, which is used to adjust the contribution of binary cross entropy loss in the total loss.

[0117] In this manual, the joint loss function combines the Dice loss function and the binary cross entropy loss function to maximize the coverage of the lesion area and improve the accuracy of the boundary area, ensuring that the initial image segmentation model has stable segmentation performance when processing breast ultrasound images. During the training process, these two loss functions are used to supervise mask prediction. The Dice loss function helps optimize the smoothness of the model when predicting the mask.

[0118] In this specification, an automated process suitable for large-scale breast ultrasound image segmentation is formed through the collaborative application of automatic hint generation, segmentation model optimization and dual-module loss function.

[0119] Figure 6 The structure diagram of an image segmentation model of an embodiment of the present specification is shown in the figure. In the figure, the image segmentation model includes: a visual encoder (also called an image encoder), a Mask Decoder (also called a mask decoder) and a Prompt encoder. The visual encoder (image encoder) is denoted as E i , let the Mask Decoder be E p , let the prompt encoder be D m The breast ultrasound image is used as the input of the image segmentation model, where the prompt P can be a point, a box, or a rough mask. Specifically, the breast ultrasound image I is input to the visual encoder E i To generate the latent representation F of breast ultrasound images i , the formula is as follows:

[0120] F i =E i (I); among them, E i represents the visual encoder (image encoder), I represents the breast ultrasound sample image, and F i Represents the potential representation of the breast ultrasound image. The potential representation of the breast ultrasound image is the image feature after the image encoder encodes the input image.

[0121] This manual uses Segment Anything Model (SAM) for precise segmentation, which includes the collaborative work of the visual encoder, prompt encoder, and mask decoder modules. It can fully adapt to the low contrast and fuzzy boundaries of breast ultrasound images, ensuring the accuracy and stability of the segmentation results.

[0122] Furthermore, the encoded image feature F i And the hint feature is input to the mask decoder (mask decoder) D m , and processed by the attention-based multimodal fusion mechanism. Among them, the prompt feature is obtained by the output of the prompt generation model, and its specific description is detailed in Figure 1 In this specification, the formula for the segmentation mask output by the mask decoder in the image segmentation model for the image features and the hint features is as follows:

[0123] Mask=D m (F i ,F p ), where Mask represents the final segmentation mask predicted by the image segmentation model. For details on the generation of the hint feature, see Figure 1 describe.

[0124] like Figure 7 The figure shows a schematic diagram of the structure of a breast ultrasound image segmentation device according to an embodiment of the present specification. The figure describes the basic structure of the breast ultrasound image segmentation device, wherein the functional units and modules can be implemented by software, or by using a general chip or a specific chip to implement breast ultrasound image segmentation. The device specifically includes:

[0125] An acquisition unit 701 is used to acquire a breast ultrasound image;

[0126] The mask prediction unit 702 is used to input the breast ultrasound image into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training the YOLO module with the first training sample set, and the image segmentation model is obtained by training the initial image segmentation model with the second training sample set.

[0127] The present invention innovatively introduces an automatic prompt generation model based on YOLO in the segmentation process by combining automatic prompt generation with a segmentation model, and uses its advantages in prompt box generation to achieve automatic prompts without manual interaction, greatly improving the automation of breast ultrasound image segmentation. In addition, the present invention uses Segment AnythingModel (SAM) for precise segmentation, which includes the collaborative work of ViT encoder, Prompt Encoder and Mask Decoder modules, which can fully adapt to the low contrast and fuzzy boundaries of breast ultrasound images, ensuring the accuracy and stability of the segmentation results. In order to further improve the adaptability of the model, the present invention optimizes the CBS, SPPF, C2f and Bottleneck modules in YOLO to achieve accurate extraction of subtle lesion features in breast ultrasound images. In terms of loss function, a dual-module structure is designed to combine the YOLO detection loss with the Dice-BCE composite loss of SAM to make the segmentation process more robust. The present invention forms an automated process suitable for large-scale breast ultrasound image segmentation through the coordinated application of automatic prompt generation, segmentation model optimization and dual-module loss function, providing a fast and accurate segmentation method for breast cancer screening and diagnosis, and has significant clinical application value.

[0128] Figure 8 As shown, a computer device provided by an embodiment of this specification. The breast ultrasound image segmentation method described in this application can be applied to the computer device. The computer device 802 may include one or more processors 804, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 802 may also include any memory 806, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 806 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 802. In one case, when the processor 804 executes an associated instruction stored in any memory or a combination of memories, the computer device 802 may perform any operation of the associated instruction. The computer device 802 also includes one or more drive mechanisms 808 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0129] The computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input devices 812) and for providing various outputs (via output devices 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface (GUI) 818. In other embodiments, the input / output module 810 (I / O), the input device 812, and the output device 814 may not be included, and the computer device 802 may be used as a computer device in a network. The computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0130] The communication link 822 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0131] Corresponds to Figures 1 to 5 Method, the embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0132] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 5 The method shown.

[0133] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0134] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0135] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0137] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0139] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0141] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A breast ultrasound image segmentation method, characterized in that: The method comprises: Obtain breast ultrasound images; The breast ultrasound image is input into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training a YOLO module with a first training sample set, and the image segmentation model is obtained by training an initial image segmentation model with a second training sample set.

2. The method according to claim 1, characterized in that The image segmentation model includes: an image encoder, a mask decoder and a prompt encoder. The prompt generation model is connected to the prompt encoder. The prompt encoder is used to convert the prompt features output by the prompt generation model into feature vectors. The image encoder is used to generate an image potential representation of a breast ultrasound image and input the image potential representation into the mask decoder.

3. The method according to claim 2, characterized in that The mask decoder is used to combine the feature vector with the image potential representation and then perform mask processing.

4. The method according to claim 1, characterized in that The prompt generation model is trained as follows: Acquire a first training sample set, wherein the first training sample set includes: a breast ultrasound sample image and a label boundary box corresponding to a lesion area in the breast ultrasound sample image; Inputting the breast ultrasound sample image into a YOLO module to obtain a predicted bounding box of the breast ultrasound sample image by the YOLO module; Calculate the loss function of the YOLO module based on the predicted bounding box and the label bounding box; The YOLO module is iteratively updated according to the loss function to construct a prompt generation model.

5. The method according to claim 4, characterized in that According to the predicted bounding box and the label bounding box, the loss function of the YOLO module is calculated including: Determining a center coordinate loss function according to the center coordinates of the label bounding box and the center coordinates of the predicted bounding box; Determining a dimensionality loss function according to the size of the label bounding box and the size of the prediction bounding box; Determining a lesion category loss function according to the lesion category indicated by the label bounding box and the lesion category indicated by the predicted bounding box; Determining a confidence loss function according to the true confidence corresponding to the label bounding box and the predicted confidence corresponding to the predicted bounding box; The loss function of the YOLO module is determined according to the center coordinate loss function, the dimension loss function, the lesion category loss function and the confidence loss function.

6. The method according to claim 1, characterized in that The image segmentation model is trained as follows: Acquire a second training sample set, wherein the second training sample set includes: breast ultrasound sample images and mask labels corresponding to the breast ultrasound sample images; Inputting the breast ultrasound sample image into an initial image segmentation model to obtain a prediction mask of the breast ultrasound sample image by the initial image segmentation model; Constructing a loss function according to the mask label and the predicted mask; The initial image segmentation model is iteratively updated according to the loss function to construct an image segmentation model.

7. The method according to claim 6, characterized in that The loss function is constructed as follows: Construct Dice loss function; Construct a binary cross entropy loss function; According to the Dice loss function and the binary cross entropy loss function, a joint loss function is constructed, and the joint loss function is the loss function for training the initial image segmentation model.

8. A breast ultrasound image segmentation device, characterized in that: The device comprises: An acquisition unit, used for acquiring a breast ultrasound image; A mask prediction unit is used to input the breast ultrasound image into a semantic segmentation model to obtain a mask in the breast ultrasound image; wherein the semantic segmentation model is composed of a prompt generation model and an image segmentation model, the prompt generation model is obtained by training a YOLO module with a first training sample set, and the image segmentation model is obtained by training an initial image segmentation model with a second training sample set.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images