Video stream three-dimensional automatic breast ultrasound tumor time sequence consistent segmentation method based on fine-tuning video segmentation base model, electronic device and computer readable storage medium

Abstract

This invention discloses a method for temporally consistent 3D automatic breast ultrasound tumor segmentation based on a fine-tuned video segmentation model, an electronic device, and a computer-readable storage medium. Addressing issues such as inter-frame tissue displacement, deformation, and low contrast in 3D ABUS video streams, this invention introduces a temporal optical flow compensation module, a dynamic attention fusion mechanism, and a video memory unit into the video segmentation model. Through fine-tuning training, it achieves temporally consistent automatic segmentation of the tumor region. Specifically, this invention achieves temporally consistent tumor region segmentation through historical information and mask alignment, optical flow consistency detection, pixel-level adaptive fusion, and cross-frame context retrieval. Compared with existing methods, this invention outputs a stable and continuous 3D volume mask through frame-by-frame reasoning of the 2D frame sequence and 3D post-processing, effectively improving the temporal consistency of inter-frame segmentation, the robustness of low-contrast regions, and the stability of 3D reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, and in particular to a method for three-dimensional automatic temporal consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] 3D ABUS images typically consist of a sequence of two-dimensional slices acquired continuously along the scanning direction. The number of slices for a single volumetric data point is generally frame-by-frame. The tissue structure between frames is affected by the uniform scanning speed of the probe, non-rigid tissue deformation, speckle noise, back attenuation, and local hypoechoic areas, resulting in positional shifts, blurred boundaries, and abrupt texture changes. For this type of data, if only a frame-by-frame independent segmentation method is used, the result of each frame is determined only by the current image, and the structural continuity between adjacent frames cannot be explicitly utilized. If only a simple propagation method based on the mask of the previous frame is used, the error in the result of the previous frame will continue to accumulate along the sequence direction. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a three-dimensional automatic temporally consistent segmentation method, electronic device, and computer-readable storage medium for breast ultrasound tumors based on a fine-tuned video segmentation model, thereby solving the temporal consistency problem in image sequence segmentation mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for three-dimensional automatic temporally consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model, comprising the following steps: S1. Acquire the medical image data to be segmented, and resample the data into a frame sequence along the scanning direction; S2. Perform preprocessing on the frame sequence, including grayscale normalization, spatial resampling, and contrast enhancement; S3. Construct a segmentation network based on SAM3, and add a temporal optical flow compensation module and a dynamic attention fusion mechanism; The temporal optical flow compensation module uses a multi-layer pyramid structure to calculate the motion vector field between adjacent frames, and performs reverse mapping of the features and mask probability map of the previous frame based on the motion vector field to achieve motion alignment. The dynamic attention fusion mechanism uses the current frame features as the query and the aligned historical features as the key and value, and generates pixel-level fusion weights based on the entropy uncertainty map. S4. Fine-tune the segmentation network using the training set, freeze the pre-trained parameters of the image encoder and cue encoder, and only update some parameters of the mask decoder, the temporal optical flow compensation module, the dynamic attention fusion mechanism, and the LoRA adapter. S5. Using the trained model, reason frame by frame about the sequence to be segmented, output a two-dimensional mask sequence, and then fill the mask sequence back into a three-dimensional volume mask according to the original scanning order. S6. Perform post-processing on the 3D volume mask and output the 3D tumor segmentation results.

[0005] A further optimization of the present invention includes a video memory unit; The video memory unit uses a key-value structure to store context information of multiple frames. Before processing the current frame, it calculates the similarity between the encoded features of the current frame and the keys retrieved from historical frames, recalls the corresponding memory features, and inputs them into subsequent modules along with the features of the current frame.

[0006] A further optimization of this invention is that the temporal optical flow compensation module adopts a three-layer pyramid structure to estimate the motion vector field at different scales to achieve multi-scale motion estimation.

[0007] In a further optimization of the present invention, the temporal optical flow compensation module simultaneously calculates the forward optical flow and the backward optical flow, using the endpoint error as a consistency evaluation index; When the endpoint error of a pixel location exceeds a preset threshold, the location is marked as an untrusted region, and the fusion weight of that location is forcibly set to a preset value so that only the information of the current frame is used.

[0008] A further optimization of the present invention is that the dynamic attention fusion mechanism includes a multi-head attention unit and a gating weight generation unit.

[0009] A further optimization of the present invention is that if the confidence level of the current frame mask is lower than a preset threshold, the empty mask is retained and the next frame is processed; the post-processing includes retaining the largest connected component above a preset voxel number threshold, and performing a three-dimensional morphological closing operation with a preset radius.

[0010] A further optimization of this invention includes training loss comprising at least one of Dice loss, pixel-level cross-entropy loss, temporal consistency loss, and boundary loss. An electronic device includes a memory and a processor, the memory for storing a computer program and the processor for executing the computer program to implement the above-described method for automatic three-dimensional temporally consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model.

[0011] A computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described method for automatic three-dimensional temporally consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model.

[0012] The beneficial effects of this invention are: (1) Improved temporal consistency. Existing methods lack explicit motion alignment, which can easily lead to mask jumps when there is displacement or deformation between frames. This invention improves the continuity of inter-frame segmentation by introducing a temporal optical flow compensation module for motion alignment before fusion and by masking untrusted regions.

[0013] (2) Improved segmentation accuracy. Existing methods mostly rely on frame-by-frame segmentation or simple historical mask propagation, which fails to fully utilize the temporal information of the feature layer. This invention adopts a dynamic attention fusion mechanism to adaptively weight the features of the current frame and historical frames, and combines the multi-frame context of the video memory unit, which significantly improves the segmentation accuracy.

[0014] (3) Improved robustness in low-contrast images. This addresses the issue of low contrast between tumors and glands in ABUS images (…). Existing methods are prone to missing segments. This invention achieves stable alignment through temporal optical flow compensation and utilizes the uncertainty gating mechanism in dynamic attention fusion to rely more on the current frame information in low confidence regions, effectively reducing missing segments and exhibiting stronger robustness in low contrast regions.

[0015] (4) Improved boundary stability. Existing methods are prone to jitter at inter-frame boundaries, resulting in uneven boundaries after 3D reconstruction. This invention maintains the continuity of adjacent frame boundaries through motion alignment and temporal consistency loss constraints.

[0016] (5) Improved ability to suppress false propagation. Existing methods are prone to propagating erroneous historical information to subsequent frames when encountering occlusion, artifacts, or severe local deformation. This invention marks untrusted regions by detecting forward-backward optical flow consistency and forces the weight of the corresponding position to 1 during dynamic fusion, thereby blocking the propagation of erroneous history and effectively suppressing false positives caused by false propagation.

[0017] (6) Deployability optimization. While ensuring segmentation performance, this invention also takes into account the memory usage requirements in actual deployment.

[0018] (7) Improved stability of 3D reconstruction. Existing methods often result in a large number of fragmented connected components in the reconstruction results due to inconsistent segmentation between frames when stacking 2D segmentation sequences into a 3D volume mask, and the lesion area fluctuates greatly between adjacent slices. This invention can effectively suppress the generation of fragmented connected components by combining temporally consistent segmentation results with maximum connected component screening and morphological closing operations in 3D post-processing. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0020] Figure 2 This is a block diagram of the basic model structure for fine-tuning video segmentation based on the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0022] I. Data Acquisition and Sequence Construction In the embodiment, a total of 3D ABUS volume data were acquired. For example, each volume data set includes the original grayscale volume data and the corresponding labeled volume data. After converting the data from DICOM format to NIFTI format, isotropic resampling is performed on the volume data, and the voxel spacing after resampling is set to... Subsequently, two-dimensional slices were extracted along the scanning direction to form a sequence. ,in Frame; each frame of image is uniformly processed by center cropping and zero padding. Datasets by The dataset was divided into training, validation, and test sets, with 224, 32, and 64 cases respectively. The training phase used a length... Training samples are constructed using overlapping segments with a step size of 8.

[0023] II. Network Structure and Key Module Parameters (I) SAM3 Main Structure. This embodiment uses the SAM3 video segmentation main network as the base model, with the image encoder being a visual Transformer structure and the patch size being [missing information]. The input is At that time, it formed The token grid is used, and the encoder outputs tokens with a dimension of 768. The cue encoder receives two types of input: point cues and box cues. The frame uses the default center point prompt. The frame uses box cues and centroid cues generated from the alignment probability map of the previous frame, with the number of cues... The mask decoder outputs one binary mask channel and one confidence channel, with the confidence threshold set to 0.5.

[0024] (II) Temporal Optical Flow Compensation Module. The optical flow estimation unit adopts a three-layer pyramid structure, with scales of... and ,exist Scale Output final optical flow at resolution The relevance search window is set to... The optical flow update iterations are fixed at 12 times. The motion compensation unit updates the optical flow based on the upsampled optical flow. right and A reverse mapping is performed using bilinear interpolation. To suppress occlusion errors, the endpoint errors of both the forward and backward optical flows are calculated simultaneously. ;when When pixels are used, the corresponding position is Forced to be set to 1.

[0025] (III) Dynamic Attention Fusion Mechanism. Multi-head attention units utilize the features of the current frame. For querying, with and Composed of key and value, number of heads The total dimension is 256, and attention is calculated in... It is executed on the token grid. The gating weight generation unit consists of two layers. The system is constructed using convolutions, with the first layer having 64 output channels and the second layer having 1 output channel. get Entropy uncertainty diagram The probability is obtained by using the current frame mask logits and then applying Softmax. Post-calculation , After being normalized to the 0 to 1 interval, it is used in the gating calculation.

[0026] (iv) Video Memory Unit. The video memory unit uses a key-value method to store the most recently viewed video data. The system reads the image features and mask clues of each frame, and writes and evicts them at one-frame update intervals. Before encoding the current frame, the system first reads the history. Frame memory to obtain cross-frame context.

[0027] III. Fine-tuning Strategies and Training Process In this embodiment, SAM3 is trained using an efficient parameter fine-tuning method: all parameters of the image encoder and cue encoder are frozen, and only the parameters of the last two linear mapping layers of the mask decoder, the parameters of the temporal optical flow compensation module, the parameters of the dynamic attention fusion mechanism, and the LoRA adapter inserted into the attention layer of the mask decoder are trained; LoRA rank Scaling factor Training takes sequence segments as input, with segment length... Frames, batch size The optimizer uses AdamW with an initial learning rate of... The weights decay to 0.01, and the learning rate is cosine annealed to... The warm-up steps are 1000, and the training rounds are 120. The total loss function is... , in Define as the current mask Compared to the optical flow compensated mask of the previous frame The L1 distance between them After calculating the edges using the Sobel operator, the cross-entropy is calculated within the edge bandwidth w = 3 pixels. For Set sample weights for low-contrast samples ,for Sample settings .

[0028] IV. Reasoning Process and 3D Reconstruction During the inference phase, the video memory is maintained frame by frame in the original sequence order. For the first frame of the sequence (t=1), since there is no historical mask information, the image center point is used as the default point to prompt the encoder to start the segmentation process. In practical applications, this initialization prompt can also be manually selected by the user or replaced by other preset methods by the system, and does not constitute a limitation on the subsequent core inference process of this invention. For frames t>1, as the core feature of the inference phase of this invention, the system first utilizes optical flow. The probability map of the previous frame Align to the current frame coordinate system to obtain Then from Automatically extract the smallest bounding rectangle as the bounding box tooltip and extract the mask centroid as the point tooltip (number of point tooltips). =1); then the current frame image, historical memory features, aligned features of the previous frame, and the automatically generated prompts are input into the network to output the mask probability map of the current frame. and binary mask If the confidence level of the current frame mask is lower than a preset threshold (e.g., 0.5), an empty mask is retained and the next frame is processed to suppress the continued propagation of low-confidence false positive regions. After the two-dimensional masks of all frames are output, the system stacks them into a three-dimensional volume mask in the original scanning order. Then, the voxel count threshold is applied. The maximum connected component selection and the three-dimensional morphological closing operation of the radius r=2 voxels are used to finally output a stable and continuous three-dimensional tumor segmentation result.

[0029] The parameters described in the above embodiments can be adjusted within the following ranges without departing from the protection scope of this invention: the frame resolution can be adjusted within the following ranges. to Adjusting between these ranges, the optical flow estimation resolution can be [adjusted / converted / advertised]. to Adjust between, memory length The number of heads can be adjusted between 4 and 16, indicating the bullish focus. It can be adjusted between 4 and 16.

[0030] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatic three-dimensional temporally consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model, characterized in that, Includes the following steps: S1. Acquire the medical image data to be segmented, and resample the data into a frame sequence along the scanning direction; S2. Perform preprocessing on the frame sequence, including grayscale normalization, spatial resampling, and contrast enhancement; S3. Construct a segmentation network based on SAM3, and add a temporal optical flow compensation module and a dynamic attention fusion mechanism; The temporal optical flow compensation module uses a multi-layer pyramid structure to calculate the motion vector field between adjacent frames, and performs reverse mapping of the features and mask probability map of the previous frame based on the motion vector field to achieve motion alignment. The dynamic attention fusion mechanism uses the current frame features as the query and the aligned historical features as the key and value, and generates pixel-level fusion weights based on the entropy uncertainty map. S4. Fine-tune the segmentation network using the training set, freeze the pre-trained parameters of the image encoder and cue encoder, and only update some parameters of the mask decoder, the temporal optical flow compensation module, the dynamic attention fusion mechanism, and the LoRA adapter. S5. Using the trained model, reason frame by frame about the sequence to be segmented, output a two-dimensional mask sequence, and then fill the mask sequence back into a three-dimensional volume mask according to the original scanning order. S6. Perform post-processing on the 3D volume mask and output the 3D tumor segmentation results.

2. The method for three-dimensional automatic temporal consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model as described in claim 1, characterized in that, It also includes a video memory unit; The video memory unit uses a key-value structure to store context information of multiple frames. Before processing the current frame, it calculates the similarity between the encoded features of the current frame and the keys retrieved from historical frames, recalls the corresponding memory features, and inputs them into subsequent modules along with the features of the current frame.

3. The method for three-dimensional automatic temporal consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model as described in claim 1, characterized in that, The temporal optical flow compensation module uses a three-layer pyramid structure to estimate the motion vector field at different scales to achieve multi-scale motion estimation.

4. The method for automatic three-dimensional temporally consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model as described in claim 1 or 3, characterized in that, The temporal optical flow compensation module calculates both forward and backward optical flow simultaneously, using endpoint error as a consistency evaluation index. When the endpoint error of a pixel location exceeds a preset threshold, the location is marked as an untrusted region, and the fusion weight of that location is forcibly set to a preset value so that only the information of the current frame is used.

5. The method for three-dimensional automatic temporal consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model as described in claim 1, characterized in that, The dynamic attention fusion mechanism includes a multi-head attention unit and a gating weight generation unit.

6. The method for three-dimensional automatic temporal consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model as described in claim 1, characterized in that, If the confidence level of the current frame mask is lower than the preset threshold, then the empty mask is retained and the next frame is processed. Post-processing includes preserving the largest connected component above a preset voxel number threshold and performing a three-dimensional morphological closing operation with a preset radius.

7. The method for automatic three-dimensional temporally consistent segmentation of breast ultrasound tumors based on a fine-tuned video segmentation model as described in claim 1, characterized in that, The training loss includes at least one of Dice loss, pixel-level cross-entropy loss, temporal consistency loss, and boundary loss.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory is used to store computer programs and the processor is used to execute the computer programs to implement the video stream-based three-dimensional automatic temporally consistent segmentation method for breast ultrasound tumors based on a fine-tuned video segmentation baseline model as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, Used to store computer programs, which, when executed by a processor, implement the video stream-based three-dimensional automatic temporally consistent segmentation method for breast ultrasound tumors based on a fine-tuned video segmentation fundamental model as described in any one of claims 1-7.

Images